New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stuck "Negotiating SSL", resulted in a large memory leak #123

Closed
sman591 opened this Issue Nov 10, 2015 · 15 comments

Comments

Projects
None yet
6 participants
@sman591

sman591 commented Nov 10, 2015

I'm trying to connect to a Postgres database over SSH, except it gets stuck in the "Negotiating SSL" state.
screenshot 2015-11-10 17 43 31

I just happened to leave my computer with this running for ~30-50 minutes, and came back to everything on my computer in a "paused" state due to no memory being left over. Postico had managed to consume 60GB of memory:
screenshot 2015-11-10 17 21 04

I tried connecting in PG Commander, and the same issue existed, except it got stuck in the SSH stage and the memory leak was far more prominent (reached 1GB in under a minute and climbed extremely fast).

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Nov 11, 2015

Owner

Thanks for telling me about this.

Often when it gets stuck at this stage there's a firewall that prevents the connection.

However, that obviously shouldn't cause a memory leak. I'll have to look at the connection loop; I might have forgotten to free some data received from libpq. The connection loop uses more efficient polling in Postico, so it makes sense that the issue is worse in PG Commander.

Could you let me know which version/build of Postico and OS X you are using?

Does the memory usage drop to normal when you click cancel?

Owner

jakob commented Nov 11, 2015

Thanks for telling me about this.

Often when it gets stuck at this stage there's a firewall that prevents the connection.

However, that obviously shouldn't cause a memory leak. I'll have to look at the connection loop; I might have forgotten to free some data received from libpq. The connection loop uses more efficient polling in Postico, so it makes sense that the issue is worse in PG Commander.

Could you let me know which version/build of Postico and OS X you are using?

Does the memory usage drop to normal when you click cancel?

@sman591

This comment has been minimized.

Show comment
Hide comment
@sman591

sman591 Nov 11, 2015

Build 1.0 (1243), OS X 10.10.5.

No, pressing cancel does not drop the memory usage, at least for the initial try. I just now got it up to 6.31GB while writing this comment. But, I then tried again, got it up to ~7GB, pressed cancel, and it went down to 6.31GB. I can't always reproduce the issue. It must be some specific sequence of events that causes it.

I found out that the hostname I was trying to connect to over SSH wasn't available to the SSH scope I was in (it was only available inside a docker container). Via command line, I could successfully SSH into the box, but if I tried to connect to the database using psql with the same credentials that I put into Postico, it wouldn't connect.

Hope that makes sense? There isn't any firewall in place - though the same sort of principle applies that the Posgres host it tries to connect to isn't resolvable. Probably something that can be done about that.

sman591 commented Nov 11, 2015

Build 1.0 (1243), OS X 10.10.5.

No, pressing cancel does not drop the memory usage, at least for the initial try. I just now got it up to 6.31GB while writing this comment. But, I then tried again, got it up to ~7GB, pressed cancel, and it went down to 6.31GB. I can't always reproduce the issue. It must be some specific sequence of events that causes it.

I found out that the hostname I was trying to connect to over SSH wasn't available to the SSH scope I was in (it was only available inside a docker container). Via command line, I could successfully SSH into the box, but if I tried to connect to the database using psql with the same credentials that I put into Postico, it wouldn't connect.

Hope that makes sense? There isn't any firewall in place - though the same sort of principle applies that the Posgres host it tries to connect to isn't resolvable. Probably something that can be done about that.

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Nov 16, 2015

Owner

Okay, so I found out that the status "Negotiating SSL..." is a bit inaccurate. This is the first status that the PostgreSQL client library libpq reports after opening the connection. What Postico really does is wait for the SSH server to open the connection to the PostgreSQL server.

Anyway, I've been looking through the connection loop and I couldn't find anything the allocates memory while waiting for the connection. I also tried reproducing your issue by connecting to an SSH server and then connecting to a non-existing address like "test.local". However, Postico did not consume any memory in my testing.

Anyway, if a host is not resolvable, usually the SSH server should return an error after a short time. Because of the hang I assumed that a firewall was involved -- they are often configured in a way to silently swallow packets. It seems that Docker does something similar, but since I have no experience with Docker I really don't know what is going on.

I'll try some other things to reproduce your issue, but for now I really don't know how to find out where that memory leak is coming from. (Unless you happen to know your way around Instruments and the allocation tool...)

Owner

jakob commented Nov 16, 2015

Okay, so I found out that the status "Negotiating SSL..." is a bit inaccurate. This is the first status that the PostgreSQL client library libpq reports after opening the connection. What Postico really does is wait for the SSH server to open the connection to the PostgreSQL server.

Anyway, I've been looking through the connection loop and I couldn't find anything the allocates memory while waiting for the connection. I also tried reproducing your issue by connecting to an SSH server and then connecting to a non-existing address like "test.local". However, Postico did not consume any memory in my testing.

Anyway, if a host is not resolvable, usually the SSH server should return an error after a short time. Because of the hang I assumed that a firewall was involved -- they are often configured in a way to silently swallow packets. It seems that Docker does something similar, but since I have no experience with Docker I really don't know what is going on.

I'll try some other things to reproduce your issue, but for now I really don't know how to find out where that memory leak is coming from. (Unless you happen to know your way around Instruments and the allocation tool...)

@jakob jakob added the bug label Dec 4, 2015

@ddwyer

This comment has been minimized.

Show comment
Hide comment
@ddwyer

ddwyer Nov 15, 2016

Hi Jakob,
i have installed Postico on my mac and "negotiating ssl..." get stuck , i am experiencing hig CPU usage . 99%

postico cpu

ddwyer commented Nov 15, 2016

Hi Jakob,
i have installed Postico on my mac and "negotiating ssl..." get stuck , i am experiencing hig CPU usage . 99%

postico cpu

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Nov 15, 2016

Owner

Can you click that "i" button in Activity monitor, then sample the process and email me the sample, so I can see what Postico is using all the CPU for?

Owner

jakob commented Nov 15, 2016

Can you click that "i" button in Activity monitor, then sample the process and email me the sample, so I can see what Postico is using all the CPU for?

@ddwyer

This comment has been minimized.

Show comment
Hide comment
@ddwyer

ddwyer Nov 15, 2016

Wow Jakob that was super quick .
here is the dump
Sample_of_Postico_from_darren.txt

ddwyer commented Nov 15, 2016

Wow Jakob that was super quick .
here is the dump
Sample_of_Postico_from_darren.txt

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Nov 15, 2016

Owner

Thanks for the trace, @ddwyer!

I was able to find a typo in my code that caused Postico to check for new data from the server in a hot loop (rather than sleeping until data is available). This might also be the reason for the memory problem reported by @sman591 -- I didn't see that typo last time I looked through the code.

Here's a new build that should fix this issue:
https://eggerapps-downloads.s3-eu-west-1.amazonaws.com/postico-1532.zip

Anyway, now Postico should use very little CPU while waiting for data from the SSH server. However, that probably won't help you -- Postico will probably still get stuck at the "Negotiating SSL..." phase.

Here's what's happening according to the sample you sent me: Postico has successfully connected to the SSH server, and it now asked the server to open a channel to the PostgreSQL server. Now it's waiting for a response from the SSH server. Common reasons for getting stuck at this phase are Firewalls or internet filters that possible that the SSH server can't connect to the PostgreSQL server because of a Firewall or an internet filter of some kind.

Owner

jakob commented Nov 15, 2016

Thanks for the trace, @ddwyer!

I was able to find a typo in my code that caused Postico to check for new data from the server in a hot loop (rather than sleeping until data is available). This might also be the reason for the memory problem reported by @sman591 -- I didn't see that typo last time I looked through the code.

Here's a new build that should fix this issue:
https://eggerapps-downloads.s3-eu-west-1.amazonaws.com/postico-1532.zip

Anyway, now Postico should use very little CPU while waiting for data from the SSH server. However, that probably won't help you -- Postico will probably still get stuck at the "Negotiating SSL..." phase.

Here's what's happening according to the sample you sent me: Postico has successfully connected to the SSH server, and it now asked the server to open a channel to the PostgreSQL server. Now it's waiting for a response from the SSH server. Common reasons for getting stuck at this phase are Firewalls or internet filters that possible that the SSH server can't connect to the PostgreSQL server because of a Firewall or an internet filter of some kind.

@ottob

This comment has been minimized.

Show comment
Hide comment
@ottob

ottob Nov 15, 2016

I have also run into this sometimes where it stops on "Working..."

Sample of Postico.txt

ottob commented Nov 15, 2016

I have also run into this sometimes where it stops on "Working..."

Sample of Postico.txt

@ddwyer

This comment has been minimized.

Show comment
Hide comment
@ddwyer

ddwyer Nov 16, 2016

Hi @jakob
you are awesome - bug squashed and Postico is working fine for me .
well done , and congratulations on such fine application.
-- Darren

ddwyer commented Nov 16, 2016

Hi @jakob
you are awesome - bug squashed and Postico is working fine for me .
well done , and congratulations on such fine application.
-- Darren

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Nov 16, 2016

Owner

@ottob Thanks for your sample! That's an interesting bug, but it's unrelated, so I'm opening a new issue for that.

Owner

jakob commented Nov 16, 2016

@ottob Thanks for your sample! That's an interesting bug, but it's unrelated, so I'm opening a new issue for that.

@jakecast

This comment has been minimized.

Show comment
Hide comment
@jakecast

jakecast Dec 20, 2016

I have started to see this error as well. It gets stuck on the 'Negotiating SSL' step while eating up all of my system memory.. I tried the latest build from @jakob on Nov. 15.
I will see if changing the firewall settings or something fixes the issue..

Cheers,
Jake

jakecast commented Dec 20, 2016

I have started to see this error as well. It gets stuck on the 'Negotiating SSL' step while eating up all of my system memory.. I tried the latest build from @jakob on Nov. 15.
I will see if changing the firewall settings or something fixes the issue..

Cheers,
Jake

@davidwparker

This comment has been minimized.

Show comment
Hide comment
@davidwparker

davidwparker Dec 28, 2016

I updated from 0.24 to 1.0.10 and it was fixed, thanks @jakob !

BTW, it still sits on 100% CPU, but it's not leaking memory, so that's not a big deal.

Keep up the good work!

davidwparker commented Dec 28, 2016

I updated from 0.24 to 1.0.10 and it was fixed, thanks @jakob !

BTW, it still sits on 100% CPU, but it's not leaking memory, so that's not a big deal.

Keep up the good work!

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob
Owner

jakob commented Dec 28, 2016

@davidwparker can you try if this build still uses 100% CPU?
https://eggerapps-downloads.s3.amazonaws.com/postico-1541.zip

@jakob jakob added the done label Jan 14, 2017

@jakob

This comment has been minimized.

Show comment
Hide comment
@jakob

jakob Jan 27, 2017

Owner

This should be fixed in the latest version on Postico (1.1), which is available now from my website, and should also be available soon as an update on the Mac App Store.

Owner

jakob commented Jan 27, 2017

This should be fixed in the latest version on Postico (1.1), which is available now from my website, and should also be available soon as an update on the Mac App Store.

@jakob jakob closed this Jan 27, 2017

@ddwyer

This comment has been minimized.

Show comment
Hide comment
@ddwyer

ddwyer Jan 27, 2017

ddwyer commented Jan 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment