Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opening connections to pgcat is very slow #616

Closed
JelteF opened this issue Oct 12, 2023 · 5 comments
Closed

Opening connections to pgcat is very slow #616

JelteF opened this issue Oct 12, 2023 · 5 comments

Comments

@JelteF
Copy link

JelteF commented Oct 12, 2023

Describe the bug
I'm running the following command on pgcat.minimal.toml it takes 5 seconds before any queries are being sent by pgbench, indicating that it takes 5 seconds to open 100 connections to localhost. This scales linearly and it takes 50 seconds to open 1000 connections.

❯ pgbench -P 1  -r -h 127.0.0.1 -T 100 --select-only -p 6434 --protocol simple -c 100
pgbench (15.3, server 12.16 (Ubuntu 12.16-1.pgdg22.04+1))
starting vacuum...end.
progress: 5.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 6.0 s, 21041.9 tps, lat 4.708 ms stddev 1.672, 0 failed
@levkk
Copy link
Contributor

levkk commented Dec 11, 2023

I think we are victims of TCP_NODELAY (https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle-quickack-best-practices/). Disabling it should improve this for localhost tests especially where the buffering the socket is actually harmful.

@JelteF
Copy link
Author

JelteF commented Feb 1, 2024

FYI pgbouncer uses TCP_NODELAY. I'm not sure if you meant disabling NOdelay or disabling THE delay.

https://github.com/pgbouncer/pgbouncer/blob/e6ce619785c93392794976c1e936e8c9f589a5ad/src/util.c#L229

@levkk
Copy link
Contributor

levkk commented Feb 1, 2024

Disable the delay. Enable TCP_NODELAY.

@GEverding
Copy link

I'm happy to give this a test on my workload.

zsand added a commit to zsand/pgcat that referenced this issue May 25, 2024
This commit adds the TCP_NODELAY option to the socket configuration in
`configure_socket` function. Without this option, we observed significant
performance issues when executing SELECT queries with large responses.

Before the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.368 ms
Time: 41.364 ms

After the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.332 ms
Time: 1.528 ms

By setting TCP_NODELAY, we eliminate the Nagle's algorithm delay, which
results in a substantial improvement in response times for large queries.

This problem was discussed in postgresml#616.
@zsand
Copy link
Contributor

zsand commented May 26, 2024

We've encountered similar performance issues with SELECT queries returning large results. Applying the fix from this pull request #749 significantly improved our performance.

levkk pushed a commit that referenced this issue May 26, 2024
…ies (#749)

This commit adds the TCP_NODELAY option to the socket configuration in
`configure_socket` function. Without this option, we observed significant
performance issues when executing SELECT queries with large responses.

Before the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.368 ms
Time: 41.364 ms

After the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.332 ms
Time: 1.528 ms

By setting TCP_NODELAY, we eliminate the Nagle's algorithm delay, which
results in a substantial improvement in response times for large queries.

This problem was discussed in #616.
@levkk levkk closed this as completed May 26, 2024
AndrewJackson2020 pushed a commit to AndrewJackson2020/pgcat that referenced this issue Sep 3, 2024
…ies (postgresml#749)

This commit adds the TCP_NODELAY option to the socket configuration in
`configure_socket` function. Without this option, we observed significant
performance issues when executing SELECT queries with large responses.

Before the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.368 ms
Time: 41.364 ms

After the fix:
postgres=> SELECT repeat('a', 1); SELECT repeat('a', 8153);
Time: 1.332 ms
Time: 1.528 ms

By setting TCP_NODELAY, we eliminate the Nagle's algorithm delay, which
results in a substantial improvement in response times for large queries.

This problem was discussed in postgresml#616.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants