Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pgBouncer into Docker image #236

Closed
rustprooflabs opened this issue Mar 8, 2022 · 4 comments
Closed

Add pgBouncer into Docker image #236

rustprooflabs opened this issue Mar 8, 2022 · 4 comments
Labels
Docker Docker image and/or related script(s) enhancement New feature or request

Comments

@rustprooflabs
Copy link
Owner

Details

Using osm2pgsql for PgOSM Flex creates a relatively large number of connections to Postgres based on the number of tables created by the Flex output. This was discovered here (#226 (comment)) and answered on the osm2pgsql discussion here. Now documented on osm2pgsql.org: https://osm2pgsql.org/doc/manual.html#number-of-connections

It is unlikely this behavior will change in osm2pgsql so the plan is to bake pgBouncer into the Docker image, example standalone pgbouncer image here. Until this is in place, running --append mode will require upping the max_connections to at least 210, preferably 300 to give headroom.

@rustprooflabs rustprooflabs added enhancement New feature or request Docker Docker image and/or related script(s) labels Mar 8, 2022
@rustprooflabs rustprooflabs added this to the 0.4.7 milestone Mar 8, 2022
@rustprooflabs rustprooflabs removed this from the 0.4.7 milestone Apr 10, 2022
@rustprooflabs
Copy link
Owner Author

It's unlikely at this point that PgBouncer will be directly added into this project's Docker image. First challenge is that as of now, pgbouncer does not actually provide any benefit. See osm2pgsql-dev/osm2pgsql#1650 (reply in thread).

Further, putting pgbouncer into the docker image means it needs to be maintained then forever, and I don't see the great benefit to adding that overhead to this project at this time.

@rustprooflabs
Copy link
Owner Author

If osm2pgsql changes their connection handling behavior to not hold onto idle connections this can be reconsidered. Though, if they do that, the biggest portion of the problem this was trying to solve would be removed too... either way: closed

@rustprooflabs
Copy link
Owner Author

rustprooflabs commented Oct 14, 2023

It looks like using pgbouncer in transaction mode might be an option again. Updated discussion on osm2pgsql. The PR was merged in pgbouncer recently, released in not in tagged version yet v1.21.0 "The one with prepared statements".

@rustprooflabs rustprooflabs reopened this Oct 14, 2023
@rustprooflabs rustprooflabs modified the milestone: 0.10.3 Oct 22, 2023
@rustprooflabs
Copy link
Owner Author

Once again, closing... Recent work in osm2pgsql to reduce idle connections has removed my reasoning to work further on this. See #376 for details. The initial import w/ default layerset dropped from 43 connections down to 10. Replication updates dropped from 207 connections down to 17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docker Docker image and/or related script(s) enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant