Add --append support to Docker process

[rustprooflabs]

Details

Builds on #222 to enable --append mode using osm2pgsql-replication. Hopefully the final part to #211.

⚠️ Bumps minimum osm2pgsql to version ❓ 1.6.1 ❓ - Needs commit: osm2pgsql-dev/osm2pgsql@464f3b2

✔️ No longer blocked

Ran into issue with conninfo string not working in osm2pgsql-replication. See osm2pgsql-dev/osm2pgsql#1643

Additional improvements

• Bump base Docker image to PostGIS 3.2
• Remove python3-psycopg2
• Resolve Post-processing SQL not running - pgosm-meta.sql #235

Steps

• Add --append option
• Add osm2pgsql-replication init to initial run
• Add osm2pgsql-replication update step with new pre/post steps
• Clean out test/debug code
• Documentation!

[rustprooflabs]

With a patch in osm2pgsql-replication (commit rustprooflabs/osm2pgsql@5d8da40) the init command now works with conninfo string and psycopg (v3). For the time being I will keep the Docker image on this branch pinned to my fork of osm2pgsql. Output from running Docker process with --append.

Command ran

docker exec -it     pgosm python3 docker/pgosm_flex.py     --ram=8     --region=north-america/us     --subregion=district-of-columbia   --append --pgosm-date 2022-01-27

truncated output

RUNNING INIT COMMAND:
osm2pgsql-replication init -d $PGOSM_CONN --osm-file /app/output/district-of-columbia-latest.osm.pbf
2022-02-25 17:04:41,633:INFO:pgosm-flex:pgosm_flex:osm2pgsql-replication output:
2022-02-25 17:04:41 [INFO]: Initialised updates for service 'http://download.geofabrik.de/north-america/us/district-of-columbia-updates'.
2022-02-25 17:04:41 [INFO]: Starting at sequence 3228 (2022-01-26 21:21:52+00:00).

2022-02-25 17:04:41,633:INFO:pgosm-flex:pgosm_flex:osm2pgsql-replication init completed.

[rustprooflabs]

Switched Docker osm2pgsql to build from master branch of osm2pgsql. Version 1.6.0 is no longer new enough to support append mode, next tagged version of osm2pgsql will be new minimum requirement.

[rustprooflabs]

Running with --append after loading data now works! Steps for testing

Build the image from this branch, run. Must set max_connections, default of 100 is not enough. Using -c max_connections=300 seems to work.

docker stop pgosm && docker build -t rustprooflabs/pgosm-flex .
docker run --name pgosm -d --rm     -v ~/pgosm-data:/app/output     -v /etc/localtime:/etc/localtime:ro     -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD     -p 5433:5432 -d rustprooflabs/pgosm-flex -c max_connections=300

Run the docker exec for a date with older data. Use `--skip-dump. Run this command a second time to use the replication update method.

docker exec -it     pgosm python3 docker/pgosm_flex.py     --ram=8     --region=north-america/us     --subregion=district-of-columbia --pgosm-date 2022-02-22 --debug --skip-dump --append

From the 2nd docker exec

2022-03-05 11:12:13  Reading input files done in 2s.
2022-03-05 11:12:13    Processed 3055 nodes in 1s - 3k/s
2022-03-05 11:12:13    Processed 639 ways in 0s - 639/s
2022-03-05 11:12:13    Processed 152 relations in 1s - 152/s
2022-03-05 11:12:14  Going over 448 pending ways (using 4 threads)

Left to process: 445...
Left to process: 0...
Left to process: 0.
2022-03-05 11:12:16  Processing 448 pending ways took 2s at a rate of 224.00/s
2022-03-05 11:12:16  Going over 185 pending relations (using 4 threads)

Left to process: 181...
Left to process: 0...
Left to process: 0.
2022-03-05 11:12:18  Processing 185 pending relations took 2s at a rate of 92.50/s

The post-processing step with nested polygons enabled.

2022-03-05 11:12:19,309:INFO:pgosm-flex:db:Finishing replication output: 
 NOTICE:  Populating nested place table
NOTICE:  Calculating nesting of place polygons
NOTICE:  Rows to update: 20
NOTICE:  Updating in batches of 100 rows
NOTICE:  table "places_for_nesting" does not exist, skipping
NOTICE:  table "place_batch" does not exist, skipping
CALL

Default max_connections fails

w/out setting max_connections higher I get a too many connections error.

2022-03-05 11:18:03  ERROR: Connecting to database failed: connection to server at "localhost" (127.0.0.1), port 5432 failed: FATAL:  sorry, too many clients already
.
Traceback (most recent call last):
  File "/usr/local/bin/osm2pgsql-replication", line 535, in <module>
    sys.exit(main())
  File "/usr/local/bin/osm2pgsql-replication", line 528, in main
    ret = args.handler(conn, args)
  File "/usr/local/bin/osm2pgsql-replication", line 399, in update
    subprocess.run(osm2pgsql, check=True)
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['osm2pgsql', '--append', '--slim', '--prefix', 'planet_osm', '--output=flex', '--style=./run.lua', '--slim', '-d', 'postgresql://postgres:mysecretpassword@localhost/pgosm?application_name=pgosm-flex', '-d', 'postgresql://postgres:mysecretpassword@localhost/pgosm?application_name=pgosm-flex', '/tmp/tmpockwgke_/osm2pgsql_diff.osc.gz']' returned non-zero exit status 1.

2022-03-05 11:18:03,967:WARNING:pgosm-flex:pgosm_flex:Failure. Return code: 1
2022-03-05 11:18:03,967:INFO:pgosm-flex:pgosm_flex:Skipping pg_dump
2022-03-05 11:18:03,967:WARNING:pgosm-flex:pgosm_flex:PgOSM Flex completed with errors. Details in output

[rustprooflabs]

Latest commit includes changes I thought I had committed yesterday, sorry. Now my prior comment about it working is accurate

[davoodheydari]

Hi