diff --git a/Dockerfile b/Dockerfile index ebe98af..832fcce 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,6 +5,7 @@ LABEL maintainer="PgOSM Flex - https://github.com/rustprooflabs/pgosm-flex" ARG OSM2PGSQL_BRANCH=master RUN apt-get update \ + && apt-get upgrade -y \ && apt-get install -y --no-install-recommends \ sqitch wget ca-certificates \ git make cmake g++ \ diff --git a/LICENSE b/LICENSE index 8d86d05..7d3781e 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2020-2022 Ryan Lambert +Copyright (c) 2020-2023 Ryan Lambert Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/Makefile b/Makefile index cab12e3..8a085e4 100644 --- a/Makefile +++ b/Makefile @@ -83,7 +83,8 @@ docker-exec-default: build-run-docker --layerset=everything \ --ram=$(RAM) \ --region=north-america/us \ - --subregion=district-of-columbia + --subregion=district-of-columbia \ + --pg-dump # pg_dump is not part of the default. Added here to ensure this usage is tested .PHONE: docker-exec-input-file @@ -110,7 +111,7 @@ docker-exec-input-file: build-run-docker --layerset=minimal \ --ram=$(RAM) \ --input-file=/app/output/$(INPUT_FILE_NAME) \ - --data-only --skip-dump --skip-nested # Make this test run faster + --data-only --skip-nested # Make this test run faster @@ -149,7 +150,7 @@ docker-exec-replication-w-input-file: build-run-docker --ram=$(RAM) \ --replication \ --input-file=/app/output/$(INPUT_FILE_NAME) \ - --data-only --skip-dump --skip-nested # Make this test run faster + --data-only --skip-nested # Make this test run faster .PHONE: docker-exec-region @@ -179,7 +180,7 @@ docker-exec-region: build-run-docker --layerset=minimal \ --ram=$(RAM) \ --region=$(REGION_FILE_NAME) \ - --data-only --skip-dump --skip-nested # Make this test run faster + --data-only --skip-nested # Make this test run faster .PHONY: unit-tests diff --git a/README.md b/README.md index 7f55c22..65e352f 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ PgOSM Flex provides high quality OpenStreetMap datasets in PostGIS using the This project provides a curated set of Lua and SQL scripts to clean and organize the most commonly used OpenStreetMap data, such as roads, buildings, and points of interest (POIs). -The easiest way to use PgOSM Flex is via [the Docker image](docs/DOCKER-RUN.md). -For ultimate control and customization, -there are [instructions for installing and running manually](docs/MANUAL-STEPS-RUN.md). +The recommended way to use PgOSM Flex is via the PgOSM Docker image +[hosted on Docker Hub](https://hub.docker.com/repository/docker/rustprooflabs/pgosm-flex). +Basic usage instructions are included in this README.md file, full Docker +usage instructions are available in [docs/DOCKER-RUN.md](docs/DOCKER-RUN.md). ## Project decisions @@ -17,13 +18,14 @@ A few decisions made in this project: * ID column is `osm_id` * Geometry stored in SRID 3857 (customizable) * Geometry column named `geom` -* Default to same units as OpenStreetMap (e.g. km/hr, meters) -* Data not deemed worthy of a dedicated column goes in side table `osm.tags`. Raw key/value data stored in `JSONB` column +* Defaults to same units as OpenStreetMap (e.g. km/hr, meters) +* Data not included in a dedicated column goes into the `osm.tags` table's `JSONB` column * Points, Lines, and Polygons are not mixed in a single table * Tracks latest Postgres, PostGIS, and osm2pgsql versions This project's approach is to do as much processing in the Lua styles -passed along to osm2pgsql, with post-processing steps creating indexes, constraints and comments. +passed along to osm2pgsql, with post-processing steps creating indexes, +constraints and comments. @@ -66,9 +68,9 @@ The PBF/MD5 source files are archived by date on your local storage with the ability to easily reload them at a later date. -### Basic Docker usage +### Docker usage -This section outlines the basic operations for using Docker to run PgOSM-Flex. +This section outlines a typical import using Docker to run PgOSM-Flex. See [the full Docker instructions in docs/DOCKER-RUN.md](docs/DOCKER-RUN.md). Create directory for the `.osm.pbf` file, output `.sql` file, log output, and @@ -100,7 +102,7 @@ docker run --name pgosm -d --rm \ ``` Use `docker exec` to run the processing for the Washington D.C subregion. -This example uses three (3) parameters to specify the totaol system RAM (8 GB) +This example uses three (3) parameters to specify the total system RAM (8 GB) along with a region/subregion. * Total RAM for osm2pgsql, Postgres and OS (`8`) @@ -127,14 +129,29 @@ it takes to download the 17 MB PBF file + ~ 1 minute processing. ### After processing -The `~/pgosm-data` directory has three (3) files from a single run. +The processed OpenStreetMap data is also available in the Docker container on port `5433`. +You can connect and query directly in the Docker container. + +```bash +psql -h localhost -p 5433 -d pgosm -U postgres -c "SELECT COUNT(*) FROM osm.road_line;" + +┌───────┐ +│ count │ +╞═══════╡ +│ 39865 │ +└───────┘ +``` + + +The `~/pgosm-data` directory has two (2) files from a typical single run. The PBF file and its MD5 checksum have been renamed with the date in the filename. This enables loading the file downloaded today again in the future, either with the same version of PgOSM Flex or the latest version. The `docker exec` command uses the `PGOSM_DATE` environment variable to load these historic files. -The output `.sql` is also saved in the `~/pgosm-data` directory. +If the optional `--pg-dump` option is used, the output `.sql` is also saved in +the `~/pgosm-data` directory. ```bash @@ -143,10 +160,8 @@ ls -alh ~/pgosm-data/ -rw-r--r-- 1 root root 17M Nov 2 19:57 district-of-columbia-2021-11-03.osm.pbf -rw-r--r-- 1 root root 70 Nov 2 19:59 district-of-columbia-2021-11-03.osm.pbf.md5 -rw-r--r-- 1 root root 156M Nov 3 19:10 pgosm-flex-north-america-us-district-of-columbia-default-2021-11-03.sql - ``` - This `.sql` file can be loaded into a PostGIS enabled database. The following example creates an empty `myosm` database to load the processed OpenStreetMap data into. @@ -160,23 +175,7 @@ psql -d myosm \ ``` -The processed OpenStreetMap data is also available in the Docker container on port `5433`. -You can connect and query directly in the Docker container. - -```bash -psql -h localhost -p 5433 -d pgosm -U postgres -c "SELECT COUNT(*) FROM osm.road_line;" - -┌───────┐ -│ count │ -╞═══════╡ -│ 39865 │ -└───────┘ -``` - - - -See [more in docs/DOCKER-RUN.md](docs/DOCKER-RUN.md) about other ways to customize -how PgOSM Flex runs. +> See [more in docs/DOCKER-RUN.md](docs/DOCKER-RUN.md) about other ways to customize how PgOSM Flex runs. ## Layer Sets @@ -191,12 +190,11 @@ See [docs/LAYERSETS.md](docs/LAYERSETS.md) for details. If you use QGIS to visualize OpenStreetMap, there are a few basic styles using the `public.layer_styles` table created by QGIS. +This data is loaded by default and can be excluded with `--data-only`. See [the QGIS Style README.md](https://github.com/rustprooflabs/pgosm-flex/blob/main/db/qgis-style/README.md) for more information. -This data is loaded by default and can be excluded with `--data-only`. - ## Explore data loaded @@ -264,8 +262,8 @@ For example queries with data loaded by PgOSM-Flex see ## Points of Interest (POIs) - -Loads an range of tags into a materialized view (`osm.poi_all`) for easy searching POIs. +PgOSM Flex loads an range of tags into a materialized view (`osm.poi_all`) for +easily searching POIs. Line and polygon data is forced to point geometry using `ST_Centroid()`. This layer duplicates a bunch of other more specific layers (shop, amenity, etc.) to provide a single place for simplified POI searches. @@ -330,10 +328,10 @@ SELECT geom_type, COUNT(*) ## One table to rule them all From the perspective of database design, the `osm.unitable` option is the **worst**! -This violates all sorts of best practices established in this project +This table violates all sorts of best practices established in this project by shoving all features into a single unstructured table. -> This style included in PgOSM-Flex is intended to be used for troubleshooting and quality control. It is not intended to be used for real production workloads! This table is helpful for exploring the full data set when you don't really know what you are looking for, but you know **where** you are looking. +> This style included in PgOSM Flex is intended to be used for troubleshooting and quality control. It is not intended to be used for real production workloads! This table is helpful for exploring the full data set when you don't really know what you are looking for, but you know **where** you are looking. Unitable is loaded with the `everything` layerset. Feel free to create your own customized layerset if needed. @@ -355,24 +353,19 @@ docker exec -it \ ## JSONB support -PgOSM-Flex uses `JSONB` in Postgres to store the raw OpenSteetMap +PgOSM-Flex uses `JSONB` in Postgres to store the raw OpenStreetMap key/value data (`tags` column) and relation members (`member_ids`). +The `tags` column only exists in the `osm.tags` and `osm.unitable` tables. +The `member_ids` column is included in: -Current `JSONB` columns: - -* `osm.tags.tags` -* `osm.unitable.tags` -* `osm.place_polygon.member_ids` -* `osm.vplace_polygon.member_ids` -* `osm.poi_polygon.member_ids` - - - -## On-server import +* `osm.place_polygon` +* `osm.poi_polygon` +* `osm.public_transport_line` +* `osm.public_transport_polygon` +* `osm.road_line` +* `osm.road_major` +* `osm.road_polygon` -Don't want to use the Docker process? -See [docs/MANUAL-STEPS-RUN.md](docs/MANUAL-STEPS-RUN.md) for prereqs and steps -for running without Docker. diff --git a/docker/db.py b/docker/db.py index 8f6dfec..c67ac93 100644 --- a/docker/db.py +++ b/docker/db.py @@ -181,25 +181,33 @@ def pg_isready(): return True -def prepare_pgosm_db(data_only, db_path, replication): +def prepare_pgosm_db(data_only, db_path, import_mode): """Runs through series of steps to prepare database for PgOSM. Parameters -------------------------- data_only : bool db_path : str - replication : bool + import_mode : import_mode.ImportMode """ - if pg_conn_parts()['pg_host'] == 'localhost': + drop_it = True LOGGER.debug('Running standard database prep for in-Docker operation. Includes DROP/CREATE DATABASE') - if replication: - LOGGER.debug('Skipping DB drop b/c of append (osm2pgsql-replication) mode') - else: - LOGGER.debug('Dropping database') + LOGGER.debug(f'import_mode: {import_mode}') + if import_mode.slim_no_drop: + if not import_mode.append_first_run: + drop_it = False + if import_mode.replication_update: + drop_it = False + + if drop_it: + LOGGER.debug('Dropping local database if exists') drop_pgosm_db() + else: + LOGGER.debug('Not dropping local DB. This is expected with subsequent import via --replication OR --update=append.') create_pgosm_db() + else: LOGGER.info('Using external database. Ensure the target database is setup properly with proper permissions.') @@ -214,7 +222,7 @@ def prepare_pgosm_db(data_only, db_path, replication): def pg_version_check(): - """Checks Postgres machine-readible server_version_num. + """Checks Postgres machine-readable server_version_num. Sends to logs and returns value. diff --git a/docker/import_mode.py b/docker/import_mode.py new file mode 100644 index 0000000..48be703 --- /dev/null +++ b/docker/import_mode.py @@ -0,0 +1,82 @@ +"""Import Mode provides class to ease logic related to various import modes. +""" +import logging + + +class ImportMode(): + """Determines logical variables used to control program flow. + + WARNING: The values for `append_first_run` and `replication_update` + are used to determine when to drop the local DB. Be careful with any + changes to these values. + """ + def __init__(self, replication, replication_update, update): + """Computes two variables, slim_no_drop and append_first_run + based on inputs. + + Parameters + -------------------------- + replication : bool + replication_update : bool + update : str or None + Valid options are 'create' or 'append', lining up with osm2pgsql's + `--create` and `--append` modes. + """ + self.logger = logging.getLogger('pgosm-flex') + self.replication = replication + self.replication_update = replication_update + + # The input via click should enforce this, still worth checking here + valid_update_options = ['append', 'create', None] + + if update not in valid_update_options: + raise ValueError(f'Invalid option for --update. Valid options: {valid_update_options}') + + self.update = update + self.set_slim_no_drop() + self.set_append_first_run() + self.set_run_post_sql() + + + def set_append_first_run(self): + """Uses `replication_update` and `update` to determine value for + `self.append_first_run` + """ + if self.replication_update: + self.append_first_run = False + else: + self.append_first_run = True + + if self.update is not None: + if self.update == 'create': + self.append_first_run = True + else: + self.append_first_run = False + + def set_slim_no_drop(self): + """Uses `replication` and `update` to determine value for + `self.slim_no_drop` + """ + self.slim_no_drop = False + + if self.replication: + self.slim_no_drop = True + + if self.update is not None: + self.slim_no_drop = True + + def set_run_post_sql(self): + """Uses `update` value to determine value for + `self.run_post_sql`. This value determines if the post-processing SQL + should be executed. + + Note: Not checking replication/replication_update because subsequent + imports use osm2pgsql-replication, which does not attempt to run + the post-processing SQL scripts. + """ + self.run_post_sql = True + + if self.update is not None: + if self.update == 'append': + self.run_post_sql = False + diff --git a/docker/osm2pgsql_recommendation.py b/docker/osm2pgsql_recommendation.py index 5dcd176..6a69ae2 100644 --- a/docker/osm2pgsql_recommendation.py +++ b/docker/osm2pgsql_recommendation.py @@ -10,7 +10,7 @@ LOGGER = logging.getLogger('pgosm-flex') -def osm2pgsql_recommendation(ram, pbf_filename, out_path, replication): +def osm2pgsql_recommendation(ram, pbf_filename, out_path, import_mode): """Returns recommended osm2pgsql command. Recommendation from Python project. @@ -22,7 +22,7 @@ def osm2pgsql_recommendation(ram, pbf_filename, out_path, replication): Total system RAM available in GB pbf_filename : str out_path : str - replication : boolean + import_mode : dict Returns ---------------------- @@ -40,12 +40,12 @@ def osm2pgsql_recommendation(ram, pbf_filename, out_path, replication): osm2pgsql_cmd = get_recommended_script(system_ram_gb, osm_pbf_gb, - replication, + import_mode, pbf_file, out_path) return osm2pgsql_cmd -def get_recommended_script(system_ram_gb, osm_pbf_gb, replication, pbf_filename, +def get_recommended_script(system_ram_gb, osm_pbf_gb, import_mode, pbf_filename, output_path): """Generates recommended osm2pgsql command from osm2pgsql-tuner. @@ -53,7 +53,7 @@ def get_recommended_script(system_ram_gb, osm_pbf_gb, replication, pbf_filename, ------------------------------- system_ram_gb : float osm_pbf_gb : float - replication : bool + import_mode : dict pbf_filename : str Can be filename or absolute path. output_path : str @@ -70,11 +70,10 @@ def get_recommended_script(system_ram_gb, osm_pbf_gb, replication, pbf_filename, """ LOGGER.debug('Generating recommended osm2pgsql command') - # This function call will change as this is implemented - # https://github.com/rustprooflabs/osm2pgsql-tuner/issues/24 rec = tuner.recommendation(system_ram_gb=system_ram_gb, osm_pbf_gb=osm_pbf_gb, - append=replication, + slim_no_drop=import_mode.slim_no_drop, + append_first_run=import_mode.append_first_run, ssd=True) osm2pgsql_cmd = rec.get_osm2pgsql_command(out_format='api', diff --git a/docker/pgosm_flex.py b/docker/pgosm_flex.py index 5a130a0..8073850 100644 --- a/docker/pgosm_flex.py +++ b/docker/pgosm_flex.py @@ -19,6 +19,7 @@ import db import geofabrik import helpers +from import_mode import ImportMode @click.command() @@ -31,10 +32,6 @@ @click.option('--subregion', required=False, help='Sub-region name matching the filename for data sourced from Geofabrik. e.g. district-of-columbia') # Remainder of options in alphabetical order -@click.option('--append', - default=False, - is_flag=True, - help='Deprecated! Use --replication. --append will be removed in 0.7.0 Details: https://github.com/rustprooflabs/pgosm-flex/issues/275') @click.option('--data-only', default=False, is_flag=True, @@ -53,6 +50,8 @@ @click.option('--language', default=None, envvar="PGOSM_LANGUAGE", help="Set default language in loaded OpenStreetMap data when available. e.g. 'en' or 'kn'.") +@click.option('--pg-dump', default=False, is_flag=True, + help='Uses pg_dump after processing is completed to enable easily load OpenStreetMap data into a different database') @click.option('--pgosm-date', required=False, default=helpers.get_today(), envvar="PGOSM_DATE", @@ -64,8 +63,6 @@ @click.option('--schema-name', required=False, default='osm', help="Change the final schema name, defaults to 'osm'.") -@click.option('--skip-dump', default=False, is_flag=True, - help='Skips the final pg_dump at the end. Useful for local testing when not loading into more permanent instance.') @click.option('--skip-nested', default=False, is_flag=True, @@ -75,10 +72,13 @@ help="SRID for data loaded by osm2pgsql to PostGIS. Defaults to 3857") @click.option('--sp-gist', default=False, is_flag=True, help='When set, builds SP-GIST indexes on geom column instead of the default GIST indexes.') -def run_pgosm_flex(ram, region, subregion, append, data_only, debug, - input_file, layerset, layerset_path, language, pgosm_date, - replication, - schema_name, skip_dump, skip_nested, srid, sp_gist): +@click.option('--update', default=None, + type=click.Choice(['append', 'create'], case_sensitive=True), + help='EXPERIMENTAL - Wrap around osm2pgsql create v. append modes, without using osm2pgsql-replication.') +def run_pgosm_flex(ram, region, subregion, data_only, debug, + input_file, layerset, layerset_path, language, pg_dump, + pgosm_date, replication, schema_name, skip_nested, + srid, sp_gist, update): """Run PgOSM Flex within Docker to automate osm2pgsql flex processing. """ paths = get_paths() @@ -86,53 +86,62 @@ def run_pgosm_flex(ram, region, subregion, append, data_only, debug, logger = logging.getLogger('pgosm-flex') logger.info('PgOSM Flex starting...') - # Starting to address issues identified in - # https://github.com/rustprooflabs/pgosm-flex/issues/275 - if append: - logger.warning('--append mode is deprecated. Use --replication instead.') - replication = True - - validate_region_inputs(region, subregion, input_file) - + # Input validation if schema_name != 'osm' and replication: err_msg = 'Replication mode with custom schema name currently not supported' logger.error(err_msg) sys.exit(err_msg) + if replication and (update is not None): + err_msg = 'The --replication and --update features are mutually exclusive. Use one or the other.' + logger.error(err_msg) + sys.exit(err_msg) + # End of input validation + + validate_region_inputs(region, subregion, input_file) if region is None and input_file: region = input_file helpers.set_env_vars(region, subregion, srid, language, pgosm_date, layerset, layerset_path, sp_gist) db.wait_for_postgres() - db.prepare_pgosm_db(data_only=data_only, - db_path=paths['db_path'], - replication=replication) if replication: replication_update = check_replication_exists() else: replication_update = False - if replication_update: + logger.debug(f'UPDATE setting: {update}') + # Warning: Reusing the module's name here as import_mode... + import_mode = ImportMode(replication=replication, + replication_update=replication_update, + update=update) + logger.debug(f'IMPORT MODE {import_mode}') + + db.prepare_pgosm_db(data_only=data_only, + db_path=paths['db_path'], + import_mode=import_mode) + + if import_mode.replication_update: + logger.info('Running osm2pgsql-replication in update mode') logger.warning('Replication mode is Experimental (getting closer!)') success = run_replication_update(skip_nested=skip_nested, flex_path=paths['flex_path']) else: - logger.info('Running osm2pgsql without osm2pgsql-replication') + logger.info('Running osm2pgsql') success = run_osm2pgsql_standard(input_file=input_file, out_path=paths['out_path'], flex_path=paths['flex_path'], ram=ram, skip_nested=skip_nested, - replication=replication) + import_mode=import_mode) if schema_name != 'osm': db.rename_schema(schema_name) dump_database(input_file=input_file, out_path=paths['out_path'], - skip_dump=skip_dump, + pg_dump=pg_dump, data_only=data_only, schema_name=schema_name) @@ -143,7 +152,7 @@ def run_pgosm_flex(ram, region, subregion, append, data_only, debug, def run_osm2pgsql_standard(input_file, out_path, flex_path, ram, skip_nested, - replication): + import_mode): """Runs standard osm2pgsql command and optionally inits for replication (osm2pgsql-replication) mode. @@ -154,7 +163,7 @@ def run_osm2pgsql_standard(input_file, out_path, flex_path, ram, skip_nested, flex_path : str ram : float skip_nested : boolean - replication : boolean + import_mode : import_mode.ImportMode Returns --------------------------- @@ -172,7 +181,7 @@ def run_osm2pgsql_standard(input_file, out_path, flex_path, ram, skip_nested, osm2pgsql_command = rec.osm2pgsql_recommendation(ram=ram, pbf_filename=pbf_filename, out_path=out_path, - replication=replication) + import_mode=import_mode) run_osm2pgsql(osm2pgsql_command=osm2pgsql_command, flex_path=flex_path) @@ -180,9 +189,10 @@ def run_osm2pgsql_standard(input_file, out_path, flex_path, ram, skip_nested, skip_nested = check_layerset_places(flex_path) post_processing = run_post_processing(flex_path=flex_path, - skip_nested=skip_nested) + skip_nested=skip_nested, + import_mode=import_mode) - if replication: + if import_mode.replication: run_osm2pgsql_replication_init(pbf_path=out_path, pbf_filename=pbf_filename) else: @@ -316,7 +326,7 @@ def get_export_filename(input_file): ---------------------- filename : str """ - # region is always set internally, even with --input-file and no --region + # always set internally, even with --input-file and no --region region = os.environ.get('PGOSM_REGION').replace('/', '-') subregion = os.environ.get('PGOSM_SUBREGION') layerset = os.environ.get('PGOSM_LAYERSET') @@ -415,7 +425,7 @@ def check_layerset_places(flex_path): return True -def run_post_processing(flex_path, skip_nested): +def run_post_processing(flex_path, skip_nested, import_mode): """Runs steps following osm2pgsql import. Post-processing SQL scripts and (optionally) calculate nested admin polgyons @@ -424,13 +434,20 @@ def run_post_processing(flex_path, skip_nested): ---------------------- flex_path : str skip_nested : bool + import_mode : import_mode.ImportMode Returns ---------------------- status : bool """ - post_processing_sql = db.pgosm_after_import(flex_path) logger = logging.getLogger('pgosm-flex') + + if not import_mode.run_post_sql: + logger.info('Running with --update append: Skipping post-processing SQL') + return True + + post_processing_sql = db.pgosm_after_import(flex_path) + if skip_nested: logger.info('Skipping calculating nested polygons') else: @@ -442,26 +459,26 @@ def run_post_processing(flex_path, skip_nested): return True -def dump_database(input_file, out_path, skip_dump, data_only, schema_name): +def dump_database(input_file, out_path, pg_dump, data_only, schema_name): """Runs pg_dump when necessary to export the processed OpenStreetMap data. Parameters ----------------------- input_file : str out_path : str - skip_dump : bool + pg_dump : bool data_only : bool schema_name : str """ - if skip_dump: - logging.getLogger('pgosm-flex').info('Skipping pg_dump') - else: + if pg_dump: export_filename = get_export_filename(input_file) export_path = get_export_full_path(out_path, export_filename) db.run_pg_dump(export_path=export_path, data_only=data_only, schema_name=schema_name) + else: + logging.getLogger('pgosm-flex').info('Skipping pg_dump') def check_replication_exists(): diff --git a/docker/tests/test_geofabrik.py b/docker/tests/test_geofabrik.py index 89fc6ef..5b9783e 100644 --- a/docker/tests/test_geofabrik.py +++ b/docker/tests/test_geofabrik.py @@ -17,7 +17,8 @@ def setUp(self): language=None, pgosm_date=PGOSM_DATE, layerset=LAYERSET, - layerset_path=None) + layerset_path=None, + sp_gist=False) def tearDown(self): @@ -37,7 +38,8 @@ def test_get_region_filename_returns_region_when_subregion_None(self): language=None, pgosm_date=PGOSM_DATE, layerset=LAYERSET, - layerset_path=None) + layerset_path=None, + sp_gist=False) result = geofabrik.get_region_filename() expected = f'{REGION_US}-latest.osm.pbf' diff --git a/docker/tests/test_import_mode.py b/docker/tests/test_import_mode.py new file mode 100644 index 0000000..06a64ec --- /dev/null +++ b/docker/tests/test_import_mode.py @@ -0,0 +1,98 @@ +""" Unit tests to cover the import_mode module.""" +import os +import unittest + +import import_mode + + +class ImportModeTests(unittest.TestCase): + + def test_import_mode_with_no_replication_or_update_returns_append_first_run_True(self): + replication = False + replication_update = False + update = None + + expected = True + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.append_first_run + self.assertEqual(expected, actual) + + def test_import_mode_with_replication_update_returns_append_first_run_False(self): + replication = True + replication_update = True + update = None + + expected = False + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.append_first_run + self.assertEqual(expected, actual) + + def test_import_mode_with_update_eq_create_returns_True(self): + replication = True + replication_update = True + update = 'create' + + expected = True + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.append_first_run + self.assertEqual(expected, actual) + + def test_import_mode_with_update_eq_append_returns_False(self): + replication = True + replication_update = True + update = 'append' + + expected = False + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.append_first_run + self.assertEqual(expected, actual) + + def test_import_mode_invalid_update_value_raises_ValueError(self): + replication = False + replication_update = False + update = False # Boolean is invalid for this + + with self.assertRaises(ValueError): + import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + + + def test_import_mode_with_update_create_sets_value_run_post_sql_True(self): + replication = False + replication_update = False + update = 'create' + expected = True + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.run_post_sql + self.assertEqual(expected, actual) + + + def test_import_mode_with_update_append_sets_value_run_post_sql_False(self): + replication = False + replication_update = False + update = 'append' + expected = False + im = import_mode.ImportMode(replication=replication, + replication_update=replication_update, + update=update) + + actual = im.run_post_sql + self.assertEqual(expected, actual) + diff --git a/docker/tests/test_osm2pgsql_recommendation.py b/docker/tests/test_osm2pgsql_recommendation.py index af4e073..6751c79 100644 --- a/docker/tests/test_osm2pgsql_recommendation.py +++ b/docker/tests/test_osm2pgsql_recommendation.py @@ -3,35 +3,101 @@ import unittest import osm2pgsql_recommendation +from import_mode import ImportMode class Osm2pgsqlRecommendationTests(unittest.TestCase): - def test_get_recommended_script_returns_str(self): + def setUp(self): + self.maxDiff = None + """Overriding default to enable seeing the very long diffs generated by + errors in this test suite. + """ + + def test_get_recommended_script_returns_type_str(self): expected = str system_ram_gb = 2 osm_pbf_gb = 10 - append = False + im = ImportMode(replication=False, + replication_update=False, + update=None) pbf_filename = 'This-is-a-test.osm.pbf' output_path = 'this-is-a-test' - result = osm2pgsql_recommendation.get_recommended_script(system_ram_gb, - osm_pbf_gb, - append, - pbf_filename, - output_path) + result = osm2pgsql_recommendation.get_recommended_script(system_ram_gb=system_ram_gb, + osm_pbf_gb=osm_pbf_gb, + import_mode=im, + pbf_filename=pbf_filename, + output_path=output_path) + actual = type(result) self.assertEqual(expected, actual) - def test_get_recommended_script_returns_expected_str(self): - expected = 'osm2pgsql -d postgresql://postgres:mysecretpassword@localhost:5432/pgosm?application_name=pgosm-flex --cache=0 --slim --drop --flat-nodes=/tmp/nodes --output=flex --style=./run.lua This-is-a-test.osm.pbf' + def test_get_recommended_script_returns_expected_value_simple_example(self): + expected = 'osm2pgsql -d postgresql://postgres:mysecretpassword@localhost:5432/pgosm?application_name=pgosm-flex --cache=0 --slim --drop --flat-nodes=/tmp/nodes --create --output=flex --style=./run.lua This-is-a-test.osm.pbf' system_ram_gb = 2 osm_pbf_gb = 10 - append = False + im = ImportMode(replication=False, + replication_update=False, + update=None) pbf_filename = 'This-is-a-test.osm.pbf' output_path = 'this-is-a-test' - actual = osm2pgsql_recommendation.get_recommended_script(system_ram_gb, - osm_pbf_gb, - append, - pbf_filename, - output_path) + actual = osm2pgsql_recommendation.get_recommended_script(system_ram_gb=system_ram_gb, + osm_pbf_gb=osm_pbf_gb, + import_mode=im, + pbf_filename=pbf_filename, + output_path=output_path) self.assertEqual(expected, actual) + + def test_get_recommended_script_returns_expected_value_update_create(self): + expected = 'osm2pgsql -d postgresql://postgres:mysecretpassword@localhost:5432/pgosm?application_name=pgosm-flex --cache=0 --slim --flat-nodes=/tmp/nodes --create --output=flex --style=./run.lua This-is-a-test.osm.pbf' + system_ram_gb = 2 + osm_pbf_gb = 10 + im = ImportMode(replication=False, + replication_update=False, + update='create') + pbf_filename = 'This-is-a-test.osm.pbf' + output_path = 'this-is-a-test' + actual = osm2pgsql_recommendation.get_recommended_script(system_ram_gb=system_ram_gb, + osm_pbf_gb=osm_pbf_gb, + import_mode=im, + pbf_filename=pbf_filename, + output_path=output_path) + self.assertEqual(expected, actual) + + def test_get_recommended_script_returns_expected_value_update_append(self): + expected = 'osm2pgsql -d postgresql://postgres:mysecretpassword@localhost:5432/pgosm?application_name=pgosm-flex --cache=0 --slim --flat-nodes=/tmp/nodes --append --output=flex --style=./run.lua This-is-a-test.osm.pbf' + system_ram_gb = 2 + osm_pbf_gb = 10 + im = ImportMode(replication=False, + replication_update=False, + update='append') + pbf_filename = 'This-is-a-test.osm.pbf' + output_path = 'this-is-a-test' + actual = osm2pgsql_recommendation.get_recommended_script(system_ram_gb=system_ram_gb, + osm_pbf_gb=osm_pbf_gb, + import_mode=im, + pbf_filename=pbf_filename, + output_path=output_path) + self.assertEqual(expected, actual) + + def test_get_recommended_script_returns_expected_value_replication_first(self): + """NOTE: This function does not have a counterpart testing when + replication_update = True. This is intentional, + osm2pgsql-tuner is not involved at those steps so no recommended script + is generated. + """ + expected = 'osm2pgsql -d postgresql://postgres:mysecretpassword@localhost:5432/pgosm?application_name=pgosm-flex --cache=0 --slim --flat-nodes=/tmp/nodes --create --output=flex --style=./run.lua This-is-a-test.osm.pbf' + system_ram_gb = 2 + osm_pbf_gb = 10 + im = ImportMode(replication=True, + replication_update=False, + update=None) + pbf_filename = 'This-is-a-test.osm.pbf' + output_path = 'this-is-a-test' + actual = osm2pgsql_recommendation.get_recommended_script(system_ram_gb=system_ram_gb, + osm_pbf_gb=osm_pbf_gb, + import_mode=im, + pbf_filename=pbf_filename, + output_path=output_path) + self.assertEqual(expected, actual) + diff --git a/docker/tests/test_pgosm_flex.py b/docker/tests/test_pgosm_flex.py index d6087bb..73d6e70 100644 --- a/docker/tests/test_pgosm_flex.py +++ b/docker/tests/test_pgosm_flex.py @@ -17,7 +17,8 @@ def setUp(self): language=None, pgosm_date=PGOSM_DATE, layerset=LAYERSET, - layerset_path=None) + layerset_path=None, + sp_gist=False) def tearDown(self): @@ -87,7 +88,8 @@ def test_get_export_filename_region_only(self): language=None, pgosm_date=PGOSM_DATE, layerset=LAYERSET, - layerset_path=None) + layerset_path=None, + sp_gist=False) input_file = None result = pgosm_flex.get_export_filename(input_file) diff --git a/docs/APPEND-MODE.md b/docs/APPEND-MODE.md index 6f6d235..f3873cf 100644 --- a/docs/APPEND-MODE.md +++ b/docs/APPEND-MODE.md @@ -1,85 +1 @@ -# Running osm2pgsql in append mode - ----- - -> The `--append` feature is experimental. - ----- - - -## Using manual steps - -This section documents differences from [MANUAL-STEPS-RUN.md](MANUAL-STEPS-RUN.md) - - -Setup - Need to use Python venv, osmium is a requirement. Something like... - -```bash -python -m venv venv && source venv/bin/activate -cd ~/git/pgosm-flex && pip install -r requirements.txt -``` - - -Run osm2pgsql. Must use `--slim` mode without drop. - - -```bash -cd pgosm-flex/flex-config - -osm2pgsql --output=flex --style=./run.lua \ - --slim \ - -d $PGOSM_CONN \ - ~/pgosm-data/district-of-columbia-latest.osm.pbf -``` - -Run the normal post-processing as you normally would. - - -`osm2pgsql-replication` is bundled with osm2pgsql install. - -https://osm2pgsql.org/doc/manual.html#keeping-the-database-up-to-date-with-osm2pgsql-replication - - -Initialize replication. - - -```bash -osm2pgsql-replication init -d $PGOSM_CONN \ - --osm-file ~/pgosm-data/district-of-columbia-latest.osm.pbf -``` - - - -Refresh the data. First clear out data that might violate foreign keys. Packaged -in convenient procedure. - - -```sql -CALL osm.append_data_start(); -``` - -Update the data. - - -```bash -osm2pgsql-replication update -d $PGOSM_CONN \ - -- \ - --output=flex --style=./run.lua \ - --slim \ - -d $PGOSM_CONN -``` - -Refresh Mat views, rebuilds nested place polygon data. - - -```sql -CALL osm.append_data_finish(); -``` - - -Skip nested: - -```sql -CALL osm.append_data_finish(skip_nested := True); -``` - +The contents of this file have moved to [MANUAL-STEPS-REPLICATION.md](MANUAL-STEPS-REPLICATION.md). \ No newline at end of file diff --git a/docs/DOCKER-BUILD.md b/docs/DOCKER-BUILD.md index 9d104ae..f03eb0f 100644 --- a/docs/DOCKER-BUILD.md +++ b/docs/DOCKER-BUILD.md @@ -74,3 +74,17 @@ Push to Docker Hub. docker push rustprooflabs/pgosm-flex:latest docker push rustprooflabs/pgosm-flex:0.6.2 ``` + + +### Ensure updates + +To be certain the latest images are being used and latest +software is installed, pull the latest PostGIS image and build +the PgOSM Flex image using `--no-cache`. + + +```bash +docker pull postgis/postgis:15-3.3 +docker build --no-cache -t rustprooflabs/pgosm-flex:dev . +``` + diff --git a/docs/DOCKER-RUN.md b/docs/DOCKER-RUN.md index 0e7ca62..73e6b56 100644 --- a/docs/DOCKER-RUN.md +++ b/docs/DOCKER-RUN.md @@ -126,48 +126,51 @@ Usage: pgosm_flex.py [OPTIONS] Run PgOSM Flex within Docker to automate osm2pgsql flex processing. Options: - --ram FLOAT Amount of RAM in GB available on the machine running - the Docker container. This is used to determine the - appropriate osm2pgsql command via osm2pgsql-tuner - recommendation engine. [required] - --region TEXT Region name matching the filename for data sourced - from Geofabrik. e.g. north-america/us. Optional when - --input-file is specified, otherwise required. - --subregion TEXT Sub-region name matching the filename for data sourced - from Geofabrik. e.g. district-of-columbia - --append Deprecated! Use --replication. --append will be - removed in 0.7.0 Details: - https://github.com/rustprooflabs/pgosm-flex/issues/275 - --data-only When set, skips running Sqitch and importing QGIS - Styles. - --debug Enables additional log output - --input-file TEXT Set filename or absolute filepath to input osm.pbf - file. Overrides default file handling, archiving, and - MD5 checksum validation. Filename is assumed under - /app/output unless absolute path is used. - --layerset TEXT Layerset to load. Defines name of included layerset - unless --layerset-path is defined. [required] - --layerset-path TEXT Custom path to load layerset INI from. Custom paths - should be mounted to Docker via docker run -v ... - --language TEXT Set default language in loaded OpenStreetMap data when - available. e.g. 'en' or 'kn'. - --pgosm-date TEXT Date of the data in YYYY-MM-DD format. If today - (default), automatically downloads when files not - found locally. Set to historic date to load locally - archived PBF/MD5 file, will fail if both files do not - exist. - --replication EXPERIMENTAL - Replication mode enables updates via - osm2pgsql-replication. - --schema-name TEXT Change the final schema name, defaults to 'osm'. - --skip-dump Skips the final pg_dump at the end. Useful for local - testing when not loading into more permanent instance. - --skip-nested When set, skips calculating nested admin polygons. Can - be time consuming on large regions. - --srid TEXT SRID for data loaded by osm2pgsql to PostGIS. Defaults - to 3857 - --sp-gist When set, builds SP-GIST indexes on geom column - instead of the default GIST indexes. - --help Show this message and exit. + --ram FLOAT Amount of RAM in GB available on the machine + running the Docker container. This is used to + determine the appropriate osm2pgsql command via + osm2pgsql-tuner recommendation engine. [required] + --region TEXT Region name matching the filename for data sourced + from Geofabrik. e.g. north-america/us. Optional + when --input-file is specified, otherwise + required. + --subregion TEXT Sub-region name matching the filename for data + sourced from Geofabrik. e.g. district-of-columbia + --data-only When set, skips running Sqitch and importing QGIS + Styles. + --debug Enables additional log output + --input-file TEXT Set filename or absolute filepath to input osm.pbf + file. Overrides default file handling, archiving, + and MD5 checksum validation. Filename is assumed + under /app/output unless absolute path is used. + --layerset TEXT Layerset to load. Defines name of included + layerset unless --layerset-path is defined. + [required] + --layerset-path TEXT Custom path to load layerset INI from. Custom + paths should be mounted to Docker via docker run + -v ... + --language TEXT Set default language in loaded OpenStreetMap data + when available. e.g. 'en' or 'kn'. + --pg-dump Uses pg_dump after processing is completed to + enable easily load OpenStreetMap data into a + different database + --pgosm-date TEXT Date of the data in YYYY-MM-DD format. If today + (default), automatically downloads when files not + found locally. Set to historic date to load + locally archived PBF/MD5 file, will fail if both + files do not exist. + --replication EXPERIMENTAL - Replication mode enables updates + via osm2pgsql-replication. + --schema-name TEXT Change the final schema name, defaults to 'osm'. + --skip-nested When set, skips calculating nested admin polygons. + Can be time consuming on large regions. + --srid TEXT SRID for data loaded by osm2pgsql to PostGIS. + Defaults to 3857 + --sp-gist When set, builds SP-GIST indexes on geom column + instead of the default GIST indexes. + --update [append|create] EXPERIMENTAL - Wrap around osm2pgsql create v. + append modes, without using osm2pgsql-replication. + --help Show this message and exit. ``` An example of running with many of the current options. @@ -356,7 +359,7 @@ docker run --name pgosm -d --rm \ -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \ -p 5433:5432 \ -d rustprooflabs/pgosm-flex:0.6.3 \ - -c max_connections=300 + -c max_connections=300 ``` > Note: The instructions for `--replication` use a specific tagged version of the PgOSM Flex Docker image. Upgrading PgOSM Flex versions with replication mode is possible with manual DDL scripts. Caution and testing is strongly recommended before proceeding on production. See the release notes, along with the scripts under `pgosm-flex/db/data-migration/`. diff --git a/docs/DUMP-AND-LOAD.md b/docs/DUMP-AND-LOAD.md index a5e682c..a720957 100644 --- a/docs/DUMP-AND-LOAD.md +++ b/docs/DUMP-AND-LOAD.md @@ -1,5 +1,8 @@ # PgOSM Flex: Dump and reload data +> These manual procedures (outside of Docker) are not regularly tested or reviewed. The recommended way to use PgOSM Flex is through the Docker image. The Docker image is capable of renaming the schema and running pg_dump when desired. + +---- To move data loaded on one Postgres instance to another, use `pg_dump`. The import from PBF to PostGIS is far more taxing on resources than general diff --git a/docs/MANUAL-STEPS-REPLICATION.md b/docs/MANUAL-STEPS-REPLICATION.md new file mode 100644 index 0000000..09e5d58 --- /dev/null +++ b/docs/MANUAL-STEPS-REPLICATION.md @@ -0,0 +1,97 @@ +# Manually run osm2pgsql in replication mode + + +> These manual procedures (outside of Docker) are not regularly tested or reviewed. The recommended way to use PgOSM Flex is through the Docker image. The Docker image is capable of using osm2pgsql-replication as described in this document. + +---- + +Needs updating, `--append` is removed, now under `--replication` + +New option to `--update` in create/append mode making osm2pgsql's append +option available. + + +---- + + +> The `--replication` feature is experimental. + +---- + + +## Using manual steps + +This section documents differences from [MANUAL-STEPS-RUN.md](MANUAL-STEPS-RUN.md) + + +Setup - Need to use Python venv, osmium is a requirement. Something like... + +```bash +python -m venv venv && source venv/bin/activate +cd ~/git/pgosm-flex && pip install -r requirements.txt +``` + + +Run osm2pgsql. Must use `--slim` mode without drop. + + +```bash +cd pgosm-flex/flex-config + +osm2pgsql --output=flex --style=./run.lua \ + --slim \ + -d $PGOSM_CONN \ + ~/pgosm-data/district-of-columbia-latest.osm.pbf +``` + +Run the normal post-processing as you normally would. + + +`osm2pgsql-replication` is bundled with osm2pgsql install. + +https://osm2pgsql.org/doc/manual.html#keeping-the-database-up-to-date-with-osm2pgsql-replication + + +Initialize replication. + + +```bash +osm2pgsql-replication init -d $PGOSM_CONN \ + --osm-file ~/pgosm-data/district-of-columbia-latest.osm.pbf +``` + + + +Refresh the data. First clear out data that might violate foreign keys. Packaged +in convenient procedure. + + +```sql +CALL osm.append_data_start(); +``` + +Update the data. + + +```bash +osm2pgsql-replication update -d $PGOSM_CONN \ + -- \ + --output=flex --style=./run.lua \ + --slim \ + -d $PGOSM_CONN +``` + +Refresh Mat views, rebuilds nested place polygon data. + + +```sql +CALL osm.append_data_finish(); +``` + + +Skip nested: + +```sql +CALL osm.append_data_finish(skip_nested := True); +``` + diff --git a/docs/MANUAL-STEPS-RUN.md b/docs/MANUAL-STEPS-RUN.md index c6a74b8..6124838 100644 --- a/docs/MANUAL-STEPS-RUN.md +++ b/docs/MANUAL-STEPS-RUN.md @@ -1,4 +1,8 @@ -# PgOSM Flex Standard Import +# PgOSM Flex Manual Import + +> These manual procedures (outside of Docker) are not regularly tested or reviewed. The recommended way to use PgOSM Flex is through the Docker image. The Docker image is capable of adjusting for regions of any size so this manual process no longer serves a necessary purpose. + +---- These instructions show how to manually run the PgOSM-Flex process. This is the best option for scaling to larger regions (North America, Europe, etc.) diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 32d547f..4b0a5c4 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -7,7 +7,7 @@ server size [outlined in the osm2pgsql manual](https://osm2pgsql.org/doc/manual. ## Versions Tested -Versions used for testing: PgOSM Flex 0.4.7 Docker image, based on the offical +Versions used for testing: PgOSM Flex 0.4.7 Docker image, based on the official PostGIS image with Postgres 14 / PostGIS 3.2. diff --git a/docs/UPDATE-MODE.md b/docs/UPDATE-MODE.md new file mode 100644 index 0000000..921af36 --- /dev/null +++ b/docs/UPDATE-MODE.md @@ -0,0 +1,148 @@ +# PgOSM Flex Update Mode + +Running in experimental Update mode enables using osm2pgsql's `--append` +option. + +> Note: This is **not** the `--append` option that existed in PgOSM Flex 0.6.3 and prior. + + +## Testing steps + +Important -- Needs higher max connections! + + + +```bash +docker stop pgosm && docker build -t rustprooflabs/pgosm-flex . +docker run --name pgosm -d --rm \ + -v ~/pgosm-data:/app/output \ + -v /etc/localtime:/etc/localtime:ro \ + -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \ + -p 5433:5432 -d rustprooflabs/pgosm-flex \ + -c max_connections=300 +``` + +Run fresh import w/ D.C. using `--update create`. This ensures osm2pgsql +uses `--slim` w/out `--drop`. Tested from commit `672d9fd`. + + +```bash +time docker exec -it \ + pgosm python3 docker/pgosm_flex.py \ + --ram=8 \ + --region=north-america/us \ + --subregion=district-of-columbia \ + --skip-nested --skip-dump \ + --update create + +... + +2022-12-27 09:02:37,654:INFO:pgosm-flex:helpers:PgOSM-Flex version: 0.6.3 672d9fd + +... + +real 0m43.904s +user 0m0.020s +sys 0m0.012s +``` + +Run with a second sub-region using `--update append`. + +```bash +time docker exec -it \ + pgosm python3 docker/pgosm_flex.py \ + --ram=8 \ + --region=north-america/us \ + --subregion=new-hampshire \ + --skip-nested --skip-dump \ + --update append + +... + +2022-12-27 10:14:26,792:INFO:pgosm-flex:helpers:2022-12-27 10:14:26 osm2pgsql took 1420s (23m 40s) overall. +2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:osm2pgsql completed +2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:Running with --update append: Skipping post-processing SQL +2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:Skipping pg_dump +2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:PgOSM Flex complete! + +real 23m47.564s +user 0m0.083s +sys 0m0.025s + +``` + +It seems to work, new output at the end: `Skipping post-processing SQL`. + +Verified that both New Hampshire and D.C. regions were loaded in `osm.place_polygon`. + + +## Smaller test + +Put the following into `~/pgosm-data/extracts/colorado-extract.json`: + +```json +{ + "directory": "/home/ryanlambert/pgosm-data/", + "extracts": [ + { + "output": "colorado-boulder-latest.osm.pbf", + "description": "Area extracted around Boulder, Colorado", + "bbox": { + "left": -105.30, + "right": -105.20, + "top": 40.07, + "bottom": 39.98 + } + }, + { + "output": "colorado-longmont-latest.osm.pbf", + "description": "Area extracted around Longmont, Colorado", + "bbox": { + "left": -105.15, + "right": -105.05, + "top": 40.21, + "bottom": 40.12 + } + } + ] +} +``` + + +Create Boulder and Longmont extracts. + +``` +osmium extract -c extracts/colorado-extracts.json colorado-2022-12-27.osm.pbf +``` + + +``` +ryanlambert@tag201:~/pgosm-data$ ls -alh | grep boulder +-rw-rw-r-- 1 ryanlambert ryanlambert 2.4M Dec 27 14:31 colorado-boulder-latest.osm.pbf +ryanlambert@tag201:~/pgosm-data$ ls -alh | grep longmont +-rw-rw-r-- 1 ryanlambert ryanlambert 988K Dec 27 14:31 colorado-longmont-latest.osm.pbf +``` + +Takes 11 seconds. + +``` +time docker exec -it \ + pgosm python3 docker/pgosm_flex.py \ + --ram=8 \ + --region=north-america/us \ + --subregion=colorado-longmont --input-file colorado-longmont-latest.osm.pbf \ + --skip-dump --update create +``` + +Takes 2 minutes. + + +``` +time docker exec -it \ + pgosm python3 docker/pgosm_flex.py \ + --ram=8 \ + --region=north-america/us \ + --subregion=colorado-boulder --input-file colorado-boulder-latest.osm.pbf \ + --skip-dump --update append +``` + diff --git a/requirements.txt b/requirements.txt index c6816a6..495ddac 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,7 @@ click>=8.1.3 coverage>=6.4.1 -osm2pgsql-tuner==0.0.5 -osmium==3.4.1 +osm2pgsql-tuner==0.0.6 +osmium>=3.4.1 psycopg>=3.1 psycopg-binary>=3.1 sh>=1.14.2