Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Performance improvement for .osm.pbf creation and splitting. Use 'v12' tag (keep) filters #46

Merged
merged 9 commits into from
Oct 22, 2021

Conversation

treee111
Copy link
Owner

@treee111 treee111 commented Oct 19, 2021

This PR…

This PR is extracted from #40 and delivers the following functionalities:

  • In the filter stage, do not re-convert the result file(s) back to .osm.pbf format. This saves some time here but more importantly osmconvert is much faster in cutting tiles out of o5m files then .osm.pbf files.
  • Switched to using 'v12' tag (keep) filters. This only keeps the tags used by Wahoo's devices render themes. Additionally, filter out all names tags except for those used, like city names. So highway names for example or filtered out.
  • At the start of the splitting phase check for the existence of a merged .osm.pbf file. This prevents unnecessary splitting tiles out of for instance Europe when the tile is already processed from a smaller, so faster, part like Germany.

treee111 and others added 7 commits October 19, 2021 15:19
- now not only for GUI but also for CLI the country argument is checked
- added a check at the start to see if a country is to process is specified. Otherwise closing the GUI without selecting anything resulted in a runtime error.
…d .osm.pbf files (#43)

- use the files modification date, not the creation time. On windows this seemed to occasionally see newly downloaded files as the ones they replaced.
- check for a full match instead of doing a wildcard match while checking for existing maps. This prevents matching to multiple maps like australia and australia-oceania.
- change pylint findings
- format constans on save
- delete unused comments
@treee111 treee111 marked this pull request as ready for review October 21, 2021 20:50
@treee111 treee111 changed the title [FEATURE] Different improvements of the OSM maps processing [FEATURE] Performance improvement for .osm.pbf creation and splitting. Use 'v12' tag (keep) filters Oct 21, 2021
Copy link
Contributor

@Ebe66 Ebe66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Untested by me but looks 100% ok !

@treee111 treee111 merged commit cef537b into develop Oct 22, 2021
@treee111 treee111 deleted the osm-maps-processing-adjustments branch October 22, 2021 13:01
treee111 added a commit that referenced this pull request Oct 22, 2021
…arge countries (#47)

- Changes how the resulting tiles.zip and maps.zip file is made. Now files are copied from the 'tilestore' to a country-name and country-name-maps folder. This folder is then passed to the compressor instead of building an enormous command line with all file names specified. This would, for large countries like Russia, result in errors because the command line was to long.
- Added advantage: Optionally you can choose via CLI to keep these folders so you can instantly copy the files to your device or check them in Cruiser.
- Copies the - currently static numbered - tile-present/version files (created in #46) to the tiles.zip archive

* change zipping map and cruiser files

- 7za / zip commands with other arguments
- filling the "new" country-maps folder

* Introduce option to keep folders

- keep or delete the country/region map folders after compression
- available via CLI argument

* refactoring / code cleanup

- delete & unify comments
- make variables setting less redundant
- introduce types for except

* delete empty line creations caused by rebase
treee111 added a commit that referenced this pull request Oct 30, 2021
…n and splitting. Use 'v12' tag (keep) filters (#46)"

This partly reverts commit cef537b.
treee111 added a commit that referenced this pull request Oct 30, 2021
…n and splitting. Use 'v12' tag (keep) filters (#46)"

This partly reverts commit cef537b.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants