Improve handling of multiple input files#1339
Merged
lonvia merged 1 commit intoosm2pgsql-dev:masterfrom Nov 26, 2020
Merged
Conversation
Osm2pgsql can handle any number of input files. The old code will just read the files one after the other which will not work if there is any overlap between the files, i.e. if the same object is in two input files. The new code will read the files in parallel. We construct a priority queue feeding in the next objects from all input files, taking off the "smallest" one by one. If the same object is in multiple files, we only process it once. If there is only a single input file a shortcut is taken which basically behaves like the old code. Note that the input files have to be from the same point in time. If there are multiple versions of the same object in the input, this will still not magically work. This commit removes support for unsorted input files which were already deprecated. See osm2pgsql-dev#1167 This commit removes support for negative ids which were already deprecated. Fixes osm2pgsql-dev#1097
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Osm2pgsql can handle any number of input files. The old code will just
read the files one after the other which will not work if there is any
overlap between the files, i.e. if the same object is in two input
files.
The new code will read the files in parallel. We construct a priority
queue feeding in the next objects from all input files, taking off the
"smallest" one by one. If the same object is in multiple files, we
only process it once.
If there is only a single input file a shortcut is taken which basically
behaves like the old code.
Note that the input files have to be from the same point in time. If
there are multiple versions of the same object in the input, this will
still not magically work.
This commit removes support for unsorted input files which were already
deprecated.
See #1167
This commit removes support for negative ids which were already
deprecated.
Fixes #1097