[Postgres provider] Speed up feature insertion when pkid column is not set (fixes #18976) #7144
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t set (fixes qgis#18976) When inserting features whose attribute column corresponding to the pkid is empty, and tat this pkid column has a default value using a sequence, then we can remove it from the INSERT statement completely, which save us from doing a SELECT nextval(...) for each row. On the provided test case of that ticket, on a debug build, this cuts down the insertion time from 5 minutes to 1 minute 10s.
nyalldawson
reviewed
Jun 1, 2018
|
|
||
| if ( !skipSinglePKField ) | ||
| { | ||
| Q_FOREACH ( int idx, mPrimaryKeyAttrs ) |
There was a problem hiding this comment.
range based for instead of Q_FOREACH
There was a problem hiding this comment.
ok, note that I didn't modify that code. Was just put inside a if ()
|
Nice! The biggest performance issue with postgres provider add features is that we don't features using copy. We should be able to detect if the FastInsert flag is set and in that case just throw all the features in via copy... That would speed up lots of postgres data operations, e.g. any method which exports/saves whole tables into postgres. But that's non trivial :) |
To benchmark I tried with ogr2ogr as well (optimized build). In insert mode, it takes ~30 seconds. In copy mode: ~10 seconds |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When inserting features whose attribute column corresponding to the pkid
is empty, and tat this pkid column has a default value using a sequence,
then we can remove it from the INSERT statement completely, which save us
from doing a SELECT nextval(...) for each row. On the provided test case
of that ticket, on a debug build, this cuts down the insertion time from 5 minutes
to 1 minute 10s.