ogr2ogr: use Arrow interface in reading and writing when possible #8544
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rouault
force-pushed
the
ogr2ogr_arrow
branch
from
310bcd1
to
ed02742
Compare
|
This is huge! Thanks Even! |
rouault
force-pushed
the
ogr2ogr_arrow
branch
2 times, most recently
from
825568b
to
db19301
Compare
That is when: - the input driver reports OLCFastGetArrowStream - and the output driver reports OLCFastWriteArrowBatch - and the input ArrowSchema is accepted by the target layer - and there are basically no ogr2ogr options (other than -dsco, -lco, -limit, -where and -spat). Note that the efficiency in case of -where and -spat can be less than using the classic OGRFeature strategy when the source is Parquet and a lot of features are discarded. With that Parquet->Parquet or GPKG->Parquet translation can be up to 3x times faster. $ time ogr2ogr out.parquet nz-building-outlines.parquet --config OGR2OGR_USE_ARROW_API NO real 0m11,246s user 0m10,934s sys 0m0,974s $ time ogr2ogr out.parquet nz-building-outlines.parquet real 0m4,311s user 0m3,968s sys 0m0,889s $ time ogr2ogr out.parquet nz-building-outlines.gpkg --config OGR2OGR_USE_ARROW_API NO real 0m12,120s user 0m11,360s sys 0m0,764s $ time ogr2ogr out.parquet nz-building-outlines.gpkg real 0m3,853s user 0m5,053s sys 0m1,028s
rouault
force-pushed
the
ogr2ogr_arrow
branch
from
db19301
to
501bef9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
That is when:
Note that the efficiency in case of -where and -spat can be less than
using the classic OGRFeature strategy when the source is Parquet and a
lot of features are discarded.
With that Parquet->Parquet or GPKG->Parquet translation can be up to 3x times faster.
$ time ogr2ogr out.parquet nz-building-outlines.parquet --config OGR2OGR_USE_ARROW_API NO
real 0m11,246s
user 0m10,934s
sys 0m0,974s
$ time ogr2ogr out.parquet nz-building-outlines.parquet
real 0m4,311s
user 0m3,968s
sys 0m0,889s
$ time ogr2ogr out.parquet nz-building-outlines.gpkg --config OGR2OGR_USE_ARROW_API NO
real 0m12,120s
user 0m11,360s
sys 0m0,764s
$ time ogr2ogr out.parquet nz-building-outlines.gpkg
real 0m3,853s
user 0m5,053s
sys 0m1,028s