Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPKG: bool values are not being read correctly when arrow is used #8998

Closed
theroggy opened this issue Dec 23, 2023 · 0 comments · Fixed by #8999
Closed

GPKG: bool values are not being read correctly when arrow is used #8998

theroggy opened this issue Dec 23, 2023 · 0 comments · Fixed by #8999
Assignees

Comments

@theroggy
Copy link
Contributor

theroggy commented Dec 23, 2023

Originally reported here: geopandas/pyogrio#334

When arrow is enabled when reading a GPKG, bool values being read seem te get random values. When arrow is disabled, the values are OK.

Tested using gdal 3.8.1

Script to reproduce:

import geopandas as gpd
from osgeo import gdal

gdal.UseExceptions()
src = "https://github.com/geopandas/pyogrio/files/13750016/sample.gpkg.zip"
interesting_rows = [0, 1, 9]
src_gdf = gpd.read_file(src)

# Translate file without Arrow
dst = "C:/temp/sample_dst.gpkg"
with gdal.config_option("OGR2OGR_USE_ARROW_API", "NO"):
    ds_output = gdal.VectorTranslate(srcDS=src, destNameOrDestDS=dst)
    ds_output = None
dst_gdf = gpd.read_file(dst)
print(f"without arrow, equal: {src_gdf.foo.tolist() == dst_gdf.foo.tolist()}")

# Translate file with Arrow -> result for bool values is not correct
dst_arrow = "C:/temp/sample_dst_arrow.gpkg"
ds_output = gdal.VectorTranslate(srcDS=src, destNameOrDestDS=dst_arrow)
ds_output = None
dst_arrow_gdf = gpd.read_file(dst_arrow)
print(f"with arrow, values wrong: {src_gdf.foo.tolist() == dst_arrow_gdf.foo.tolist()}")
print(f"src foo values for rows {interesting_rows}: {src_gdf.foo[interesting_rows].tolist()}")
print(f"dst foo values for rows {interesting_rows}: {dst_gdf.foo[interesting_rows].tolist()}")
print(f"dst_arrow foo values for rows {interesting_rows}: {dst_arrow_gdf.foo[interesting_rows].tolist()}")

Output:

without arrow, equal: True
with arrow, values wrong: False
src foo values for rows [0, 1, 9]: [False, True, True]
dst foo values for rows [0, 1, 9]: [False, True, True]
dst_arrow foo values for rows [0, 1, 9]: [True, False, True]
@theroggy theroggy changed the title GPKG: wrong bool values being read using arrow GPKG: bool values are not being read correctly when arrow is used Dec 23, 2023
@rouault rouault self-assigned this Dec 24, 2023
rouault added a commit to rouault/gdal that referenced this issue Dec 24, 2023
…fects ogr2ogr from GPKG/FlatGeoBuf to something else) (fixes OSGeo#8998)
rouault added a commit that referenced this issue Dec 24, 2023
…fects ogr2ogr from GPKG/FlatGeoBuf to something else) (fixes #8998)
ralphraul pushed a commit to 1SpatialGroupLtd/gdal that referenced this issue Mar 11, 2024
…fects ogr2ogr from GPKG/FlatGeoBuf to something else) (fixes OSGeo#8998)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants