Parquet list columns read incorrectly

[bmschmidt]

What happens?

When scanning parquet files with list columns, results with incorrect offsets can be returned.

To Reproduce

import numpy as np
import pyarrow as pa
import duckdb
import pyarrow.parquet as pq
R = 100_000
fake_data = pa.array([np.arange(i, i + 100).astype('float64') for i in range(R)])
tb = pa.table({'word0': pa.array([str(i) for i in range(0, R)]), 'year_counts': fake_data})
pq.write_table(tb, "test.parquet")

con = duckdb.connect(":memory:")

con.query("SELECT year_counts FROM parquet_scan('test.parquet') WHERE word0='90000'")

This should return a list-column of length 100 starting at 90,000; instead it starts with 912.

pq.read_table("test.parquet", columns = ['year_counts'], filters = [("word0", "=", "90000")])['year_counts']

returns the correct result.

Environment (please complete the following information):

• OS: OS X
• DuckDB Version: 0.3.1-dev550
• DuckDB Client: Python / WASM

Before Submitting

• Have you tried this on the latest master branch?

• Python: pip install duckdb --upgrade --pre
• R: install.packages("https://github.com/duckdb/duckdb/releases/download/master-builds/duckdb_r_src.tar.gz", repos = NULL)
• Other Platforms: You can find binaries here or compile from source.

• Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

[hannes]

Thanks for reporting, we will have a look

[FenrirWillow]

Hello everyone, this bug impacted one of the prototypes we were exploring using DuckDB + Node for, so I took a stab at fixing the problem myself. I will add some regression tests to ensure that this return in some nasty form, but in the meantime if there is anything else that needs doing to help this get merged, please let me know :)

[Mytherin]

This should be fixed now.