value being picked incorrectly in left join

[rkarthik29]

We are trying to a left join of a dataset, basically there are two file , file one containts, account_id and some additional metadata like name,date etc , along with a struct that contains some session-info. The second file has just a list of account_id that need to be identified. When performing a left join i am seeing that duckdb returns all the columns for the join correctly, except in some cases the data inside the struct fields seems to be getting modified and it seems to be picking it from a different record. saw this with both 0.8.1 and 0.9.1 . I will ask permission to paste data here, but for now this is all i can share. Has anyone seen similar behaviour?

[Tishj]

Language? (Python, c++, c, go, rust etc..)
Source of the data? Another in-memory dataframe library (Arrow, Pandas, Julia Dataframes, R dataframes etc..) or csv,parquet,json?

Tried running it singlethreaded?

[rkarthik29]

language is python. Source of data is parquet, registered in duckdb as a pyarrow table. Did try running in singlethreaded. did duckdb.sql("set threads to 1") . Do i have to set the pyarrow reader to be singlethreaded as well?

[rkarthik29]

when running with 1 thread i still see the same issue

[rkarthik29]

also when i do read().to_pandas() on the same data and run a query , the join runs fine.

[rkarthik29]

Also noticed duckdb seems to give wrong data from struct fields when i use in (list of long values). it is failing for certain matches. unfortunately unable to share data.

[Tishj]

I have a PR up to fix an issue in our arrow scan, you might be suffering from the same issue.
In any case, without any reproduction there's very little we can do

[rkarthik29]

Can you share the PR, i think this is related to arrow. I dont reporduce the error 1. if i query the parquet directly through duckdb. 2. if it do the modifications directly in pyarrow or pandas.