Support UNION_BY_NAME option in parquet_scan read_parquet

[douenergy]

#4837 , #4720 , #4699 and #3238

We can make these queries much easier !

SELECT * FROM 'p1.parquet' UNION BY NAME
SELECT * FROM 'p2.parquet' UNION BY NAME
SELECT * FROM 'p3.parquet' 

to

SELECT *  FROM parquet_scan('p*.parquet' ,  union_by_name=TRUE);

• All Parquet files matched the glob pattern will union by their col name, If any Parquet file without some specified cols NULL will be generated for missing parquet cols.

• As described in Add UNION BY NAME to CSV and Parquet import #4720, parquet_scan with union_by_name isn't reversible.

If we have p1.parquet

col1
a
b
c

and p2.parquet

col1
1
2
3

INSERT INTO tbl SELECT * FROM read_parquet('*.parquet', FILENAME=TRUE, UNION_BY_NAME=TRUE);

then

COPY (SELECT * FROM tbl WHERE filename='p2.parquet') TO 'p2new.parquet';

p2new.parquet and p2.parquet will have different type. Maybe we should add some warning?

[Mytherin]

Thanks for the PR! Looks good. Some comments below:

[Mytherin]

This code looks very similar to the code in read_csv.cpp - could we extract this into a separate class (e.g. UnionByName - similar to how we handle hive partitioning in the HivePartitioning class)?

[douenergy]

Yes! That makes code much reusable

[Mytherin]

I'm not sure if modifying the Parquet Reader itself is necessary here - can't we handle the union_by_name in the ParquetScanFunction outside of the ParquetReader class? That should also allow for more code re-use between the CSV and Parquet readers for this functionality - by e.g. moving SetNullUnionCols to a shared UnionByName class.

[Mytherin]

Could you also have a look at the failing CI?

[douenergy]

Could you also have a look at the failing CI?

parquet_union_by_name.test 's glob pattern also match the files created by parquet_3989.test 😅

[danthegoodman1]

Very excited for this!

[danthegoodman1]

If a given file does not contain a column of interest (say we do SELECT col_b FROM read_parquet... WHERE col_b IS NOT NULL), will DuckDB be able to recognize from the metadata that the column does not exist and thus not read the file that does not contain the column?

[douenergy]

If a given file does not contain a column of interest (say we do SELECT col_b FROM read_parquet... WHERE col_b IS NOT NULL), will DuckDB be able to recognize from the metadata that the column does not exist and thus not read the file that does not contain the column?

if you're saying , there are two parquet files
p1.parquet

colA   colB
  1          a
  2          b
  3          c

p2.parquet

colA   colC
  1          x
  2          y
  3          z

For the query SELECT colB FROM read_parquet... WHERE colB IS NOT NULL just read only all metadata and completely skip p2.parquet

Current implementation cant do that. But I think we can improve it in another PR.

[danthegoodman1]

@douenergy sounds good :D I think it could be a worthwhile enhancement in the future!

[danthegoodman1]

@douenergy is there a way to make this behave like UNION ALL BY NAME?

[douenergy]

@douenergy is there a way to make this behave like UNION ALL BY NAME?

Currently, parquet_scan with union_by_name just work like UNION ALL BY NAME.
It allows duplicate row.

[danthegoodman1]

Currently, parquet_scan with union_by_name just work like UNION ALL BY NAME. It allows duplicate row.

Perfect!

Does this issue still occur (it reading all files for something like a count even with a filter)? #5790

[douenergy]

@danthegoodman1
I'think parquet_scan (with union_by_name) would help for your use case.
Because there will no UNION operator in your query plan(even you select from VIEW).

[danthegoodman1]

That'd be AMAZING. I can't get duckdb to build on my machine or in a github codespace to save my life so I will look forward to testing once a release comes out!

[douenergy]

Thanks @Mytherin
I think I've now ready for another review.

1. Created an UnionByName template in common/union_by_name.hpp
2. Rename BaseCSVReader sql_types , col_names to return_types names. This make BaseCSVReader and ParquetReader have same data member names for same purpose.
3. fix the test.

[douenergy]

Unrelated CodeCov github action CI failed
See codecov/codecov-action#598

[Mytherin]

Thanks for the fixes! LGTM