parquet encoding decoding issues

[superichmann]

To reproduce

download train.parquet and validation_example_preds.parquet from https://signals.numer.ai/data/v2.0
try to import them into qdb on windows with read_parquet function.

first file will produce error: error in PartitionDecoder.decodeRowGroup: could not decode row group 0: could not decode page for column "composite_figi" in row group 0: encoding not supported, physical type: Int32, encoding RleDictionary, logical type Some(Unknown), converted type: None, column type ColumnType(26/Varchar)

second file will produce error: error in PartitionDecoder.decodeRowGroup: could not decode row group 0: could not decode page for column "signal" in row group 0: encoding not supported, physical type: Double, encoding Plain, logical type None, converted type: None, column type ColumnType(10/Double)

QuestDB version:

9.0

OS, in case of Docker specify Docker and the Host OS:

Windows 10

File System, in case of Docker specify Host File System:

NTFS

Full Name:

Super Richmann

Affiliation:

None

Have you followed Linux, MacOs kernel configuration steps to increase Maximum open files and Maximum virtual memory areas limit?

• Yes, I have

Additional context

No response

[nwoolmer]

Thanks @superichmann , we're circling back to bring Parquet from beta to GA release, which includes wider type support and other bits and pieces.

Let me know if you need help finding a workaround!

[puzpuzpuz]

@superichmann these issues will be fixed in the next release. Other that, we're making the read_parquet() SQL function more tolerant to unsupported column types - it'll be simply skipping those and returning what it was able to parse.