Adding parquet files error: Failed to create checkpoint because of error: Could not read enough bytes from file

[alicia-peters-dev]

What happens?

Hi,

I have been experiencing some issues inserting data from parquet files into duckdb. In most cases this works well, but I have a few cases where I have consistent errors, that look something like this:

TransactionContext Error: Failed to commit: Failed to create checkpoint because of error: Could not read enough bytes from file "/tmp/.db": attempted to read 262144 bytes from location 4869745866884263936

In this particular case I am exporting a large table from BigQuery as parquet files (5000, totally ~125GB). Some time around loading the 2000th file into duckdb I will get an error thrown like the above.

In order to add the files, I have tried both of the following statements:

COPY db.{table} 
FROM '{file}' (FORMAT parquet)

and

INSERT INTO db.{table}
SELECT {select_clause} FROM read_parquet('{file}')

I've also tried v1.3.2 and v1.4.1 of the python package, but the error still happens.

To Reproduce

N/a

OS:

linux

DuckDB Package Version:

1.3.2

Python Version:

3.9

Full Name:

Alicia Peters

Affiliation:

Benchling

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[taniabogatsch]

Hi @alicia-peters-dev - the easiest way to look into this would be by sharing the data (up to the 2000ish file). See my other reply.

"/tmp/.db" - that file name looks odd, how are you attaching/opening the file?

Besides that, it's very hard to say why the location passed to the file read function is uninitialized/"out of bounds". It's a bug, and maybe with a stack trace we could know more, is it possible to run this with lldb and get that, maybe?

As for workarounds, maybe you could try running manual checkpoints more aggressively. I.e., I assume you're running

COPY db.{table} 
FROM '{file}' (FORMAT parquet)

in a loop for each file? We auto-checkpoint every 16mb, but maybe you can decrease that (SET checkpoint_threshold = ...), or run manual checkpoints via CHECKPOINT db. However, that would likely just hide the bug, if it does anything...

Another idea to work around it is to disable different compression methods - depending on your data, you could only run with subsets and see if that makes a difference, to eliminate the code path with the bug, if its compression-related. Available compression functions are "rle", "dictionary", "pfor", "bitpacking", "fsst", "chimp", "patas", "zstd", "alp", "alprd", "roaring", "dict_fsst". You can disable them via: SET disabled_compression_methods='fsst,rle';.