Potential regression in version 23.4.* while reading Parquet files leads to ParquetInvalidOrCorruptedFileException. The issue is not present in version 23.3.*.

[shawel]

I get the following error reading some parquet file. this file works in databricks and versions >= 23.3

Code: 1001. DB::Exception: Received from localhost:9000. DB::Exception: parquet::ParquetInvalidOrCorruptedFileException: Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.. (STD_EXCEPTION)

SELECT count()
FROM s3('https://s3.us-east-1.amazonaws.com/.../.snappy.parquet', '***, '***, 'Parquet', 'timestamp Nullable(DateTime), dist Nullable(Float64),  event_name Nullable(String), gh Nullable(String), ipAddress Nullable(String), latitude Float64, longitude Float64, source_id Nullable(String)')

[qoega]

Can you provide example of such file? It will make it much easier to fix

[shawel]

@Quid37 ok i will get back to you on this. dm possibly?

[al13n321]

Likely #49525 . Try adding SETTINGS remote_filesystem_read_method='read' to the query and see if it helps.

If that doesn't work, please let me know, and I'll investigate (and will ask you for the file or some metadata from it, etc).

[shawel]

@al13n321 SETTINGS remote_filesystem_read_method='read' does fix the issue

[warleysa]

This is also happening when updated to 23.4.* as well for all of our implementations with S3. I have added SETTINGS to the queries, but this seems to be a larger issue

[anton-zelenskiy]

remote_filesystem_read_method='read' setting also fixes the issue "Orig exception: Code: 32. DB::Exception: Attempt to read after eof: While executing ParquetBlockInputFormat: While executing S3. (ATTEMPT_TO_READ_AFTER_EOF) (version 23.4.2.11 (official build))"

[den-crane]

outdated