Unable to read Parquet file written by databricks

[jaychia]

Describe the bug

Attached an example Parquet file which was written through databricks.

It seems other Rust-based engines also fail to read it:

Polars, pqrs

However, I was able to successfully read it via duckdb and pyarrow.

File:
part-00000-68793c08-5b0e-480e-bd6b-0a7568d49906.c000.snappy.parquet.zip

To Reproduce

df = daft.read_parquet("file.parquet")

This fails with File out of specification: Invalid thrift: bad data

import pyarrow.parquet as papq

f = papq.ParquetFile("myfile.parquet")
f.metadata

This prints

<pyarrow._parquet.FileMetaData object at 0x145c60720>
  created_by: parquet-mr compatible Photon version 0.2 (build 16.4)
  num_columns: 10
  num_rows: 76
  num_row_groups: 1
  format_version: 1.0
  serialized_size: 4432

Expected behavior

No response

Component(s)

Parquet

Additional context

No response

[desmondcheongzx]

Took some time to look at this. Potentially this is due to something that the parquet-mr compatible photon writer shoves into the metadata that's messing up with the rust implementations of the Thrift compact protocol (which all have the same lineage from parquet-format-safe from the parquet2 days I believe). Haven't root caused it, but there's an interesting sequence of bytes where this causes issues:

<truncated>
00000970: 36 37 64 00 18 1e 63 6f  6d 2e 64 61 74 61 62 72  |67d...com.databr|
00000980: 69 63 6b 73 2e 73 70 61  72 6b 2e 63 6c 75 73 74  |icks.spark.clust|
00000990: 65 72 49 64 18 14 30 36  30 37 2d 30 30 35 34 30  |erId..0607-00540|
000009a0: 33 2d 34 32 65 37 77 67  61 6f 00 18 23 63 6f 6d  |3-42e7wgao..#com|
000009b0: 2e 64 61 74 61 62 72 69  63 6b 73 2e 73 70 61 72  |.databricks.spar|
000009c0: 6b 2e 77 72 69 74 65 54  69 6d 65 73 74 61 6d 70  |k.writeTimestamp|
000009d0: 18 1e 32 30 32 35 2d 30  36 2d 30 37 54 30 31 3a  |..2025-06-07T01:|
000009e0: 31 32 3a 32 36 2e 34 34  36 34 35 34 36 35 38 5a  |12:26.446454658Z|
000009f0: 00 18 35 70 61 72 71 75  65 74 2d 6d 72 20 63 6f  |..5parquet-mr co|
00000a00: 6d 70 61 74 69 62 6c 65  20 50 68 6f 74 6f 6e 20  |mpatible Photon |
00000a10: 76 65 72 73 69 6f 6e 20  30 2e 32 20 28 62 75 69  |version 0.2 (bui|
00000a20: 6c 64 20 31 36 2e 34 29  08 ff ff 01 a1 0e 22 18  |ld 16.4)......".|
          l  d     1  6  .  4  )
                                   ^ type = string
                                      ^^^^^^^^ read i16 as -16384 (set as new field id)
                                               ^^^^^ read i32 as 1825 for num bytes to read
                                                     ^ supposed start of 1825 char string
00000a30: 00 01 00 f2 12 10 00 20  00 1c 00 18 00 10 00 0c  |....... ........|
                   ^^ invalid utf8 char
<truncated>

Reading -16384 as a field id smells awfully like a bug somewhere.

[jaychia]

For what it’s worth, rewriting the table with databricks serverless Spark SQL fixed the issue hahaha…

…

-- Jay Chia Founder, Eventual <https://www.eventualcomputing.com/>

On Tue, Jun 10, 2025 at 6:49 PM Desmond Cheong ***@***.***> wrote: *desmondcheongzx* left a comment (Eventual-Inc/Daft#4515) <#4515 (comment)> Took some time to look at this. Potentially this is due to something that the parquet-mr compatible photon writer shoves into the metadata that's messing up with the rust implementations of the Thrift compact protocol (which all have the same lineage from parquet-format-safe from the parquet2 days I believe). Haven't root caused it, but there's an interesting sequence of bytes where this causes issues: <truncated> 00000970: 36 37 64 00 18 1e 63 6f 6d 2e 64 61 74 61 62 72 |67d...com.databr| 00000980: 69 63 6b 73 2e 73 70 61 72 6b 2e 63 6c 75 73 74 |icks.spark.clust| 00000990: 65 72 49 64 18 14 30 36 30 37 2d 30 30 35 34 30 |erId..0607-00540| 000009a0: 33 2d 34 32 65 37 77 67 61 6f 00 18 23 63 6f 6d |3-42e7wgao..#com| 000009b0: 2e 64 61 74 61 62 72 69 63 6b 73 2e 73 70 61 72 |.databricks.spar| 000009c0: 6b 2e 77 72 69 74 65 54 69 6d 65 73 74 61 6d 70 |k.writeTimestamp| 000009d0: 18 1e 32 30 32 35 2d 30 36 2d 30 37 54 30 31 3a |..2025-06-07T01:| 000009e0: 31 32 3a 32 36 2e 34 34 36 34 35 34 36 35 38 5a |12:26.446454658Z| 000009f0: 00 18 35 70 61 72 71 75 65 74 2d 6d 72 20 63 6f |..5parquet-mr co| 00000a00: 6d 70 61 74 69 62 6c 65 20 50 68 6f 74 6f 6e 20 |mpatible Photon | 00000a10: 76 65 72 73 69 6f 6e 20 30 2e 32 20 28 62 75 69 |version 0.2 (bui| 00000a20: 6c 64 20 31 36 2e 34 29 08 ff ff 01 a1 0e 22 18 |ld 16.4)......".| l d 1 6 . 4 ) ^ type = string ^^^^^^^^ read i16 as -16384 (set as new field id) ^^^^^ read i32 as 1825 for num bytes to read ^ supposed start of 1825 char string 00000a30: 00 01 00 f2 12 10 00 20 00 1c 00 18 00 10 00 0c |....... ........| ^^ invalid utf8 char <truncated> Reading -16384 as a field id smells awfully like a bug somewhere. — Reply to this email directly, view it on GitHub <#4515 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEG7ELQ2CEGSLJDX5IMJWF33C6DK7AVCNFSM6AAAAAB66PHVX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRQHE4DQNRUGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[desmondcheongzx]

LOL what version of dbr did you write the initial file with

[desmondcheongzx]

Haha @jaychia I thought I'd test your hypothesis that the Rust libraries are doing UTF-8 validation while the other java/c++-based libraries aren't. Turns out it's spot on the money. If I modify the thrift compact protocol to ignore UTF-8 errors, via

diff --git a/src/thrift/protocol/compact.rs b/src/thrift/protocol/compact.rs
index ec6b329..2218ceb 100644
--- a/src/thrift/protocol/compact.rs
+++ b/src/thrift/protocol/compact.rs
@@ -250,7 +250,17 @@ where
 
     fn read_string(&mut self) -> Result<String> {
         let bytes = self.read_bytes()?;
-        String::from_utf8(bytes).map_err(From::from)
+        let result: crate::thrift::Result<String> = String::from_utf8(bytes).map_err(From::from);
+        match result {
+            Ok(s) => {
+                println!("DEBUG: read_string - value: '{}'", s);
+                Ok(s)
+            }
+            Err(e) => {
+                println!("DEBUG: read_string - error: {:?}", e);
+                Ok("placeholder".to_string())
+            }
+        }
     }
 
     fn read_list_begin(&mut self) -> Result<TListIdentifier> {

we can then read the file with no issues:

>>> daft.read_parquet("file.parquet").show()

╭────────────────────────────────┬──────────┬─────────────────────┬─────────┬────────────────────────────────┬───────┬────────────────┬─────────────────┬────────────────────────────────┬────────────────────────────────╮
│ url                            ┆ NSFW     ┆ similarity          ┆ LICENSE ┆ caption                        ┆ key   ┆ original_width ┆ original_height ┆ image_id                       ┆ unity_volume_url               │
│ ---                            ┆ ---      ┆ ---                 ┆ ---     ┆ ---                            ┆ ---   ┆ ---            ┆ ---             ┆ ---                            ┆ ---                            │
│ Utf8                           ┆ Utf8     ┆ Float64             ┆ Utf8    ┆ Utf8                           ┆ Int64 ┆ Int64          ┆ Int64           ┆ Utf8                           ┆ Utf8                           │
╞════════════════════════════════╪══════════╪═════════════════════╪═════════╪════════════════════════════════╪═══════╪════════════════╪═════════════════╪════════════════════════════════╪════════════════════════════════╡
│ http://t0.gstatic.com/images?… ┆ UNLIKELY ┆ 0.30712828040122986 ┆ ?       ┆ bedroom minimalist home inter… ┆ 9048  ┆ 277            ┆ 182             ┆ 693edbb0-9e4c-4b7f-8414-e00d9… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ http://img.beckett.com/images… ┆ UNLIKELY ┆ 0.3508765399456024  ┆ ?       ┆ 2001 Absolute Memorabilia #19… ┆ 6649  ┆ 318            ┆ 231             ┆ 4a389ffc-cd48-4619-a12a-ce73c… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ https://ae01.alicdn.com/kf/HT… ┆ UNLIKELY ┆ 0.359369695186615   ┆ ?       ┆ high waist sleeveless mini so… ┆ 102   ┆ 800            ┆ 800             ┆ e1791473-aff7-49a7-aa7d-aae8c… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ https://images.wolfgangsvault… ┆ UNLIKELY ┆ 0.3070274293422699  ┆ ?       ┆ "Al Hirt / Boston Pops / Arth… ┆ 9286  ┆ 190            ┆ 190             ┆ 0d8e74cf-1ed4-4571-92a9-425b0… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ http://www.wildernesswife.com… ┆ UNLIKELY ┆ 0.34780216217041016 ┆ ?       ┆ rice pudding, cranberry rice … ┆ 9030  ┆ 578            ┆ 866             ┆ 09a15d8b-17fc-4f27-afbc-efa45… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ https://gamingbolt.com/wp-con… ┆ UNSURE   ┆ 0.3310202658176422  ┆ ?       ┆ Lightning Returns: Final Fant… ┆ 7333  ┆ 650            ┆ 917             ┆ 2252a846-fb97-4576-b486-c9e74… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ https://assets1.bmstatic.com/… ┆ UNLIKELY ┆ 0.39273834228515625 ┆ ?       ┆ CultureShock! South Africa. A… ┆ 8802  ┆ 190            ┆ 320             ┆ 83a98a9c-e0b2-454f-b35f-27695… ┆ dbfs:/Volumes/jaytest-unity/d… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ https://www.marketingproducti… ┆ UNLIKELY ┆ 0.3566654622554779  ┆ ?       ┆ Tips para usar Google my busi… ┆ 2400  ┆ 150            ┆ 150             ┆ 4a9cb89a-3002-4a49-b882-0d783… ┆ dbfs:/Volumes/jaytest-unity/d… │
╰────────────────────────────────┴──────────┴─────────────────────┴─────────┴────────────────────────────────┴───────┴────────────────┴─────────────────┴────────────────────────────────┴────────────────────────────────╯

(Showing first 8 rows)

We can read all the metadata that Parquet-java reads too. What we see:

=== PARQUET FILE METADATA ===
Version: 1
Number of rows: 76
Number of row groups: 1
Created by: parquet-mr compatible Photon version 0.2 (build 16.4)

--- SCHEMA ---
Schema name: spark_schema
Number of fields: 10
  Field 0: url
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional
  Field 1: NSFW
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional
  Field 2: similarity
    - Type: Double
    - Logical Type: None
    - Repetition: Optional
  Field 3: LICENSE
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional
  Field 4: caption
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional
  Field 5: key
    - Type: Int64
    - Logical Type: Some(Integer(Int64))
    - Repetition: Optional
  Field 6: original_width
    - Type: Int64
    - Logical Type: Some(Integer(Int64))
    - Repetition: Optional
  Field 7: original_height
    - Type: Int64
    - Logical Type: Some(Integer(Int64))
    - Repetition: Optional
  Field 8: image_id
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional
  Field 9: unity_volume_url
    - Type: ByteArray
    - Logical Type: Some(String)
    - Repetition: Optional

--- ROW GROUPS ---
Row Group 0: 76 rows, 27934 bytes
  Columns:
    Column 0: ["url"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "url", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 7808 bytes
      - Compressed size: 6037 bytes
    Column 1: ["NSFW"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "NSFW", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(2), Encoding(3), Encoding(4)]
      - Uncompressed size: 116 bytes
      - Compressed size: 118 bytes
    Column 2: ["similarity"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "similarity", repetition: Optional, id: None }, logical_type: None, converted_type: None, physical_type: Double })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 665 bytes
      - Compressed size: 518 bytes
    Column 3: ["LICENSE"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "LICENSE", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(2), Encoding(3), Encoding(4)]
      - Uncompressed size: 71 bytes
      - Compressed size: 75 bytes
    Column 4: ["caption"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "caption", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 4791 bytes
      - Compressed size: 4348 bytes
    Column 5: ["key"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "key", repetition: Optional, id: None }, logical_type: Some(Integer(Int64)), converted_type: None, physical_type: Int64 })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 665 bytes
      - Compressed size: 413 bytes
    Column 6: ["original_width"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "original_width", repetition: Optional, id: None }, logical_type: Some(Integer(Int64)), converted_type: None, physical_type: Int64 })
      - Compression: Snappy
      - Encodings: [Encoding(2), Encoding(3), Encoding(4)]
      - Uncompressed size: 589 bytes
      - Compressed size: 396 bytes
    Column 7: ["original_height"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "original_height", repetition: Optional, id: None }, logical_type: Some(Integer(Int64)), converted_type: None, physical_type: Int64 })
      - Compression: Snappy
      - Encodings: [Encoding(2), Encoding(3), Encoding(4)]
      - Uncompressed size: 605 bytes
      - Compressed size: 401 bytes
    Column 8: ["image_id"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "image_id", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 3153 bytes
      - Compressed size: 3010 bytes
    Column 9: ["unity_volume_url"]
      - Type: PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "unity_volume_url", repetition: Optional, id: None }, logical_type: Some(String), converted_type: Some(Utf8), physical_type: ByteArray })
      - Compression: Snappy
      - Encodings: [Encoding(0), Encoding(3), Encoding(4)]
      - Uncompressed size: 9471 bytes
      - Compressed size: 3528 bytes

--- KEY-VALUE METADATA ---
  org.apache.spark.legacyDateTime: Some("")
  org.apache.spark.timeZone: Some("Etc/UTC")
  org.apache.spark.version: Some("3.5.2")
  org.apache.spark.sql.parquet.row.metadata: Some("{\"type\":\"struct\",\"fields\":[{\"name\":\"url\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"NSFW\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"similarity\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}},{\"name\":\"LICENSE\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"caption\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"key\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"original_width\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"original_height\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"image_id\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"unity_volume_url\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}")
  com.databricks.spark.jobGroupId: Some("01f0433c-75d1-1289-90cd-a44e7024c67d")
  com.databricks.spark.clusterId: Some("0607-005403-42e7wgao")
  com.databricks.spark.writeTimestamp: Some("2025-06-07T01:12:26.446454658Z")

What parquet-java sees:

File path:  file.parquet
Created by: parquet-mr compatible Photon version 0.2 (build 16.4)
Properties:
                  org.apache.spark.timeZone: Etc/UTC
                   org.apache.spark.version: 3.5.2
            com.databricks.spark.jobGroupId: 01f0433c-75d1-1289-90cd-a44e7024c67d
        com.databricks.spark.writeTimestamp: 2025-06-07T01:12:26.446454658Z
            org.apache.spark.legacyDateTime: 
  org.apache.spark.sql.parquet.row.metadata: {"type":"struct","fields":[{"name":"url","type":"string","nullable":true,"metadata":{}},{"name":"NSFW","type":"string","nullable":true,"metadata":{}},{"name":"similarity","type":"double","nullable":true,"metadata":{}},{"name":"LICENSE","type":"string","nullable":true,"metadata":{}},{"name":"caption","type":"string","nullable":true,"metadata":{}},{"name":"key","type":"long","nullable":true,"metadata":{}},{"name":"original_width","type":"long","nullable":true,"metadata":{}},{"name":"original_height","type":"long","nullable":true,"metadata":{}},{"name":"image_id","type":"string","nullable":true,"metadata":{}},{"name":"unity_volume_url","type":"string","nullable":true,"metadata":{}}]}
             com.databricks.spark.clusterId: 0607-005403-42e7wgao
Schema:
message spark_schema {
  optional binary url (STRING);
  optional binary NSFW (STRING);
  optional double similarity;
  optional binary LICENSE (STRING);
  optional binary caption (STRING);
  optional int64 key (INTEGER(64,true));
  optional int64 original_width (INTEGER(64,true));
  optional int64 original_height (INTEGER(64,true));
  optional binary image_id (STRING);
  optional binary unity_volume_url (STRING);
}


Row group 0:  count: 76  247.95 B records  start: 4  total(compressed): 18.402 kB total(uncompressed):27.279 kB 
--------------------------------------------------------------------------------
                  type      encodings count     avg size   nulls   min / max
url               BINARY    S RB_     76        79.43 B    0       "http://1.bp.blogspot.com/..." / "https://www.rewonline.com..."
NSFW              BINARY    S RBR     76        1.55 B     0       "NSFW" / "UNSURE"
similarity        DOUBLE    S RB_     76        6.82 B     0       "0.30038222670555115" / "0.4123469591140747"
LICENSE           BINARY    S RBR     76        0.99 B     1       "?" / "?"
caption           BINARY    S RB_     76        57.21 B    0       ""3S ""Power Cell"" 25C Li..." / "stock-vector-set-of-femal..."
key               INT64     S RB_     76        5.43 B     0       "102" / "9786"
original_width    INT64     S RBR     76        5.21 B     0       "64" / "1800"
original_height   INT64     S RBR     76        5.28 B     0       "50" / "1200"
image_id          BINARY    S RB_     76        39.61 B    0       "005f7374-aa9d-4aa8-8840-b..." / "fc66e356-540e-4b53-b5af-8..."
unity_volume_url  BINARY    S RB_     76        46.42 B    0       "dbfs:/Volumes/jaytest-uni..." / "dbfs:/Volumes/jaytest-uni..."

[desmondcheongzx]

Unfortunately this patch does require pulling parquet-format-safe in tree...