Skip to content

Add test file with no path_in_schema fields#108

Open
etseidl wants to merge 1 commit into
apache:masterfrom
etseidl:path_in_schema_optional
Open

Add test file with no path_in_schema fields#108
etseidl wants to merge 1 commit into
apache:masterfrom
etseidl:path_in_schema_optional

Conversation

@etseidl
Copy link
Copy Markdown
Contributor

@etseidl etseidl commented May 12, 2026

Test file for apache/parquet-format#563. Created with arrow-rs via:

% parquet-rewrite -i nested_maps.snappy.parquet -o no_path_in_schema.zstd.parquet --compression zstd --statistics-enabled none --write-path-in-schema false

% parquet-cli meta no_path_in_schema.zstd.parquet

File path:  no_path_in_schema.zstd.parquet
Created by: parquet-rs version 58.3.0
Properties:
                               ARROW:schema: /////wgCAAAQAAAAAAAKAAwACgAJAAQACgAAABAAAAAAAQQACAAIAAAABAAIAAAABAAAAAMAAAB4AAAASAAAABQAAAAQABYAEAAAAA8ABAAAAAgAEAAAABgAAAAcAAAAAAAAAxgAAAAAAAYACAAGAAYAAAAAAAIAAAAAAAEAAABjAAAAxP7//xAAAAAYAAAAAAAAAhQAAAAU////IAAAAAAAAAEAAAAAAQAAAGIAAAC8////GAAAAAwAAAAAAAERSAEAAAEAAAAIAAAA5P7//xD///8cAAAADAAAAAAAAA0YAQAAAgAAAOgAAAAYAAAACP///xAAFAAQAA4ADwAEAAAACAAQAAAAGAAAAAwAAAAAAAERoAAAAAEAAAAIAAAAOP///2T///8cAAAADAAAAAAAAA1wAAAAAgAAADQAAAAIAAAAXP///4j///8UAAAADAAAAAAAAAYMAAAAAAAAAHj///8FAAAAdmFsdWUAAACw////GAAAACAAAAAAAAACHAAAAAgADAAEAAsACAAAACAAAAAAAAABAAAAAAMAAABrZXkACQAAAGtleV92YWx1ZQAAAAUAAAB2YWx1ZQAAABAAFAAQAAAADwAEAAAACAAQAAAAGAAAAAwAAAAAAAAFEAAAAAAAAAAEAAQABAAAAAMAAABrZXkACQAAAGtleV92YWx1ZQAAAAEAAABhAAAA
  org.apache.spark.sql.parquet.row.metadata: {"type":"struct","fields":[{"name":"a","type":{"type":"map","keyType":"string","valueType":{"type":"map","keyType":"integer","valueType":"boolean","valueContainsNull":false},"valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"b","type":"integer","nullable":false,"metadata":{}},{"name":"c","type":"double","nullable":false,"metadata":{}}]}
Schema:
message arrow_schema {
  optional group a (MAP) {
    repeated group key_value {
      required binary key (STRING);
      optional group value (MAP) {
        repeated group key_value {
          required int32 key;
          required boolean value;
        }
      }
    }
  }
  required int32 b;
  required double c;
}


Row group 0:  count: 6  58.50 B records  start: 4  total(compressed): 351 B total(uncompressed):270 B 
--------------------------------------------------------------------------------
                                   type      encodings count     avg size   nulls   min / max
a.key_value.key                    BINARY    Z _ R     6         16.00 B            
a.key_value.value.key_value.key    INT32     Z _ R     9         10.44 B            
a.key_value.value.key_value.value  BOOLEAN   Z   _     9         5.22 B             
b                                  INT32     Z _ R     6         9.17 B             
c                                  DOUBLE    Z _ R     6         9.83 B             

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants