Skip to content
Davit Mnatobishvili edited this page Nov 1, 2024 · 43 revisions

Parquet File Name

File name is determine via the fileName field inside the JSON definition.

  "fileName": "map_example.parquet",

Options

Options determine the overall settings of the file like:

  • the size of a row group
  • the size of a page
  • version of the parquet writer
  • compression
  • applying bloom filer on columns

Datatypes

Regular Types

Integers

Int8

Full example here

  {
    "name": "int8",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "INT8",
    "data": [1, 2, 3, 4, 5]
  }

Int16

Full example here

  {
    "name": "int16",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "INT16",
    "data": [1, 2, 3, 4, 5]
  }

Int32

Full example here

  {
    "name": "int32",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "INT32",
    "data": [1, 2, 3, 4, 5]
  }

Int64

Full example here

  {
    "name": "int64",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "INT64",
    "data": [1, 2, 3, 4, 5]
  }

UInt8

Full example here

  {
    "name": "uint8",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "UINT8",
    "data": [1, 2, 3, 4, 5]
  }

UInt16

Full example here

  {
    "name": "uint16",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "UINT16",
    "data": [1, 2, 3, 4, 5]
  }

UInt32

Full example here

  {
    "name": "uint32",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "UINT32",
    "data": [1, 2, 3, 4, 5]
  }

Int64

Full example here

  {
    "name": "int64",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "UINT64",
    "data": [1, 2, 3, 4, 5]
  }

UTF8

Full example here

  {
    "name": "utf8",
    "schemaType": "required",
    "physicalType": "BINARY",
    "logicalType": "UTF8",
    "data": ["one", "two", "three", "four", "five", "\uD83D\uDCE6"]
  }

Decimal That Fits Into INT32 Physical Type

Full example here

  {
    "name": "decimal_int32",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "DECIMAL",
    "precision": 3,
    "scale": 2,
    "data": [123, 321, 424]
  }

Large Decimal That Doesn't Fit Into INT32 Physical Type

Full example here

  {
    "name": "decimal_int64",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "DECIMAL",
    "precision": 10,
    "scale": 3,
    "data": [2147483648, 2147483649, 2147483650]
  }

Decimal Annotated To BINARY

Full example here

  {
    "name": "decimal_binary",
    "schemaType": "required",
    "physicalType": "BINARY",
    "logicalType": "DECIMAL",
    "precision": 10,
    "scale": 3,
    "data": ["213", "421", "1234"]
  }

DATE

Full example here

  {
    "name": "date_field_int32",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "DATE",
    "data": [18628]
  }

TIME_MILLIS

Full example here

  {
    "name": "time_millis_field",
    "schemaType": "required",
    "physicalType": "INT32",
    "logicalType": "TIME_MILLIS",
    "data": [12345678, 23456789, 34567890, 45678901, 56789012]
  }

TIME_MICROS

Full example here

  {
    "name": "time_micros_field",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "TIME_MICROS",
    "data": [123456789012, 234567890123, 345678901234, 456789012345, 567890123456]
  }

TIMESTAMP_MICROS

Full example here

  {
    "name": "timestamp_micros_field",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "TIMESTAMP_MICROS",
    "data": [1609459200000000, 1609545600000000, 1609632000000000, 1609718400000000, 1609804800000000]
  }

TIMESTAMP_MILLIS

Full example here

  {
    "name": "timestamp_millis_field",
    "schemaType": "required",
    "physicalType": "INT64",
    "logicalType": "TIMESTAMP_MILLIS",
    "data": [1609459200000, 1609545600000, 1609632000000, 1609718400000, 1609804800000]
  }

JSON and BSON

Full example here

    {
      "name": "json_field",
      "schemaType": "optional",
      "physicalType": "BINARY",
      "logicalType": "JSON",
      "data": [
        "{\"key1\": \"value1\"}",
        "{\"key2\": \"value2\"}",
        "{\"key3\": \"value3\"}",
        "{\"key4\": \"value4\"}",
        "{\"key5\": \"value5\"}"
      ]
    },
    {
      "name": "bson_field",
      "schemaType": "optional",
      "physicalType": "BINARY",
      "logicalType": "BSON",
      "data": [
        "{\"key1\": \"value1\"}",
        "{\"key2\": \"value2\"}",
        "{\"key3\": \"value3\"}",
        "{\"key4\": \"value4\"}",
        "{\"key5\": \"value5\"}"
      ]
    }

STRING

Full example here

  {
    "name": "binary_field",
    "schemaType": "required",
    "physicalType": "BINARY",
    "logicalType": "STRING",
    "data": ["one", "two", "three", "four", "five"]
  }

ENUM

Full example here

    {
      "name": "enum_field",
      "schemaType": "required",
      "physicalType": "BINARY",
      "logicalType": "ENUM",
      "data": ["a", "b", "c", "d", "e"]
    }

UUID

Full example here

  {
    "name": "uuid_field",
    "schemaType": "required",
    "physicalType": "FIXED_LEN_BYTE_ARRAY",
    "logicalType": "UUID",
    "length": 16,
    "data": [
      "550e8400e29b41d4a716446655440000",
      "550e8400e29b41d4a716446655440001",
      "550e8400e29b41d4a716446655440002",
      "550e8400e29b41d4a716446655440003",
      "550e8400e29b41d4a716446655440004"
    ]
  }

Note

Here length is the specified length of the FIXED_LEN_BYTE_ARRAY which is 16 for the given uuid values.

  • name: Column name.
  • schemaType: Specifies whether the column allows null values (required means no null values).
  • physicalType: Defines the physical data type.
  • logicalType: Defines the logical type for better data interpretation.
  • data: An array of values to populate the column.

Complex Types

Array

Full example here

{
  "name": "person",
  "schemaType": "repeatedGroup",
  "fields": [
    {
      "name": "name",
      "schemaType": "optional",
      "physicalType": "BINARY",
      "logicalType": "STRING"
    },
    {
      "name": "age",
      "schemaType": "required",
      "physicalType": "INT32"
    }
  ],
  "data": [
    {
      "name": "Alice",
      "age": 30
    },
    {
      "name": "Bob",
      "age": 25
    }
  ]
}

Nested Array

Full example here

    {
      "name": "attributes",
      "schemaType": "repeatedGroup",
      "logicalType": "INT32",
      "fields": [
        {
          "name": "key_value",
          "schemaType": "repeatedGroup",
          "fields": [
            {
              "name": "key",
              "schemaType": "required",
              "physicalType": "BINARY",
              "logicalType": "STRING"
            },
            {
              "name": "value",
              "schemaType": "optional",
              "physicalType": "BINARY",
              "logicalType": "STRING"
            }
          ]
        }
      ],
      "data": [{"key_value": [{"key": "tqwqmcqvqo", "value": "gkqcl"}]}]
    }

Tuple

Full example here

    {
      "name": "person",
      "schemaType": "requiredGroup",
      "fields": [
        {
          "name": "name",
          "schemaType": "optional",
          "physicalType": "BINARY",
          "logicalType": "STRING"
        },
        {
          "name": "age",
          "schemaType": "required",
          "physicalType": "INT32"
        }
      ],
      "data": [
        {
          "name": "Alice",
          "age": 30
        },
        {
          "name": "Bob",
          "age": 25
        }
      ]
    }

Nested Tuple

Full example here

    {
      "name": "attributes",
      "schemaType": "optionalGroup",
      "logicalType": "INT32",
      "fields": [
        {
          "name": "key_value",
          "schemaType": "requiredGroup",
          "fields": [
            {
              "name": "key",
              "schemaType": "required",
              "physicalType": "BINARY",
              "logicalType": "STRING"
            },
            {
              "name": "value",
              "schemaType": "optional",
              "physicalType": "BINARY",
              "logicalType": "STRING"
            }
          ]
        }
      ],
      "data": [{"key_value": [{"key": "tqwqmcqvqo", "value": "gkqcl"}]}]
    }
  • repeatedGroup: Defines an array of objects.
  • requiredGroup and optionalGroup: Define tuple-like structures.

Clone this wiki locally