Skip to content

File Formats

themarioga edited this page Nov 8, 2026 · 1 revision

File Formats

Every format is read and written incrementally: a file is never loaded in memory. A JSON or XML record is materialized with its children, but the file never is. Files of 1 GB are imported with a 256 MB heap in the test suite.

CSV

  • Read with Commons CSV, with the configured separator, quote and encoding.
  • Strict decoding: an invalid byte fails the file with ENCODING and its physical row, after every earlier record has been returned (so STREAMING delivers them).
  • A UTF-8 byte order mark is removed.
  • Values are not trimmed when read (the field engine trims); rawValues keep them as they are.

Rows, header and preamble

Rows are numbered from 1 over the whole file, empty ones included; a quoted value with line breaks is one row.

  • With hasHeader = true the header is row firstDataRow - 1; without header, data starts at firstDataRow. Earlier rows are a preamble, skipped without looking at them.
  • A completely empty line is always skipped and does not count as a record.
  • A line of separators (;;) is a record with every field empty, unless skipBlankRecords = true, which also skips rows whose values are all blank (" ";;).
  • A file that ends before its header row has no records and no error.

Columns

columnLocation How fields find their column Checks
HEADER (default) By header name (column, or the field name), ignoring case and spaces An unknown header column, a duplicate one, or a missing column of a required field fail the file (STRUCTURAL); optional: true allows a missing column. An empty header cell is ignored with EXTRA_COLUMNS
POSITION The n-th input field reads the n-th column A header that does not match a field name gives HEADER_MISMATCH; extra columns give EXTRA_COLUMNS and are ignored
LETTER A, B… ZZZ Letters beyond the file's columns give MISSING_COLUMN and the fields are empty

Fields with inInput = false never consume a column. Two fields may read the same column; each gets its own chain of steps.

XLSX

  • Read from a file with random access (never from a stream, which would load the zip in memory). Provided streams are copied to the working directory first.
  • The sheet is streamed with StAX; POI provides the package, the shared strings and the styles.
  • POI's zip-bomb limits apply: a file above them fails with UNSAFE_CONTENT.
  • sheet selects a sheet by name; the default is the first. A missing sheet is STRUCTURAL.

Cell values

TIE reads the stored value, never the formatted one:

Cell Value read
Number Canonical: 42, 1234.5 (a cell shown as 1.234,50 € reads 1234.5)
Date format ISO: 2026-12-31, or 2026-12-31T10:00:00 if it has a time
Boolean true / false
Formula Its cached value
Text As is

A row whose cells have no value (only styles) is a blank row and is skipped. DATE, DATETIME and DECIMAL fields always accept the canonical form in addition to their typeFormat.

JSON

  • Records are the elements of the root array, or of the array under recordProperty.
  • Nested objects are flattened with dotted paths: {"a":1,"b":{"c":2}} gives a and b.c.
  • A field reads its path: column: customer.nif, or a field named customer.nif.
  • An array must be a declared child layout (nestedKey); any other array is STRUCTURAL.
  • null is an empty value. Numbers keep their text (1.50 stays 1.50).
  • "children": [] is a declared but empty container; an absent key or null is not declared (TypedRecord.childrenDeclared()).
  • Duplicate keys are STRUCTURAL.

XML

  • Records are the recordElement elements that are not inside another record.
  • Elements match by local name: namespaces and prefixes are ignored.
  • Attributes are @name (@id, customer.@type). The text of an element with children is ignored.
  • A child layout's nestedKey names a container element; each element inside it is a child record.
  • A repeated element that is not a child layout is STRUCTURAL (no value is silently lost).
  • A <!DOCTYPE> fails the file with UNSAFE_CONTENT without being processed: DTDs and external entities are disabled.
  • The encoding option is used to read the file (the XML declaration is not).

Writing

Format Shape
CSV Header with the field names (or column in HEADER mode) unless hasHeader = false; LETTER puts each field in its letter and leaves the others empty; records end with \r\n
XLSX Streaming workbook; text as text cells, NATIVE numbers, booleans and dates as typed cells
JSON The root array, or {"recordProperty": […]}; dotted paths become nested objects; an empty value is null
XML <rootElement> with one <recordElement> per record; fields are elements (dotted paths nest), @x are attributes; an empty value is an empty element or an omitted attribute; a child record is named by its layout's recordElement, or else its layout name

Exports never neutralize formulas: a value =1+1 is written as is. Result reports do (see Result Reports).

Clone this wiki locally