Skip to content

0.25.0

Choose a tag to compare

@lawrence-u10d lawrence-u10d released this 31 Jul 20:54
c38745b

Enhancements

  • Partition arbitrary JSON and NDJSON: partition_json() and partition_ndjson() now handle any valid JSON/NDJSON payload, not just serialized Unstructured output. Arrays (and NDJSON files) of serialized elements keep rehydrating as before; any other valid payload (bare objects, arrays of records, NDJSON lines, scalars) becomes Text elements containing the pretty-printed JSON instead of raising. The schema pre-gates in partition() are removed accordingly, a compact single-line JSON object now detects as FileType.JSON rather than NDJSON (JSON/NDJSON disambiguation examines at most the first 1 MiB of the file), and malformed input still raises ValueError (empty or whitespace-only documents yield no elements). One degraded case: NDJSON whose first record alone exceeds the 1 MiB disambiguation bound now classifies as JSON and fails partition() with ValueError (calling partition_ndjson() directly still handles it). Rehydration is chosen by an explicit shape predicate, with these consequences: an element-shaped payload whose contents cannot be rehydrated (e.g. corrupt metadata) raises ValueError with the underlying error chained, and an array (or NDJSON file) mixing element-shaped and arbitrary items partitions whole as arbitrary JSON - no partial rehydration that silently drops the arbitrary items. An empty JSON object yields one Text containing {} (an empty array yields no elements). One intended routing note: a one-record serialized-element file (a single object, not an array) routed through partition()/detect_filetype() now emits pretty-printed Text with alphabetized keys instead of rehydrating, since rehydration applies only to arrays (direct partition_ndjson() behavior is unchanged).
  • Serialized TableChunk elements now rehydrate: elements_from_dicts() (and with it partition_json() and partition_ndjson()) previously dropped serialized TableChunk elements silently because the type is not in the shared element-type map; it is now special-cased like CheckBox. This completes the table-reconstruction feature (#4291), whose reconstruct_table_from_chunks() expects deserialized chunks and now has a deserialization path to feed it. Behavior change: payloads of serialized chunked output containing split tables now return the TableChunk elements (previously omitted from results).
  • is_json_processable() and is_ndjson_processable() are deprecated: partitioning and file-type detection no longer route through these prefix-sniffing helpers. They keep working unchanged for downstream callers - now emitting a DeprecationWarning - and will be removed in a future release.