You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Enhancements
Partition arbitrary JSON and NDJSON: partition_json() and partition_ndjson() now handle any valid JSON/NDJSON payload, not just serialized Unstructured output. Arrays (and NDJSON files) of serialized elements keep rehydrating as before; any other valid payload (bare objects, arrays of records, NDJSON lines, scalars) becomes Text elements containing the pretty-printed JSON instead of raising. The schema pre-gates in partition() are removed accordingly, a compact single-line JSON object now detects as FileType.JSON rather than NDJSON (JSON/NDJSON disambiguation examines at most the first 1 MiB of the file), and malformed input still raises ValueError (empty or whitespace-only documents yield no elements). One degraded case: NDJSON whose first record alone exceeds the 1 MiB disambiguation bound now classifies as JSON and fails partition() with ValueError (calling partition_ndjson() directly still handles it). Rehydration is chosen by an explicit shape predicate, with these consequences: an element-shaped payload whose contents cannot be rehydrated (e.g. corrupt metadata) raises ValueError with the underlying error chained, and an array (or NDJSON file) mixing element-shaped and arbitrary items partitions whole as arbitrary JSON - no partial rehydration that silently drops the arbitrary items. An empty JSON object yields one Text containing {} (an empty array yields no elements). One intended routing note: a one-record serialized-element file (a single object, not an array) routed through partition()/detect_filetype() now emits pretty-printed Text with alphabetized keys instead of rehydrating, since rehydration applies only to arrays (direct partition_ndjson() behavior is unchanged).
Serialized TableChunk elements now rehydrate: elements_from_dicts() (and with it partition_json() and partition_ndjson()) previously dropped serialized TableChunk elements silently because the type is not in the shared element-type map; it is now special-cased like CheckBox. This completes the table-reconstruction feature (#4291), whose reconstruct_table_from_chunks() expects deserialized chunks and now has a deserialization path to feed it. Behavior change: payloads of serialized chunked output containing split tables now return the TableChunk elements (previously omitted from results).
is_json_processable() and is_ndjson_processable() are deprecated: partitioning and file-type detection no longer route through these prefix-sniffing helpers. They keep working unchanged for downstream callers - now emitting a DeprecationWarning - and will be removed in a future release.