fix(core): convert an Avro map to an Arrow map - #664
Open
linliu-code wants to merge 3 commits into
Open
Conversation
Squashed view of apache#639-apache#662 for review. Not for merge — the reviewable increments are those PRs; this is the same code in one diff. Ports the merge-on-read file group reader from onehouseinc/hudi-rs-internal into hudi-core, wires it behind a switch that defaults to the reader that has always served reads, and brings its end-to-end test harness across. The ported reader is `pub(crate)` and reached only through `hoodie.read.merge.engine = v2`. Nothing changes for anyone who does not set it. It is not at parity yet: the gaps are pinned as ignored cases carrying their findings, and the outstanding decisions are in the PR descriptions. Four fixes land on paths the existing reader shares: decimals had no Arrow conversion, Avro timestamp logical types lost their UTC zone, the properties-escaped create schema was not being unescaped by one of its two consumers, and the base file reader had no way to accept a pushdown predicate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A delete block whose ordering value is anything but a small integer fails to
read, with errors like `Union index 1490 out of bounds: 2`. The index is
nonsense because the byte stream is misaligned, not because a branch was chosen
wrongly.
`orderingVal` is declared here as a union of primitives. Hudi writes a union of
per-type wrapper records, and inserted `BooleanWrapper` at position 1, so every
position from `int` onward names a different type than this crate assumes.
Position 3 is `float` here and `LongWrapper` there. Reading a long as a float
consumes four bytes instead of two, and the next record's key length is read
from the middle of the previous value.
The delete block in `table_delete_ord_long` is exactly self-consistent under
Hudi's schema and not under this one:
04 | 02 02 34 | 02 00 | 06 c0 3e | 02 02 33 | 02 00 | 06 f0 2e | 00
two records, keys "4" and "3", ordering position 3, values 4000 and 3000
Decoded here, position 3 is a float, so `c0 3e 02 02` is eaten and `33` becomes
the next key's union index: zigzag 0x33 is -26, which is the reported error.
So this takes Hudi's schema. Two things follow from it:
The Arrow side wants a scalar, not a record with one field, so the wrapper is
unwrapped after the schema is narrowed — narrowing reads the position Hudi
wrote, and unwrapping rewrites it, so the order matters.
The wrapper is chosen for the value rather than for the column, so a table
whose ordering column is a long can still carry an `IntWrapper` for a small
value. The delete batch's ordering column is now cast to the type the data
schema declares. The old schema hid this by calling position 2 a long
regardless, which happened to match the two fixtures that exercise it.
`ArrayWrapper` orders by a list and is rejected in both places that read the
position, rather than mapped to something that would disagree with its value.
Older tables written with the primitive union are not supported. No fixture
here uses one, and the two shapes cannot be told apart from the bytes: where
they differ, decoding usually fails, and at position 2 both succeed and yield
the same number.
Un-ignores the eleven cases that pinned this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An Avro map was modelled as `Dictionary(Utf8, V)`. An Arrow dictionary key must be an integer, so that is not a valid type — and it does not reconcile against the `Map` a parquet base file carries, so any table with a map column fails to read once a log block has to be merged with its base file. Avro maps become `Map(key_value: struct<key: string, value: V>)`, matching what the parquet reader produces, so the two agree by name and by shape. The array side builds a `MapArray`. Entries are materialized as two-field records so the existing struct machinery builds both children, which is also why `child_schema_lookup` now registers those two positions — a struct-valued map needs its own fields resolvable underneath them. Entries are emitted in key order. Avro maps are unordered and the Arrow type says so, but a stable order keeps a read reproducible rather than dependent on hash iteration. This is on the shared conversion, so it fixes the existing read path too: a map column has never been readable through either reader. Un-ignores the case covering NULL elements inside containers. Two other cases that were pinned on this stay pinned, now on a decimal column reading as NULL — a separate gap this one was masking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #639–#663 — review only the last commit.
The bug
avro_to_arrow/schema.rsmodelled an Avro map as:An Arrow dictionary key must be an integer type.
Dictionary(Utf8, V)is not a valid Arrow type — not merely an odd choice. And it cannot reconcile against theMapa parquet base file actually carries:So any table with a map column fails as soon as a log block has to merge with its base file. This affects the existing reader too — it is a shared conversion, and a map column has never been readable through either.
The fix
An Avro map becomes
Map(key_value: struct<key: string, value: V>, sorted=false)— the same shape and the same entry-field name the parquet reader produces, so the two agree by name.On the array side,
build_map_arrayconstructs a realMapArray. Entries are materialized as two-field records (key,value) so the existing struct machinery builds both children; that is also whychild_schema_lookupnow registers those two positions, since a struct-valued map needs its own fields resolvable underneath them.Entries are emitted in key order. Avro maps are unordered and the Arrow type says
sorted = false, but a stable order keeps a read reproducible rather than dependent on hash iteration.What it unblocks
harness_null_container_elements— maps and lists containing NULL elements — now passes. That was the hardest of the three map cases.The other two (
harness_all_data_types,harness_mixed_column_types) still fail, but on something else: a decimal column in a log block reads as NULL where the Spark snapshot has a value. The map error was failing those cases earlier and hiding it. They stay pinned, now carrying that finding instead.This also removes the reason base-file-only slices are routed away from the merge-on-read engine in #659 — that guard exists purely because every parquet fixture here has a map column.
Tests
Two direct conversion tests: a map of primitives produces
Mapwith a non-nullablekey_valueentry struct and a non-nullableUtf8key; a map of records keeps the value as a nested struct rather than collapsing it.Plus the harness case above, which exercises the array builder end to end against a Spark snapshot.
Full workspace green: 1178 lib + 79 table-read + 39 datafusion + 21 + 12. Ignored 8 → 7.
🤖 Generated with Claude Code