Skip to content

fix(arrow-avro): reset union offsets after flush - #10971

Merged
Jefffrey merged 1 commit into
apache:mainfrom
linhongyu510:fix/avro-union-decoder-batch-offsets
Sep 3, 2026
Merged

fix(arrow-avro): reset union offsets after flush#10971
Jefffrey merged 1 commit into
apache:mainfrom
linhongyu510:fix/avro-union-decoder-batch-offsets

Conversation

@linhongyu510

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

UnionDecoderBranches::counts tracks the next dense-union offset for each branch. flush drained the type IDs, offsets, and child decoders but left these counters at the previous batch totals, so the next batch referenced positions beyond its newly emptied children.

What changes are included in this PR?

  • Reset all per-branch union counters after UnionArray::try_new succeeds.
  • Add a regression test that decodes and flushes two consecutive batches through the same dense union decoder and verifies each batch starts at offset zero.

The reset intentionally happens only after successful array construction, so a failed flush does not additionally mutate the counters.

Are these changes tested?

Yes. TDD evidence on current main:

  • Before the production change, the second flush failed with Offsets must be non-negative and within the length of the Array.
  • After the change, the focused regression passes.
  • cargo test -p arrow-avro --all-features: 487 passed.
  • Doc tests: 27 passed, 1 repository-marked ignored.
  • cargo fmt --all -- --check: passed.
  • cargo clippy -p arrow-avro --all-targets --all-features -- -D warnings: passed.
  • git diff --check: passed.

The first all-features run had 399 passes and 88 failures because the official testing submodule was not initialized; every failure referenced missing testing/data/avro fixtures. After git submodule update --init --depth 1 testing, the complete suite passed as reported above.

Are there any user-facing changes?

Yes. A decoder containing a multi-branch Avro union can now emit multiple batches without producing stale dense-union offsets. There are no API changes.

AI assistance

I used AI assistance to investigate the decoder state, implement the focused regression and fix, and prepare this description. I reviewed the complete diff and verified all reported commands and outputs.

@github-actions github-actions Bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate labels Sep 3, 2026
@linhongyu510
linhongyu510 marked this pull request as ready for review September 3, 2026 06:56
@Jefffrey Jefffrey added the bug label Sep 3, 2026
@Jefffrey
Jefffrey merged commit 300c772 into apache:main Sep 3, 2026
33 checks passed
@Jefffrey

Jefffrey commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

thanks @linhongyu510

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-avro arrow-avro crate bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[arrow-avro] Decoder produces invalid offsets in the second batch when the schema has a multi-branch union

2 participants