feat(core): detect corrupt log blocks and resume at the next one

[linliu-code]

Stacked on #639–#665 — review only the last commit.

What was there

fn create_corrupted_block_if_needed(&mut self, _pos: u64, _len: Option<u64>) -> Option<LogBlock> {
    // TODO: support creating corrupted block
    None
}

A corrupt or truncated block failed the entire log file. BlockType::Corrupted existed in the model but nothing could ever produce one.

What it does now

A well-formed Hudi log block records its total size twice — in the header length field and again in a trailing reverse pointer — and is followed by either another block or EOF. Three checks:

1. the trailing pointer has to lie inside the file
2. the size it records has to agree with the header
3. what follows the block has to be a magic marker or the end

Every offset is computed with checked_add/checked_sub, so a garbage length reports corruption rather than panicking or allocating against it. That is why the check runs before the body is parsed rather than after — a bogus content length would otherwise reach a vec![0u8; len].

When a block fails, the reader emits a corrupt marker and resumes at the next magic marker, found by scanning in 1 MB windows that overlap by MAGIC.len() - 1 so a marker straddling a boundary is not missed. One bad block now costs its own span instead of the rest of the file.

Adopted from the internal reader, which has carried this and the recovery tests for some time.

Tests

Two harness cases un-ignored and passing — a corrupt tail block, and a delete-ordering fixture that was failing behind the same read.

Three unit tests on the check itself, since the harness cases only exercise one shape:

• a length disagreeing with the trailing pointer is corrupt, and the real length is not — the negative case matters, since a check that always says "corrupt" would pass the positive one
• a length past EOF, and u64::MAX, are corrupt — decided arithmetically, never by reading there
• the recovery offset always lands within the file

Full workspace green: 1185 lib + 79 table-read + 39 datafusion + 21 + 12. Ignored 7 → 5.

Note

While verifying this I hit 117 unrelated failures that turned out to be a stale fixture-extraction cache under $TMPDIR/hudi-rs-test-fixtures, not the change — they reproduce on a clean tree. Clearing the directory fixes it. The cache key includes the zip's mtime, so it should self-invalidate; worth a look separately if it recurs in CI.

🤖 Generated with Claude Code