fix(xz): deterministic async XZ decoder wakeups, no worker deadlock [DEV-124]#70
Merged
Merged
Conversation
…DEV-124] The async XZ path drove the worker-backed decoder through the blocking Codec::process, whose wait_event() parks the executor thread on the worker's event channel. Under a slow, one-byte-at-a-time async source the executor could park with no registered waker and the worker could finish between the owner's try_recv and its park, losing the wakeup and hanging. Add a non-blocking Codec::poll_process (defaulting to process) and route the async adapter through it. In the XZ codec, refactor process() into a shared drive(.., waker: Option<&Waker>): with None it keeps the byte-for-byte blocking behavior; with Some it registers the waker before the deciding try_recv and returns Poll::Pending on Empty. A cloneable EventSink wakes the async owner after every NeedInput/Output send, and decode_worker wakes the retained waker Arc only after all senders drop, so a parked owner's next try_recv observes Disconnected instead of Empty. Add a park/unpark std executor plus a process-abort watchdog to the async regression suite and a focused async_xz_never_deadlocks_under_slow_source test (50 iterations over the one-byte source); route the three at-risk existing tests through the same guarded executor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P4suta
enabled auto-merge (squash)
July 23, 2026 20:06
P4suta
added a commit
that referenced
this pull request
Jul 23, 2026
…Zstd, AES-256 [RM-303] (#71) * feat(7z): multi-folder coder graph, one active decoder [RM-303] Replace the single-folder 7z seek-reader model with a Vec<Folder>: - parse_streams_info now yields one FolderInfo per folder. Drops the num_pack != 1 and NumFolders == 1 refusals; a single coder still means one pack stream per folder, so pack streams and folders pair 1:1 and each folder gets its own absolute pack offset from the shared base. - Generalize SubStreamsInfo across folders: per-folder substream sizes (single-substream folders omit their stored size) and the CRC digest count rule numDigests = Σ (n==1 && folderCrc ? 0 : n). Folder CRC presence is tracked per folder via read_digests_defined. - FileRec carries (folder_index, offset_in_folder); content files draw down a flat folder-then-file substream list, resetting the per-folder offset on each folder change. - SevenZSeekReader keeps exactly one folder decoder live at a time. It is built lazily on first use of each folder and torn down (reclaiming the raw source, freeing the codec workspace) before seeking to and building the next — bounded memory regardless of folder count. Each folder is finalized (exact size, no trailing bytes) before switching and at end of stream. - Encoded next-header decoding keeps its single-folder restriction. Still LZMA/LZMA2, single coder per folder; the writer is unchanged (one solid LZMA2 folder). #![forbid(unsafe_code)] holds; no new deps. Differential tests: non-solid archives (each file its own folder) from sevenz-rust2 via push_archive_entry, for both LZMA2 and plain LZMA, with interleaved directory/empty-file entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(7z): general coder-graph parse + linear-chain decode infra [RM-303] Landing step 2 of the 7z coder-graph work. Pure infra: no behavior change for existing archives (differential/interop tests unchanged). Parser: - Replace the single-coder FolderCoder model with a spec-complete coder graph (Coder/BindPair/Folder). read_folder now parses any number of coders, bind pairs, and packed-stream indices; read_coder handles complex (multi-in/out) coders and coders without attributes; classify_method keeps only LZMA/LZMA2 decodable, every other id lists as Unsupported. - parse_streams_info reads one CodersUnpackSize per output stream across all folders, assigns each folder its own back-to-back pack streams (multi-pack aware), and derives each folder's final-output (unpack) size. - resolve_folder computes a topological decode order via post-order DFS with in-progress cycle detection, and rejects stream overlaps (an input fed by more than one source or none, an output bound twice) as typed Malformed. Decode: - SevenStage: a recursive, statically dispatched Read chain (Source pack stream wrapped by each coder's reader). build_folder_reader folds decode_order into the chain; only linear LZMA/LZMA2 folders over one pack stream are built, so BCJ2/PPMd/multi-in-out list but report Unsupported at build. Still exactly one decode chain live at a time (bounded memory). Codec infra: - Generalize PipelineRead into codec_read::CodecReader<Inner: Read, C: Codec>; filtered_io reuses it for zstd/LZ4 via a thin PipelineRead alias, and PipelineCodec now implements Codec. Verified: build -p libarchive_oxide with sevenz, "sevenz aes", --no-default-features --features sevenz, and default (zstd/lz4); xtask codec-policy (portable stays C/FFI-free) and no-dyn (static dispatch) pass; clippy clean; sevenz_differential, interop_foundation, and fuzz_replay green. #![forbid(unsafe_code)] holds. Cross/qemu/macOS matrices deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(7z): delta and BCJ decode filters in the coder graph [RM-303] Landing step 3 of the 7z coder-graph work: decode-only delta and BCJ byte filters, wired into the folder decode chain so a linear graph that chains them with LZMA2/LZMA over one pack stream now extracts (previously listed as Unsupported). Exactly one folder decoder stays live; bounded memory and static dispatch are unchanged. Filters (new filter/delta.rs, filter/bcj.rs, under the sevenz feature): - DeltaDecoder: per-byte reconstruction, distance 1..=256 over a 256-byte history ring. Fully streaming (no held state), so any chunking matches. - BcjDecoder: BranchKind covering x86 (stateful E8/E9 with running mask and IP), the 4-byte-stride RISC families (ARM/ARM64/PPC/SPARC), 2-byte ARM Thumb, 16-byte IA-64, and variable-width RISC-V. Instructions straddling a chunk boundary are held back and re-scanned; at end of input the remaining tail is emitted untransformed, exactly as the encoder left it. Decode is a bit-for-bit inverse of the reference encoders. - Both implement the sans-I/O Codec and are driven by the shared CodecReader, so dispatch stays static. codec_read is now compiled under sevenz too, and its struct bounds moved onto the impls so SevenStage can nest it. Parser/decode (sevenz.rs): - classify_method recognizes the delta id (0x03, distance = prop + 1) and the BCJ ids (x86/PPC/IA64/ARM/ARMT/SPARC long-form, ARM64 0x0A, RISC-V 0x0B), mapping them to CoderMethod::Filter(SevenFilter::{Delta,Bcj}). Unknown property shapes still list as Unsupported. - SevenStage gains Delta/Bcj stages wrapping CodecReader over the stage below; wrap_coder builds them. A filter coder is single-in/single-out, so a folder chaining it with LZMA2/LZMA stays a linear decodable chain. Tests: - Differential vs sevenz-rust2: delta+LZMA2 (distances 1/4/256) and BCJ+LZMA2 (x86/ARM/PPC/SPARC) archives are produced by sevenz-rust2 and decoded by arca to byte equality, cross-checked against sevenz-rust2's own reader. - Filter unit tests decode every branch kind and every delta distance against lzma_rust2's independent encoder/decoder, across tiny (1/3-byte) and bulk output buffers and small/large input chunks, plus near-window-boundary lengths — proving chunk-size independence. Verified: build -p libarchive_oxide with sevenz, "sevenz aes", and --no-default-features --features sevenz, plus default; sevenz_differential and filter unit tests green on each; clippy clean; xtask no-dyn and codec-policy pass (no new deps; portable stays C/FFI-free). #![forbid( unsafe_code)] holds. Cross/qemu/macOS matrices deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(7z): reuse Deflate/BZip2/Zstd codecs in the coder graph [RM-303] Step 4 of the 7z coder-graph landing: decode the general-purpose 7z content coders by reusing arca's existing codecs through the shared PipelineCodec dispatch, wrapped as a new SevenStage::Pipeline chain stage (no second parser; ADR-0004 preserved). One active folder decoder, static dispatch, and #![forbid(unsafe_code)] all hold — the codecs live in existing deps. - core: add FilterId::Deflate (raw RFC 1951, no framing, no probe signature). - gzip: add RawInflateDecoder, a bare raw-DEFLATE Codec over the same miniz_oxide raw-inflate core GzipDecoder sits on, with a decoded_total guard. - pipeline_codec: add the FilterId::Deflate arm (sevenz-gated). - sevenz: classify methods 04 01 08 (Deflate), 04 02 02 (BZip2), 04 F7 11 01 (Zstd) into SevenFilter::Pipeline; BZip2/Zstd list-only when their features are off. wrap_coder builds the pipeline stage and bounds BZip2/Zstd working sets against codec_memory (validate_pipeline_memory), mirroring the LZMA dictionary check; the native zstd backend additionally caps the window. - tests: differential decode of sevenz-rust2 Deflate/BZip2/Zstd folders, the latter two gated on arca's bzip2/zstd features; dev-dep sevenz-rust2 gains the deflate + zstd features to produce those folders. Verified: cargo build -p libarchive_oxide --features sevenz, "sevenz aes", and --no-default-features --features sevenz; cargo build -p xtask; xtask codec-policy (portable stays C/FFI-free) and no-dyn; cargo test --features sevenz (and "sevenz aes") --test sevenz_differential (11 passed); clippy clean on touched files. Cross/qemu and macOS matrices deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(7z): AES-256/SHA-256 decryption coder [RM-303] Add the 7z AES-256/SHA-256 encryption coder (method 06F10701) to the single seek parser's coder graph, gated on the `aes` feature. * New `filter/aes7z.rs`: parses the coder properties (numCyclesPower/salt/IV; 0x3F external-key marker and out-of-range work factors list but do not decode), derives the key with 7-Zip's own KDF (a single SHA-256 context iterating salt || password || counter 2^ncp times, password encoded as UTF-16LE — the ZIP AE-2 path instead hashes raw bytes via PBKDF2-HMAC-SHA1), and decrypts with cbc::Decryptor<Aes256> and the stored IV. Trailing ciphertext padding is truncated against the coder's declared output size rather than erroring. * Thread Option<SecretBytes> from SeekArchiveReader through SevenZSeekReader::new and the seek dispatch site (previously dropped) down to the coder build. A missing password on an AES folder is a typed Unsupported error; secrets are never logged. * Retain per-substream CRC-32 (previously discarded in the digest reader) and verify it after decode; a mismatch is ErrorKind::Integrity, reported as "wrong password or corrupt data" on an encrypted folder. * Add `cbc` to the `aes` feature. Differential coverage vs sevenz-rust2 AES256SHA256 + Password: correct password decodes the encrypted header and content to plaintext; a wrong password trips Integrity; a missing password is Unsupported. Static dispatch, bounded memory, and #![forbid(unsafe_code)] are preserved; portable-codecs stays C/FFI-free. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(7z): docs, structured coder-graph fuzz, and deferred seams [RM-303] Close RM-303 with the accountability + fuzz layer over the 7z coder graph: - Support matrix: rewrite the 7z row to list the full coder-graph method set (LZMA/LZMA2, Delta, BCJ family, Deflate, BZip2, Zstandard; multi-folder and general graphs), AES-256 in its own encryption column, and PPMd/BCJ2 as structured Unsupported. Add PPMd and BCJ2 deficit rows to the deficit table. - ADR-0012: add tracked-deficit rows for PPMd (new decoder deferred) and BCJ2 (multi-stream decode deferred) as typed Unsupported with declared resolution paths; refresh the "entire ledger" note. - fuzz: add the structured read_7z_graph target. It synthesizes a random 7z StreamsInfo (coder count, complex flags, arities/props, bind pairs, packed indices, coder/substream sizes) from `arbitrary`, frames it under correct start/next-header CRCs so the graph parser is always reached, and asserts the RM-303 invariant: no panic, bounded work, and only typed Malformed/Unsupported/ Integrity/Limit errors — stressing cycles, stream overlaps, and truncation (recomputed under the header CRC). Body lives in the portable fuzz_lib and is wired into TARGETS/run_target, the libFuzzer shim, and fuzz/Cargo.toml. - Seed fuzz/corpus/read_7z_graph with six differential-scenario archives (arca solid, sevenz-rust2 multi-folder, delta+LZMA2, BCJ+LZMA2, LZMA, Deflate). Verified: cargo build -p libarchive_oxide --features sevenz / "sevenz aes" / --no-default-features --features sevenz; cargo build -p xtask; codec-policy (portable stays C/FFI-free); sevenz_differential (14/14, incl. aes); fuzz_replay (3/3 — 76960 mutants + arbitrary batch across 18 targets uphold the invariant). Building the cargo-fuzz bin needs nightly + sanitizer and is deferred to CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(7z): cover Deflate variant in async poll_process after DEV-124 rebase [RM-303] The DEV-124 async poll_process dispatch (merged as #70) matches PipelineCodec variants explicitly; the sevenz Deflate variant added on this branch left that match non-exhaustive once rebased. Add the sevenz-gated Deflate arm, delegating to the default poll_process (RawInflateDecoder is an in-process codec). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(7z): zeroize AES key schedule and drop the password copy [RM-303] Two hardenings for the security-sensitive 7z AES path (CodeQL flagged the area): - Enable the `zeroize` feature on the `aes` crate so the expanded AES-256 key schedule held inside the CBC decryptor is wiped on drop. Previously the derived key was zeroized but the cipher's round keys lingered in freed memory for both the 7z CBC and ZIP AE-2 CTR paths. - Rewrite `password_utf16le` over `slice::utf8_chunks` instead of `String::from_utf8_lossy`, so a password with invalid UTF-8 is never copied into an intermediate owned `String` that escapes zeroization; `chunk.valid()` borrows from the caller's buffer. Behaviour is byte-identical (one U+FFFD per maximal invalid subsequence), locked by a new equivalence test against the lossy path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the flaky async XZ decoder hang that intermittently timed out CI (
big-endian (s390x, qemu),test (macos-latest),test (ubuntu-latest)), blocking the RM-400 all-CI-green gate. Linear: DEV-124.Root cause
The portable XZ decoder (
filter/xz.rs) drives a blocking pull decoder (lzma_rust2::XzReader) on an OS worker thread over twosync_channel(2)channels.Codec::processmixes non-blockingpoll_event(try_recv) with a blockingwait_event(events.recv()). Because the async adapterAsyncCodecReader::poll_readcallsprocessinline,wait_eventblocks the executor thread. Once theNeedInputevent has been drained by an earliertry_recv, the worker waits for input while the caller waits for an output event with nothing in flight — a mutual wait that slow qemu/loaded-macOS scheduling turns into a ~14-minute hang. Native (liblzma) builds are unaffected.Fix — worker→task waker + non-blocking codec drive
Codecgains a defaultedpoll_process(.., waker)(non-worker codecs inherit it; only XZ overrides).EventSink { SyncSender, Arc<Mutex<Option<Waker>>> }that wakes the registered task waker after every event send (bothNeedInputandOutput).processis refactored intodrive(waker: Option<&Waker>):Nonekeeps the byte-for-byte blocking sync path;Some(w)registers the waker (viaWaker::will_wake) before the decidingtry_recvand returnsOk(None)→Poll::PendingonEmpty.decode_workerruns the decode loop in an inner scope so all event senders drop (channel disconnects) before it wakes via a retained wakerArc— so a parked owner's nexttry_recvalways observesDisconnected, never a lostEmpty.pipeline_codec.rsmirrorspoll_process;async_filter.rs::poll_readcallspoll_process(.., cx.waker())and mapsOk(None)→Poll::Pending.No
unsafe, static dispatch preserved, bounded memory unchanged, syncPipelinebehavior byte-for-byte identical,Sync/RefUnwindSafepreserved. Not a skip — the deterministic wakeup is the fix.Tests
New
async_xz_never_deadlocks_under_slow_sourcedrives the async XZ reader over a 1-byte-per-poll source × N=50 under a park/unpark executor and a process-abort watchdog (a hang fails fast instead of eating the 15-min budget); the three previously at-risk tests are routed through the same harness.async_stream_v2passes on portable and native profiles.🤖 Generated with Claude Code