Step 3: Ingest and normalization (#17)

[craig-dt]

Closes #17. PLAN.md step 3, built test-first against docs/spec.md §8 "Ingest".

What this builds

src/flabel/ingest.py — normalize(capture: Path, workdir: Path) -> NormalizedCapture, in spec §8's order and no other:

1. Sniff by magic bytes, never by extension. All four pcap magics (both byte orders × micro/nanosecond), pcapng, gzip. A gzip payload is sniffed again after decompression: gzip says nothing about what it wraps, and a .pcap.gz holding a pcapng is ordinary.
2. Decompress gzip to an intermediate in the workdir, streamed.
3. Validate by walking record headers ourselves — pcap records and pcapng blocks — because no tool in the dependency set reports a truncation offset. Yields packets_read and truncated_at_offset. Driven by seeks and header reads, so cost scales with packet count, not capture size.
4. Unreadable header → CaptureError, no output.
5. Truncated pcap → input_status = "partial" with the offset recorded.
6. Truncated pcapng → hard failure naming the editcap repair command.
7. pcapng → editcap -F pcap. Mixed link types: dominant type by packet count is kept, the rest are discarded into discarded_link_types / discarded_packets with input_status = "partial".
8. Every transformation recorded in normalization.

tests/fixtures/make_awkward.py extends make_canary.py: truncated pcap (mid-payload and mid-record-header), truncated pcapng, multi-datalink pcapng in four variants, bad header, corrupt gzip, the four pcap byte-order/precision encodings, and gzipped versions of each. Nothing is committed as a binary — tests generate into tmp_path, and write_plain_pcap reproduces the committed benign.pcap byte for byte, which is asserted.

Decisions that go beyond the plan's wording

Each is a consequence of what the tools actually do, verified rather than assumed.

1. A truncated pcap's incomplete tail record is dropped from the normalized copy. The plan says "proceed, input_status = partial". That is impossible if the bytes pass through untouched: on the pinned toolchain, Zeek 8.0.9 exits 1 with fatal error: failed to read a packet ... truncated dump file, and Suricata 8.0.6 logs pcap: error code -1 truncated dump file. Passing it through would convert a loss condition the spec says to report into a tool failure that ends the run. packets_read and truncated_at_offset state exactly what was dropped and where, so the trim removes bytes, not information. A test asserts capinfos rejects the input and accepts the output.

2. Multiple link types are detected from our own walk, not from editcap's complaint. Spec §8 step 7 says "if it reports multiple link types". editcap reports it as an English error string (The capture file being read can't be written as a "pcap" file.), and branching on parsed tool prose is fragile across versions. The offset-accurate walk already knows every interface's link type and packet count, so the decision is made from data.

3. Discarding link types requires editcap -T <encap>. Verified on Wireshark 4.6.7: editcap picks the output encapsulation from the interface description blocks, not from the packets that survive -r selection — so a pcapng that ever declared two link types still refuses to become a pcap even after the minority packets are dropped, whether the selection happens in one step or two. The kept packets are selected as 1-based ranges (-r) and the encapsulation is asserted with -T. Since -T is a claim about the kept packets, it is only ever passed a name for the type those packets actually carry: LINK_TYPES maps libpcap link types to editcap's own names, and a dominant type absent from that table is a hard CaptureError with instructions rather than a mislabelled capture. Tested with USER0 (147).

4. Ties in "dominant by packet count" break to the lowest link type. A tie has no meaningful winner, so the rule is chosen to be stable: Goal 2 needs two runs over one capture to keep the same packets, which dict order would not guarantee. Asserted by comparing two runs, not by inspecting the rule.

5. editcap's output is re-walked and its packet count checked. Exit 0 is not evidence that the requested packets were written, and a silently short conversion would surface downstream as flows that never existed rather than as an ingest failure.

6. EditcapError(ToolError) carries the ToolFailure. Spec §11 wants a tool failure in tool_failures[] and a hard failure, but NormalizedCapture has no field for one (a normalize that fails returns nothing at all), and models.py/errors.py are read-only in this step. So the record travels on the exception: cli.py reads .failure, and anything catching plain ToolError still gets exit 1. Flagged for step 9 — see open questions.

Open questions, not guessed at

• sha256 / bytes_total describe the input as handed to us, so a compressed capture hashes as the compressed file, consistent with capture_format reporting pcap.gz. Meanwhile NormalizedCapture.path is the derived file. Spec §10's run block has a single input.path, so step 8 must decide whether that key carries original_path or path — one of the two facts is otherwise unrecorded in labels.json. My reading: original_path, since sha256, bytes and format all describe the original.
• truncated_at_offset is an offset into the uncompressed capture. A gzip member has no record boundaries, so nothing else would be actionable. Worth one clause in docs/spec.md §10 or §11 if the field is documented for consumers.
• tests/fixtures/README.md does not mention make_awkward.py and still describes only the two canaries. Not edited (read-only in this step); it should gain a short section on the generated awkward fixtures.
• No new model field was needed — NormalizedCapture and ToolFailure as landed in step 2 cover everything here.

How it was tested

uv run pytest -q → 162 passed, 8 skipped (baseline before this branch: 120 passed, 8 skipped; no change to the skip set). uv run pytest -q --require-tool-tests green, so the tool layer genuinely executed. uv run ruff format . && uv run ruff check . clean.

42 new tests. editcap and capinfos run for real (14 marked requires_tools), with capinfos used as an independent oracle for packet counts and encapsulation — checking flabel's walk against flabel's own reader would prove only that the code agrees with itself. No test touches the network, and one asserts that at runtime by making socket.socket raise for the duration of a normalize.

Coverage of the plan's required outcomes:

Required outcome	Test
truncated pcap → partial with the correct offset	offset from the generator, again from independent arithmetic, and again for a cut inside the record header
truncated pcapng → CaptureError, no output	message carries the offset and editcap; workdir does not exist
bad header → CaptureError, no output	plain, and gzipped (the one path that has already written to disk before it can validate)
multi-datalink → dominant kept, discards recorded	kept frames compared to the originals; capinfos confirms count and encapsulation; dominance re-tested with the majority flipped; tie-break re-tested across two runs
gzip transparent	pcap, pcapng, and a truncated pcap inside gzip
plain pcap → byte-identical	plus all four magic variants, plus workdir holding only normalized.pcap
tool failure recorded	missing binary via emptied PATH (spec §11's fault injection) and a real non-zero editcap exit
conversion reproducible	two runs byte-identical
fixtures deterministic	every fixture generated twice and byte-compared; gzip header mtime asserted zero

Also verified by hand, outside the suite: Zeek 8.0.9 reads both the trimmed truncated output and the split multi-datalink output cleanly (conn.log written, no errors).

🤖 Generated with Claude Code

[craig-dt]

Review findings addressed in 0e1f547. CI green: 175 passed, 9 skipped in the toolchain container (4 of those skips are the tshark-dependent tests, see finding 5); locally with tshark installed, 176 passed, 8 skipped. ruff clean. Same three files, nothing outside them.

#	Finding	Fix
1	Atomicity hole	_sha256, stat and the NormalizedCapture construction moved inside the try. Cleanup now removes every directory mkdir(parents=True) created, deepest first, and only while empty. Two new tests: a late failure (injected by making the hash raise) leaves nothing, and a nested runs/2026/08/capture_1200 workdir leaves no runs/ behind — plus one asserting a pre-existing ancestor survives.
2	Messages named a deleted file	Every message raised after decompression now carries a subject — the operator's path, marked (decompressed) — and the pcapng repair command names the original file. editcap reads gzipped captures directly (verified: it repaired a .pcapng.gz and salvaged 5 packets), so the original is also the right file to repair. New test asserts the command names the .gz and that decompressed.capture appears nowhere in the message.
3	struct.error escaping	Block bodies are length-checked against each type's mandatory fields before unpacking (MIN_BODY_BYTES), and walk() has an outer struct.error → CaptureError guard. Six new tests: bad total_length, oversized packet block, trailer mismatch, undefined interface reference, a 12-byte IDB with no body, a 12-byte pcap file header — and a big-endian pcapng, which had no fixture.
4	16 MiB bound too broad	Renamed MAX_PACKET_BYTES and applied to packet blocks only. Bodies are now peeked at (32 bytes) and seeked past rather than read, so a large block is never loaded. Tested with a real 17 MiB decryption secrets block: walks clean, capinfos agrees, and normalization succeeds.
5	Range selection breaks on interleaving	Replaced with tshark. One correction: the one-shot tshark -r in -Y ... -F pcap -w out.pcap does not work — tshark picks the output encapsulation from the interface description blocks exactly as editcap does, so it refuses a per-packet-encap file even with a filter applied (rc=9, "can't be written as a pcap file"), and filtering to pcapng first does not help because the unused IDBs survive. So it is two steps: tshark -Y '<ids>' -F pcapng then editcap -F pcap -T <encap>, and -T plus its name table stays. The filter is a disjunction over interface ids, because one link type can span several interfaces — a single-interface_id filter would have silently dropped 4 Ethernet packets in the new interleaved fixture. MAX_SELECTION_RUNS is gone; nothing now scales with interleaving.
6	Fabricated ToolFailure	_verify_converted now passes the real argv and exit_code=None (the process did exit 0; the message carries the contradiction). _run_editcap → _run_tool, and EditcapError → ConversionError, since tshark can raise it too. The missing-binary test is parametrised over both tools and asserts the recorded tool names the one actually missing.
7	make_awkward.py outdir default	Removed; a missing argument prints usage and exits 2. The 17 MiB fixture is excluded from write_all's command-line output so the documented command stays cheap.

New fixtures: interleaved multi-datalink (3 interfaces, 2 of them Ethernet, alternating), big-endian pcapng, short pcap header, bad block length, trailer mismatch, undefined interface, short block body, large decryption secrets block, gzipped truncated pcapng.

Requested notes for the spec §10 amendment — what the code actually does:

• packets_read counts complete records/packet blocks in the (decompressed) input. An incomplete tail record is neither counted nor added to discarded_packets; truncated_at_offset is its only report, and exactly one record is implied (the bytes from that offset to EOF). discarded_packets counts link-type discards only. So: packets in normalized.pcap == packets_read - discarded_packets, and a truncated capture legitimately shows discarded_packets: 0.
• truncated_at_offset is a byte offset into the uncompressed capture — a gzip member has no record boundaries for an offset to refer to. For a .pcap.gz the offset therefore does not index the file on disk, which a consumer reading input.path alongside it needs told.
• sha256 / bytes_total unchanged from the original PR: they describe the input as handed over, so a .gz hashes as the compressed bytes.

Also unchanged and still open for step 8: spec §10's run block has one input.path while NormalizedCapture carries both original_path and the derived path.