Skip to content

A pcap/pcapng reader that takes bytes, not filenames #100

Description

@EONRaider

Target: 2.2.0 (Tier 4 — proof as the product)

A no-I/O codec currently has to hand-wave the question "how do I get
bytes?". We already answer it internally: tests/conftest.py:17-35 has
a working classic-pcap reader, explicitly labelled test-only. RootWire
needs the same thing. So does anyone evaluating the library.

Reading a file is not sniffing. Accepting a buffer keeps the
purity rule intact: the caller does the I/O, we do the parsing.

This is an open slot, not a chore

There is no maintained, well-typed, pure-Python pcap+pcapng reader on
PyPI. python-pcapng (rshk) was last released 2022-08-23;
pcapng (cloojure) 2020-09-10. Everything else lives inside dpkt
(frozen since 2024, untyped) or scapy (21 MB, GPL). A typed,
zero-dependency reader would be genuinely new.

Format notes for whoever implements it

Classic pcap is a 24-byte header plus 16-byte record headers; four
possible magic byte-orders (0xA1B2C3D4 µs, 0xA1B23C4D ns, each in
either endianness).

⚠️ The IETF draft's endianness table appears inverted relative to
universal practice. Implement from real captures and from dpkt/pcap.py,
not from that table.

pcapng is easier than it looks — uniform block framing
(type | total_length | body | total_length) means unknown blocks can
be skipped wholesale. Budget the effort for two real traps:

  1. if_tsresol (IDB option 9): MSB=0 → 10⁻ⁿ (6 = µs, 9 = ns);
    MSB=1 → 2⁻ⁿ. Absent means microseconds. Resolution is
    per-interface, so it is parser state, not a constant.
  2. EPB timestamps are a single 64-bit unit count split across two
    32-bit words
    not a seconds/microseconds pair as in classic
    pcap. Getting this wrong is the classic pcapng bug.

Also: a file may contain multiple sections with different
endianness, decided per-SHB by the 0x1A2B3C4D byte-order magic.

Relationship to memoryview

This creates the large-contiguous-buffer case where zero-copy walking
finally pays. Measured: slicing frames out of one 400 KB buffer runs at
26,786 f/s with bytes slicing versus 47,643 f/s with memoryview
— 1.8×
. (For a single small frame memoryview is marginally slower,
so this is the scenario that justifies it, not a general rule.)

Acceptance criteria

  • Classic pcap and the SHB/IDB/EPB/SPB subset of pcapng parse.
  • The API takes a buffer; no filename or file opening in src/.
  • Nanosecond and non-default if_tsresol captures give correct
    timestamps, with fixtures proving it.
  • tests/conftest.py drops its private copy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions