Skip to content

netprotocols 2.2.0

Latest

Choose a tag to compare

@EONRaider EONRaider released this 04 Sep 20:54
· 15 commits to master since this release
401a1c6

Added

  • Typed accessors for every enum-backed field. _enums.py defines
    EtherType, IPProtocol, ARPOperation and ARPHardwareType, but
    until now no decoded field exposed them — only the *_name display
    strings, and the enums themselves were used only for dispatch. Every
    field with a fixed wire vocabulary now has an *_enum companion
    alongside the existing int field and its *_name string, mirroring
    the src/src_address precedent 1.3.0 established for IP addresses:
    Ethernet.ethertype_enum, VLAN.ethertype_enum, GRE.protocol_enum,
    IPv4.protocol_enum, IPv6.next_header_enum (and the three IPv6
    extension headers that share the field: IPv6HopByHopOptions,
    IPv6DestinationOptions, IPv6Routing, IPv6Fragment),
    ARP.oper_enum, ARP.ptype_enum, ARP.htype_name/ARP.htype_enum
    (new — ARPHardwareType was exported but referenced nowhere), and
    DHCP.htype_name/DHCP.htype_enum. Each returns None — never
    raises — for a wire value this library does not enumerate, so
    bytes(decode(x)) == x is unaffected for unrecognized values; the
    raw int field stays canonical and round-trips regardless (#95).

  • IPv4, TCP and DHCP curate a short __match_args__ by hand. Their
    full auto-generated positional form runs to 11-15 fields — unusable
    positionally, since nobody writes, or gets right, a fourteen-slot
    pattern. IPv4.__match_args__ == ('src', 'dst', 'protocol'),
    TCP.__match_args__ == ('src_port', 'dst_port', 'flags'),
    DHCP.__match_args__ == ('op', 'chaddr', 'yiaddr') — the two or
    three fields someone drafting case IPv4(...) actually reaches for.
    Every other header (including the still-wide IPv6, at 8 fields)
    keeps the plain auto-generated tuple, and keyword patterns
    (case IPv4(protocol=6)) are unaffected either way and stay the
    documented default. __match_args__ is part of the public API once
    documented, so the chosen order is pinned by
    tests/test_pattern_matching.py rather than left free to drift.
    ARCHITECTURE.md and docs/CLAIMS.md are updated to describe this as
    a deliberate, tested exception rather than the "this library never
    hand-declares __match_args__" absolute they stated before (#94).

  • DNS resource records expose typed RDATA, and every question, not
    just the first.
    DNSResourceRecord.rdata_text decoded A/AAAA/MX/
    SOA/etc. into a display string only; a new rdata_value field adds
    the typed decoding alongside it (rdata_text is unchanged) —
    ipaddress.IPv4Address/IPv6Address for A/AAAA, the decompressed
    target str for NS/CNAME/PTR, a new MXRecord(preference, exchange)
    for MX, a new SOARecord(mname, rname, serial, refresh, retry, expire, minimum) for SOA, list[str] of character-strings for TXT,
    and None — never raises — for the types this library does not
    decode. Computed eagerly at parse time, like rdata_text always has
    been: a name inside RDATA (a CNAME's target, for instance) can use
    DNS compression against the whole message, so decoding it needs
    the same message-wide context rdata_text already required, which a
    property computed lazily from rdata alone could not resolve.
    Separately, DNS.questions adds a tuple[DNSQuestion, ...] walking
    every entry of the question section — question_name/question_type
    /question_class exposed only the first and are unchanged (#96).

  • A typed DHCPOption, alongside DHCP.option_map. option_map
    was the only accessor in the library returning a bare
    dict[int, bytes] rather than typed objects; it is unchanged (still
    returns exactly what it always did). A new parsed_options property
    wraps the same, already RFC-3396-concatenated, mapping into a
    tuple[DHCPOption, ...], mirroring TCPOption/IPv4Option: code,
    data, code_name, and a decoded .value for the option codes this
    library understands — a single ipaddress.IPv4Address for Subnet
    Mask (1)/Requested IP Address (50)/Server Identifier (54), a tuple of
    one or more for Router (3)/Domain Name Server (6) (RFC 2132 permits
    repeating either), the integer seconds for IP Address Lease Time
    (51), the raw byte for Message Type (53). None — never raises —
    for codes this library does not decode and malformed option data
    (#96).

  • IPv4Option.value decodes Record Route, Timestamp and Router
    Alert.
    IPv4Option had kind_name and no .value — the three
    common kinds (RFC 791 §3.1, RFC 2113) were named but their contents
    left raw, unlike TCPOption, which already decodes its values.
    Record Route (7) decodes to a tuple[ipaddress.IPv4Address, ...] of
    the addresses recorded so far — the option's own pointer byte says
    how many of the address slots are actually filled, not the option's
    total length. Timestamp (68) decodes to tuple[int, ...] of plain
    millisecond timestamps when its flag selects that shape, or
    tuple[tuple[ipaddress.IPv4Address, int], ...] of (address,
    timestamp) pairs when the flag says each entry carries one — the
    overflow counter and pointer are not decoded, read data raw for
    those. Router Alert (148) decodes to the 2-byte value as int.
    None — never raises — for every other kind and for malformed data
    on one of these three (a bad pointer, an unrecognized Timestamp
    flag, a short buffer) (#96).

  • IPv6Routing.segments and IPv6Option.value for Router Alert /
    Jumbo Payload.
    IPv6Routing.data was entirely unparsed — no
    segment-list extraction for any routing type. RH0 (routing_type
    0, deprecated by RFC 5095 but still seen — RFC 2460 §4.4) and
    Mobile IPv6 (2, RFC 6275 §6.4) both store a 4-byte reserved field
    followed by one IPv6 address per segment; segments decodes that
    into a tuple[ipaddress.IPv6Address, ...]. RPL Source Routing (3,
    RFC 6554) is deliberately not decoded: RFC 6554 §3 elides a
    shared prefix from each intermediate address relative to the
    enclosing packet's destination address, context this
    one-extension-header accessor does not have — data stays available
    raw. None for every other routing type and for malformed address
    data. Separately, IPv6Option.value (Hop-by-Hop / Destination
    Options TLVs) now decodes Router Alert (5, RFC 2711 §2.1) and Jumbo
    Payload (194, RFC 2675 §2) into typed ints, the last of #96's four
    pieces (IPv6Option "likewise decode[d] no values"); None for
    every other type and malformed data, same contract throughout this
    tier (#96).

  • CI proves the library runs under a real Pyodide (WebAssembly)
    runtime
    , not just that the modules scapy/dpkt need are
    individually blocked. A new pyodide job (.github/workflows/ci.yml)
    boots actual Pyodide under Node via scripts/pyodide/run_in_pyodide.mjs,
    installs the wheel this job just built, and decodes the entire
    real-capture fixture corpus with it — see
    scripts/pyodide/check_in_pyodide.py. That surfaced a genuine bug
    along the way: IPv6.decode()/bytes(IPv6(...)) used
    socket.inet_ntop/inet_pton(AF_INET6, ...), and Pyodide's CPython
    build has AF_INET6 sockets disabled, so every IPv6 frame failed to
    decode there. _base.py's ipv6_to_bytes/bytes_to_ipv6 are now a
    pure-Python implementation (ipaddress for parsing; a hand-rolled
    RFC 5952 canonicalizer, differentially verified against glibc's
    inet_ntop across 500,000+ random addresses plus both of its
    dotted-quad special cases, for formatting — str(ipaddress.IPv6Address)
    was tried first and rejected: it disagrees with glibc on IPv4-mapped
    addresses, and disagrees with itself between Python 3.11 and 3.12).
    IPv4 addressing is unaffected; behavior for every existing platform
    is unchanged, exception type included (#99).

  • Nightly fuzzing with a moving seed. tests/test_fuzz.py's
    "netprotocols" Hypothesis profile is deterministic on purpose (200
    examples, derandomize=True) so a pull request is reproducible — but
    that also means every build since it was written has run the same
    200 inputs. A new "nightly" profile (10,000 examples, a real random
    seed each run) runs the whole suite on a new schedule,
    .github/workflows/fuzz.yml (03:00 UTC daily, plus manual dispatch),
    kept out of ci.yml deliberately — ci.yml is reused by
    release.yml via workflow_call, and a schedule: trigger there
    would fire the whole PR/release gate on a cron, not just this
    exploratory job. .hypothesis/'s example database accumulates
    across nightly runs via actions/cache (keyed by run id with a
    prefix restore-key, since a fixed key would restore forever but
    never actually save a new entry) and uploads as a workflow artifact
    on failure, so a counterexample is recoverable without repo write
    access; a failed scheduled run's own red build is the notification
    (GitHub already does this by default), so nothing here files an
    issue on top of it. Reproduce locally:
    HYPOTHESIS_PROFILE=nightly uv run pytest. Also adds a targeted
    strategy building well-formed TCP SYN options (MSS, window scale,
    SACK-Permitted, SACK, timestamps) — the real-capture corpus never
    caught a SYN, so unlike NOP/Timestamps these otherwise depend on
    max_examples alone stumbling into a well-formed TLV by chance
    (#98).

  • A pcap/pcapng reader that takes bytes, not filenames. New
    netprotocols.pcap: read_captures(buffer) auto-detects classic
    pcap vs. pcapng from its magic number and yields CapturedFrame
    (timestamp in nanoseconds since the Unix epoch, normalized from
    whatever resolution the source recorded, data the frame's raw
    bytes); read_pcap()/read_pcapng() are the same for a caller who
    already knows the format. pcapng support covers exactly the block
    types frames can come from — Section Header, Interface Description
    (read only for its if_tsresol option), Enhanced Packet, and Simple
    Packet (which the format gives no timestamp at all, hence 0);
    every other block type is skipped wholesale. A malformed or
    truncated capture raises the new MalformedCaptureError
    (ProtocolError family, no lax mode — a corrupt container is a
    different failure shape than a malformed header inside one already-
    extracted frame). Format detection is eager; producing frames is
    lazy (a generator), so a bad record downstream doesn't invalidate
    what already iterated cleanly, and a huge capture is never forced
    into a list of frames nobody asked for.

    tests/conftest.py drops the private classic-pcap reader every test
    file reached for — pcap_frames() is now a thin adapter over the
    shipped read_pcap(), and ~10 test files were migrated onto it (a
    real migration, not a rename: tests/test_pcap.py keeps its own
    independent reference reader, deliberately never importing the
    module it is cross-checking, the same "standalone, so a shared bug
    can't cancel itself out" precedent as scripts/benchmark.py and
    scripts/check_fixtures.py).

    One design idea was tried and reverted on measurement: slicing each
    frame lazily out of a memoryview over the whole buffer, to keep
    large captures zero-copy. Measured across synthetic captures up to
    ~140MB, it was 0.91x-0.98x — never faster, sometimes slower — because
    a real capture is many small frames, and a memoryview slice's own
    overhead is paid per frame; #88's identical finding for a single
    frame generalizes rather than being contradicted. docs/CLAIMS.md
    5.8 is corrected accordingly — it previously forward-referenced this
    issue with an unverified "1.8x" figure (#100).

Documentation

  • The comparative-claims embargo is lifted (#107, #124). docs/CLAIMS.md
    1.7 gained a per-project table auditing ten comparable Python packet
    libraries' CI configurations (dpkt, scapy, pypacker, construct,
    pcapkit, dnspython, pyshark, nfstream, stackforge, PyTCP-net_proto)
    for an actual performance-regression gate — cited file:line or
    workflow, dated 2026-09-04; none of the ten gate on one, so the
    comparative form of 1.7's claim is now stated with that evidence
    (#124). Every section that previously carried a
    COMPARATIVE — HELD banner is re-measured against the current tree
    and published: decode throughput (1.1, 1.2), encoding throughput
    (1.3), import time (1.4), wheel size (1.5), decode depth (1.6), the
    mypy --strict comparison (2.1), match/case exclusivity (2.2),
    the five-property combination (2.3), Pyodide/browser support (3.1),
    and the permissive-licence wedge (4.1). One finding moved against
    this project rather than for it and is reported as such: three tiers
    of added surface since Tier 1 (#87's dispatch rewrite, #88's chain
    walker, #91's structured errors chief among them) reversed decode
    throughput from 1.16× faster than dpkt back to roughly 11% slower —
    still within the 15% band claim 1.2 names, just no longer ahead.
    Two new reproduction scripts back claims that previously had none:
    scripts/benchmark_encode.py (1.3) and scripts/benchmark_import.py
    (1.4). README.md gains a "Why NETProtocols" section drawing the
    now-published claims into the project's front door, in the tone
    docs/CLAIMS.md 4.2 sets for writing about competitors.