netprotocols 2.0.0

Development

• The round-trip property (bytes(decode(x)) == x) is now
  Hypothesis-generated for all 18 protocols, not 4. It previously
  held only for Ethernet, UDP, TCP and IPv6Fragment; the other
  14 rested on a single example each. tests/strategies.py adds one
  reusable strategy per protocol — reusable because a strategy that
  generates valid instances is useful for more than this one
  property — and tests/test_fuzz.py::TestGeneralizedRoundTrips
  asserts the property in one method, parametrized over all 14.

  The interdependent-field protocols the issue specifically flagged
  are generated as consistent combinations, never independently: IPv4
  draws ihl first and sizes options to match; the three IPv6
  extension headers draw hdr_ext_len first and size their TLV bytes
  to match; GRE draws flags first and computes fields' length from
  the same _optional_len() the decoder itself uses, so the two can
  never silently disagree.

  The 14 single-example assertions already living in each protocol's
  own test file (test_arp.py::test_round_trip and so on) are kept as
  regression anchors — a fixed example pinned to a real captured header
  catches a specific regression fast and readably, which a generated
  property does not replace.

  One adjacent gap closed while surveying this: DHCP and GRE were
  missing from test_fuzz.py::ALL_PROTOCOLS entirely, so neither had
  ever been fuzzed with untrusted bytes for basic decode safety (any input either decodes or raises ProtocolError, never anything else).
  Both are now included.

Added

• match/case dissection is documented. It already worked on
  every decoded header, with zero code changes needed — a frozen
  dataclass auto-generates __match_args__, and every field with a
  fixed wire vocabulary is an IntEnum, so a value pattern binds the
  plain int the decoder stored. None of that appeared anywhere:

  match packet.layers:
      case [_, IPv4(protocol=IPProtocol.TCP) as ip, TCP(flags_str=f), *_] \
              if "SYN" in f and "ACK" not in f:
          print(f"connection attempt from {ip.src}")

  README gains a first-screen example and a dedicated section;
  ARCHITECTURE.md explains the two mechanisms and the two ways a
  contributor can quietly break them (reordering a class's fields
  changes what a positional pattern binds to; degrading a fixed-
  vocabulary field from IntEnum to a bare int makes a value
  pattern stop matching without raising). tests/test_pattern_matching.py
  pins both mechanisms directly and runs the README's own example over
  the corpus, so a regression in either is caught rather than
  incidental.

• Structured diagnostics on every ProtocolError. Every exception
  used to carry only a formatted string — the full attribute set on a
  raised TruncatedHeaderError was args, add_note,
  with_traceback. A fuzzing harness, conformance suite, or
  protocol-validation tool that wanted to know where a parse failed
  had to regex the message. Every raise site in src/ now attaches:

  try:
      packet = decode_frame(frame)
  except ProtocolError as e:
      print(e.protocol, e.field, e.offset, e.frame_offset, e.expected, e.actual)
      # <class 'netprotocols.layer3.ip.IPv4'> ihl 0 14 >=5 0

  • protocol — the class that raised. Set on every single raise site.
  • field — the attribute at fault, where one field is at fault.
  • offset — byte offset of the problem, relative to the decode()
    buffer for a fixed-header error, or to the field attribute
    (options, body, sections) for an on-demand parse — a TCP
    option error's offset is relative to header.options, not the
    frame. None for a __post_init__ validation error, which sees
    field values, never the bytes they came from.
  • frame_offset — offset rebased to the whole captured frame.
    decode_frame is the only code holding the cursor needed to do
    this, so it's the only thing that sets it; a bare
    SomeClass.decode() call leaves it None.
  • expected / actual — added wherever there's a concrete pair to
    state beyond the message text.

  All five default to None, and no message string changed — every
  existing str(err) and match= assertion holds. New public name:
  MaxDepthExceededError gains the same attributes as every other
  ProtocolError subclass (it already existed as of the decode_frame
  work; this is the first release to give it structured context too).

• decode_frame(): the chain walker ships with the library. The
  README taught a hand-rolled eight-line loop, ARCHITECTURE.md showed
  it again, and the test suite kept its own copy — so the library's
  most-used function was the one function it did not provide. It does
  now, with the parts a copy-pasted loop never has:

  from netprotocols import decode_frame
  
  packet = decode_frame(frame)     # Packet(Ethernet(...), IPv4(...), TCP(...))
  frame[packet.consumed:]          # whatever the chain did not decode

  • An explicit starting layer. decode_frame(buf, start=IPv4)
    walks a buffer that begins mid-stack — a tunnel payload, a packet
    quoted inside an ICMP error, a non-Ethernet link type. There was
    previously no way to ask for this.
  • Bounded depth. max_depth (default 32) caps the chain and
    raises the new MaxDepthExceededError, rooted at ProtocolError
    like everything else. Every header already validated its own
    length, so a chain always terminated; the bound turns a crafted
    frame's very long walk into an immediate named error. The deepest
    chain in the 97-frame corpus is 5.
  • A lax mode that reports instead of raising. lax=True ends the
    walk on a ProtocolError and returns the layers decoded so far,
    with the reason on packet.stopped_by — what a capture tool needs
    when frame 4,000,001 is malformed. It relaxes the walk, never a
    decoder: every layer returned was decoded under the ordinary strict
    rules.
  • Per-call decoder overrides. decode_as={"udp.port": {6969: DNS}}
    reads DNS on a nonstandard port for one call, without touching
    global state; registry= takes a prepared registry for bulk work.

  Packet gains stopped_by (why a walk ended early, None for a
  packet you built) and consumed (the bytes its headers occupy).
  Both default to the constructed-packet values, so Packet(eth, ip)
  is unchanged.

  On memoryview: the walker slices what it is given and does not
  convert. Measured on the corpus, wrapping each frame in a
  memoryview runs at 0.95x the plain-bytes walk — for one small
  frame the view costs more to build than the copy it saves — while a
  memoryview over a large capture buffer keeps slices zero-copy.
  Converting internally would have been worse than either.

• A public protocol registry: third parties can now extend the decode
  walk without editing library source. Dispatch was four hardcoded
  functions with dict literals inside them and no hook of any kind — a
  protocol this library does not implement (MPLS, VXLAN, a proprietary
  telemetry header) could only be added by forking. It is now five named
  tables owned by netprotocols.registry, and registering against one
  is all it takes:

  from netprotocols.registry import register
  
  @register("ethertype", 0x8847)
  class MPLS(Protocol):
      ...
  
  register("ip.proto", 132, SCTP)   # for a class you did not write

  The tables are ethertype, ip.proto, ip.proto.v6, udp.port and
  tcp.port, each named after the wire field it dispatches on.
  ip.proto.v6 inherits ip.proto, which is how the IPv6-only
  gating generalises: the four extension headers are registered in the
  v6 table alone, so an IPv4 packet with protocol=0 still cannot
  conjure a Hop-by-Hop layer — and because inheritance is resolved at
  registration rather than at lookup, the gating costs nothing on the
  hot path. Dispatch behaviour is unchanged entry for entry.

  Registrations land in the process-wide DEFAULT registry;
  Registry.from_defaults() gives an isolated one for embedding or test
  isolation, and Registry.derive() makes a copy-on-write child.
  Registering over an existing key raises RegistryConflictError unless
  override=True is passed, so two packages claiming the same port
  cannot silently resolve by import order; re-registering the same
  class to the same key is a no-op, since a decorator re-runs whenever
  its module does.

  New public names: Registry, register, register_all, DEFAULT,
  RegistryConflictError, UnknownTableError.

Changed

• next_protocol() accepts an optional registry. Threading a
  per-call registry through the walk needs the dispatch to be
  redirectable, and next_protocol() read the process-wide tables
  directly. It now takes registry=None, passed to the same dispatch
  helpers as before, so no table knowledge is duplicated and the
  default path stays a single dict.get. Measured in one process, the
  optional parameter costs +2.1 ns per dispatch — roughly 0.1% of a
  frame decode. Every existing zero-argument call is unaffected.
• Port-based dispatch is a table lookup rather than a rebuild.
  udp_app_class and tcp_app_class re-ran their deferred imports and
  rebuilt a dict literal on every call — they were outside the scope
  of the earlier dispatch-hoisting work, which named only the EtherType
  and IP-protocol functions. Both now read the registry's flat tables:
  udp.port 303.1 → 46.0 ns (6.6x), tcp.port 186.8 → 45.1 ns (4.1x),
  measured old and new shapes in one process. ethertype loses its
  lazy-build guard (51.8 → 47.7 ns) and ip.proto is unchanged, having
  already been a table. Corpus throughput moves about 1% — only 28 of
  the 97 corpus frames reach a transport header and therefore do a port
  dispatch at all — which is below this machine's run-to-run noise; the
  per-call figures above are the measurement that resolves.
• The layer modules no longer import each other's classes. The
  built-in decoder map moved into _defaults.py, which registers it
  once from __init__.py after every protocol class exists, so the
  function-body imports that kept the module graph acyclic are gone
  from vlan.py, gre.py, tcp.py, udp.py and ipv6_ext.py; those
  dispatch helpers are now ordinary module-level imports. The tables
  are built at package import instead of on first dispatch, so the
  lazy-build branch is off the hot path entirely.
• DNS section parsing happens once, and no longer re-serializes the
  message to read it. Six helpers called bytes(self) — a full
  re-serialization of the whole message — so a single .answers access
  rebuilt the message 14 times; and answers, authorities and
  additionals each re-parsed all three sections, so reading all
  three parsed the message three times. The parsers are now module-level
  functions working directly from the sections bytes the instance
  already holds, in section-relative offsets (a compression pointer,
  which counts from the start of the message, is translated once), and
  the three accessors share a single parse cached on those immutable
  bytes. Measured here on a corpus response: .answers 16.84 → 0.20 us,
  all three sections 52.08 → 0.54 us, and one uncached parse 17.03 →
  12.17 us from dropping the re-serializations alone. Nothing is stored
  on the frozen instance — the cache is a bounded module-level memo
  keyed by value, so equal messages share a parse and records (frozen
  dataclasses) are shared rather than copied. The byte-exact round-trip
  is unchanged. A compression pointer that addresses the fixed header
  now raises InvalidFieldError instead of decoding header bytes as
  labels. No API change.
• The decoder no longer re-validates the addresses it just
  generated. Every header's __post_init__ ran the address
  validators, including for instances built by decode() — so a MAC
  mechanically rendered from six bytes by bytes_to_mac() was
  immediately matched against mac_regex to confirm what the
  conversion had already guaranteed. Ethernet, ARP and IPv4 now
  build their decoded instance directly (object.__new__ plus
  object.__setattr__ per field), skipping __init__ and
  __post_init__ on that path only. Measured here: Ethernet.decode
  2330 → 830 ns (2.8x), IPv4.decode 4936 → 2596 ns (1.9x), a corpus
  walk 76,500 → 113,200 frames/sec (1.48x). Strictness on
  construction is unchanged — every public constructor still
  validates and still raises, which is now asserted alongside a test
  that the decode path runs no regex at all. The __post_init__ checks
  bypassed this way are ones decode() establishes itself (documented
  at each site and in _base.py). No API change.
• Protocol dispatch is a table lookup, not a table construction.
  _ethertype_class() and _ip_protocol_class() re-ran their deferred
  imports and rebuilt a dict literal on every call — once per layer
  per frame, the hottest path in the library. Both now populate a
  module-level table on first use (the imports stay deferred, so the
  layer modules remain acyclic) and reduce the call to a lookup.
  Measured here: _ip_protocol_class 1439 → 104 ns (13.9x),
  _ethertype_class 712 → 86 ns (8.3x), and a corpus walk 58,300 →
  76,500 frames/sec (1.31x). The IPv6-only gating is unchanged: the
  IPv4 table simply omits those numbers, so an IPv4 packet with
  protocol=0 still cannot decode a Hop-by-Hop layer, now asserted at
  the table level as well as through the public API. No API change.
• bytes_to_mac renders addresses with bytes.hex(":") instead of a
  generator over format(). The old form ran seven generator steps and
  six format() calls per address, twice per Ethernet frame, and showed
  up in a corpus profile as 88,200 generator calls. Measured 16–29×
  faster on the call depending on run. Output is byte-for-byte
  identical, and memoryview.hex takes a separator too, so a
  decode-time view still works. No API change.

Removed

• BREAKING: Packet.payload is gone. It was always exactly
  bytes(self) — a redundant duplicate of __bytes__/bytes(packet),
  which already existed and is the unambiguous spelling — so it is
  deleted outright rather than deprecated. This is the breaking change
  behind the 2.0.0 major bump. Replace packet.payload with
  bytes(packet).

Added

• Packet indexes by protocol type, not just position, and is
  hashable. packet[TCP] returns the first TCP layer in wire order
  (KeyError if there is none); packet.get(TCP) returns None
  instead of raising. packet[0] keeps indexing positionally —
  __getitem__ branches on whether the key is an int or a type.
  Packet.__hash__ mirrors exactly what __eq__ already compares
  (layers, and stopped_by's type and str(), not its identity —
  ProtocolError has no custom __eq__), so a Packet is now usable
  as a dict key or set member.

• Canonical, direction-independent flow keys. netprotocols.flow
  is a new small module (FlowKey, flow_key()) that folds a TCP/UDP
  segment and its enclosing IPv4/IPv6 header into a key that is
  identical for both directions of one conversation:

  from netprotocols import flow_key
  
  flow_key(tcp, ip=ipv4) == flow_key(reply_tcp, ip=reply_ipv4)  # True

  FlowKey is a NamedTuple — canonicalizing the two directions is
  comparing the two (address, port) endpoint tuples and always
  emitting the lexicographically smaller one first, which a plain
  tuple already orders natively; every other frozen dataclass in this
  codebase models a wire format (decode()/__bytes__/_struct), and
  a derived key isn't one. Packet.flow_key() is the convenience form:
  it walks self.layers for the first IPv4/IPv6 and TCP/UDP layers and
  delegates to the free function, None if either is missing. Reads
  whichever of IPv4.protocol / IPv6.next_header the enclosing
  header actually has — the same semantic field, different attribute
  name. A transport layer with no ports (ICMP) returns None from
  both forms, rather than raising or inventing a port-slot convention
  for a message type that has none.

• ICMPv4/ICMPv6 gain embedded_chain: an error message's
  embedded packet, already decoded. decode_frame(lax=True, start=...) already handled this — an RFC-792 error message quotes
  only the invoking IP header plus 8 bytes of what follows, never a
  full TCP/UDP header, so decoding it needs the lax path and no
  try/except, verified with zero new code before this landed.
  embedded_chain is the pre-wired convenience next to the existing
  raw embedded_packet, same shape as an accessor-plus-typed-view pair
  elsewhere in this codebase:

  icmp.embedded_chain
  # decode_frame(icmp.embedded_packet, lax=True, start=IPv4)
  # → Packet([IPv4(...)]), stopped_by=TruncatedHeaderError(...)

  None for the same cases embedded_packet degrades to (non-error
  message types, an empty body); starts at IPv4 for an ICMPv4
  message, IPv6 for ICMPv6. The README documents the distinction
  the issue asked for: embedded_chain's lax=True is the right
  default here because the truncation is RFC-mandated, expected input
  — not a license to reach for lax=True on a complete frame that
  fails to decode, which is still a bug to raise on.

Fixed

• The release workflow can no longer publish untested code. Pushing
  a v* tag ran uv build and uv publish with no dependency on the
  lint, typecheck or test jobs — and because the CI workflow triggers
  only on pushes and pull requests targeting master, a tag push ran
  no checks at all. A tag against broken code published to PyPI
  unchallenged, and a published version can only be yanked, never
  replaced. ci.yml is now also a reusable workflow (workflow_call),
  and release.yml invokes it against the tagged commit with the
  publish job gated behind it, so the release gate and the pull-request
  gate are one definition and cannot drift apart.

Fixed

• A mistagged release can no longer publish the wrong version.
  release.yml built from pyproject.toml and never consulted the tag,
  so pushing v1.4.0 while the project still read 1.3.0 would either
  fail at upload against an existing version or, if that version was
  never released, silently publish the wrong one. The QA ladder cannot
  catch this — every check passes, because the code is fine and only the
  tag is wrong. The release now reads the version back out of the built
  artifact and refuses to publish unless it matches the tag.
• The README no longer advertises shipped work as upcoming. Its
  Roadmap section listed the eight decoder-depth items tracked by #67 —
  all of which shipped in 1.3.0, with #67 itself closed — so the front
  page presented finished work as planned and pointed readers at a
  closed issue. It now links the current roadmap (#107) and its five
  release-mapped epics instead of restating individual items, which is
  what went stale.
• ARCHITECTURE.md no longer misstates the corpus or the source
  layout. It described "93 frames across 16 scenarios" long after the
  corpus reached 97 across 17, and its layout map omitted vlan.py,
  gre.py and dhcp.py. The counts now live in
  tests/fixtures/MANIFEST.md and the README alone — ARCHITECTURE.md
  points at the MANIFEST rather than keeping a third copy, which is
  what drifted.
• Layer sub-packages now export everything their layer defines.
  from netprotocols.layer7 import DHCP failed while
  from netprotocols import DHCP worked, because each layer's own
  __init__ carried a subset of what the top-level package re-exports.
  layer3 was missing GRE, IPv4Option, IPv6Option, NDPOption
  and IGMPv3GroupRecord; layer4 was missing TCPOption; layer7
  exported only DNS, omitting DHCP, DNSOverTCP and
  DNSResourceRecord. Purely additive — no name changed meaning.

Development

• Comparative claims are under embargo until the roadmap closes.
  docs/CLAIMS.md gains a standing rule and a per-claim
  COMPARATIVE — HELD marker: nothing mentioning scapy, dpkt or any
  other library reaches the README, release notes or package metadata
  until #107 is finished. The measurements keep being recorded — the
  register exists to hold evidence, not to publish it — and ship once,
  together, rather than in pieces that each need defending. The
  rationale is in the numbers: claim 1.2 went from "2.9x slower than
  dpkt" to "1.16x faster" inside a single tier.
• Three differentiators found during Tier 1 are now tracked as claims
  rather than living in pull-request descriptions: decode depth (1.6 —
  the corpus walk reaches a deeper layer than dpkt on 27 of 97 frames,
  reproducible with scripts/benchmark.py --depth, which this change
  adds), the CI throughput gate (1.7), and strict construction on a
  decode path that pays nothing for it (5.4). The stale "we are ~86% of
  dpkt" line in "Claims we must not make" is corrected, and a rule
  added that a superiority claim must name its axis — scapy crafts,
  sends and covers thousands of protocols, so the codec axes are where
  the evidence is.
• A reproducible decode benchmark, and a CI job that fails on a
  regression. scripts/benchmark.py walks the 97-frame corpus and
  reports frames/sec; --compare times dpkt and scapy on the same
  frames (both dev-only extras in a new bench dependency group — the
  package still has zero runtime dependencies). Because absolute
  throughput is a property of the machine, every run also times a fixed
  calibration workload and reports a normalized figure, which is what
  --check compares against benchmarks/baseline.json. The CI job
  blocks at 15% below the baseline: measured, that is about four
  times the noise it must tolerate — the job's first run landed 3.7%
  from a baseline recorded on entirely different hardware, and
  run-to-run spread on one machine is ~2%, while the Tier 1 changes
  moved the number by 30-90% each. A failing check re-measures once
  before failing, so a runner that loses CPU to a neighbour does not
  block an unrelated pull request. Comparison output carries its own
  caveats — notably that the libraries are not asked for identical
  work, since dpkt leaves the DNS-over-TCP payload as raw bytes where
  this library decodes it — because a benchmark nobody can check is the
  problem #86 set out to fix, not the goal.
• tests/test_version.py holds netprotocols.__version__ against the
  version in pyproject.toml. The two were kept in step by hand and
  nothing checked them, so a bump that missed one would ship a package
  whose metadata and __version__ disagree.
• tests/test_docs.py holds documented facts against the fixtures:
  the MANIFEST's and README's frame/scenario counts must match the real
  corpus, ARCHITECTURE.md must not reintroduce a third copy of them,
  and the layout map must mention every shipped module. It also
  guards the README's Roadmap section against pointing back at the
  superseded #67 or re-listing work that has already shipped.
• tests/test_exports.py keeps the layer and top-level export sets in
  agreement, deriving the expectation from each object's __module__
  rather than a hand-written list, so a protocol added to the top level
  but forgotten in its layer fails the suite.
• Coverage is now enforced, not merely reported: fail_under = 98 in
  [tool.coverage.report]. The suite covers 99% of 1441 statements —
  the only misses are Packet.__repr__ and its __eq__
NotImplemented branch — so the gate is set below the current figure
  to leave room for a legitimately unreachable branch, and well above
  the level at which the bar would stop meaning anything.
• tests/test_workflows.py asserts the release gate stays wired:
  that ci.yml remains callable, that some release job invokes it,
  that publish depends on that job, and that every local uses: ./…
  reference resolves to a workflow declaring workflow_call. The
  wiring is otherwise easy to remove silently — releases keep working
  while the gate quietly stops existing — and it cannot be verified by
  running it, since publishing is irreversible.
• docs/CLAIMS.md records the positioning claims produced by the
  post-1.3.0 competitive analysis, each with its evidence, a
  reproduction path and a publication status — VERIFIED,
  GATED ON #nn, or NEEDS RE-MEASUREMENT. Several of the strongest
  claims are not true yet (decode throughput against dpkt is gated on
  the Tier 1 performance work; the browser claim is untested until a
  Pyodide CI job exists), so the register also lists what must not be
  said and why. Feeds the README as each roadmap tier lands; see #101
  and the roadmap in #107.