v1.0.0-rc.2 — flow collection, enterprise firewalls, shared time range

The headline is flow collection: NetInv can now answer what is on a link, not only how much. Plus two firewall platforms, a shared time range across every graph, a customisable dashboard, and configurable retention.

87 commits since v1.0.0-rc.1. Still a release candidate — the v1.0 gates in CLAUDE.md are unchanged and listed at the bottom.

Flow collection (ADR-020, ADR-022 · doc 34)

A seventh service, netinv-flow, receives NetFlow v5, v9 and IPFIX on UDP 2055 and 4739 and reads the version from the datagram, so any format works on either port. sFlow is not decoded.

It aggregates at ingest rather than storing flows. A flow record is a wide event keyed by source, destination, port and protocol; stored as time series its cardinality is the number of host pairs on the network. So each interval is reduced to top-N talkers, conversations and applications per interface, and only those are written — netinv_flow_bytes and netinv_flow_packets. Raw per-flow retention stays deferred and needs its own datastore.

The cost is stated rather than hidden: a conversation outside the top N for an interval was never written. It did not go unqueried; it does not exist.

Where it surfaces:

• Flow tab on device detail — talkers / conversations / applications, an interface filter, and a chart of the heaviest few.
• Top flow dashboard panel, fleet-wide or per exporter, repeatable.
• In-product setup guide with a format selector and per-vendor snippets (Cisco IOS, Junos, Huawei VRP, FortiGate, PAN-OS, RouterOS, VyOS, softflowd, UniFi), the collector's address filled in, and the traps that produce silence stated before the commands.

v9 and IPFIX make the collector stateful: a data record is meaningless without the template describing it. A restart therefore loses every template and flow is missing until exporters resend — counted and reported as awaiting_template, deliberately not as "undecodable", so nobody hunts a fault that is not there. The template cache is capped and expiring, because it is attacker-influenced state on an unauthenticated UDP port.

Validated on real hardware

A pilot UniFi gateway exports v5 into the collector and it decodes cleanly — not one packet refused or undecodable. Sampling was verified against the wire, not assumed: at Hash 1-in-512 the header read mode=2 interval=512 and the decoder scaled by 512; with sampling off it reads mode=0 interval=0.

Two findings worth repeating, because both correct things this project previously believed:

• UniFi gateways do export flow. Earlier documentation said they could not. That claim was never tested — it was inferred from a missing sFlow MIB on the switches and generalised to the gateways. The feature is in UniFi Network under Traffic Logging → NetFlow (IPFIX), offering v5/v9/IPFIX. It is now documented with the click-path and its three traps: export is selected per network rather than per interface, sampling ships on, and the "Timeout Rate" is the active timeout.
• Flow totals are a fraction of the SNMP counters, by design. Measured at 3.3% and 10.5% on two interfaces with sampling off. The dominant cause was confirmed rather than reasoned: every interface sat at the ten-bucket top-N cap in every one-minute interval. This is why ADR-020 makes SNMP the authority for volume and flow the answer to composition — the rule now has a measurement behind it.

Flow is attributed by the datagram's source address, and routers routinely export from an uplink or loopback. Devices therefore carry a flow_exporters list, and the Flow tab's empty state names the unattributed addresses it can see and offers to claim one.

Enterprise firewalls (ADR-021 · doc 10)

fortinet-fortios and paloalto-panos connectors — inventory, interfaces and health like any connector, plus device-level session gauges (netinv_firewall_session_count, _session_max, _session_setup_rate). Anything policy-, rule-, NAT- or VPN-scoped remains deferred: the line is drawn where the data stops being a number about the appliance and becomes a table about its configuration.

FortiOS publishes no session ceiling, so no utilization is invented from one. PAN-OS does, so utilization is a query over count and max rather than a stored number that could drift. PAN-OS is also the only platform here whose CPU and memory come from HOST-RESOURCES-MIB.

Neither is validated against hardware, and neither has a fixture — every OID comes from the published vendor MIBs. Six of eight connectors are now unvalidated, which doc 10 states per connector.

Graphs, dashboard and retention

• One shared time range across every graph, using Cacti's nineteen timespans, persisted per browser. Presets beyond retention are shown disabled rather than hidden.
• Customisable dashboard: add, remove, reorder and widen panels, including an embedded weathermap and a chart of any published metric. Saved per user.
• Metrics retention is configurable and defaults to two years. The API's range ceiling reads the same setting, so a query limit cannot drift from what the store actually keeps.
• Platform → Capacity answers "will the disk hold this retention", measured from the running system rather than derived from configuration.
• Fleet-wide poll interval settable from the UI (1/5/10/15 min), with rate() windows sized from it.
• Weathermap: links can be deleted without deleting their nodes; a link shows one flow in the heavier direction with the reverse rate still readable; labels name their destination.

Deployment

• Proxmox LXC: deploy/proxmox/netinv-lxc.sh create|verify|destroy, and doc 33. Verified end to end on Proxmox VE 9.2.10, including a fix for unprivileged-container ICMP that Docker's ping_group_range clamping breaks.
• The Helm chart and image build now cover the flow service. It declares listen ports rendered as a Service — the first component that must be reachable from the device network — and is pinned to a single replica, because UDP has no replay and template state is per exporter.

Notable fixes

• A device can be re-addressed instead of deleted and recreated; PATCH /devices/{id} silently ignored mgmt_ip and snmp_port at both layers.
• The Flow tab hid real data between a bursty exporter's sends — a presence check on the instant lookback while the table summed over the selected range.
• Connectors reported the health capability twice.
• A recognised service port now beats the lower one when naming an application, so WireGuard on 51820 is not filed under a client's ephemeral port.

Still open before v1.0

Unchanged from rc.1 and not fudged: real-hardware validation of the six untested vendor connectors, a 72-hour soak plus chaos-lite on staging Kubernetes, the doc 20 §12 security checklist against TLS, and a backup/restore drill on pilot data. These need real infrastructure and time.

sFlow remains the one flow format NetInv does not read; it samples packet headers rather than exporting flow records and shares none of the template machinery.

Hardware validation reports remain the single most useful contribution — see CONTRIBUTING.md and the ask in Discussions.