[sidecar/pipeline] P4 cleanup, getting us to 16 (non-mcast) / 19 (mcast) at current table sizes - #208
Closed
zeeshanlakhani wants to merge 3 commits into
Closed
[sidecar/pipeline] P4 cleanup, getting us to 16 (non-mcast) / 19 (mcast) at current table sizes#208zeeshanlakhani wants to merge 3 commits into
zeeshanlakhani wants to merge 3 commits into
Conversation
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
2 times, most recently
from
February 4, 2026 01:53
ed6d7d9 to
a75a02e
Compare
zeeshanlakhani
requested review from
FelixMcFelix,
Nieuwejaar and
rcgoodfellow
February 4, 2026 01:53
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
from
February 9, 2026 21:08
0d3cbfc to
4367608
Compare
zeeshanlakhani
marked this pull request as ready for review
February 10, 2026 23:39
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
2 times, most recently
from
February 20, 2026 11:47
580af39 to
499f3c7
Compare
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
2 times, most recently
from
February 24, 2026 06:24
52ac6a8 to
c951d18
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
4 times, most recently
from
March 2, 2026 04:25
ab643d8 to
0b8a5d7
Compare
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
2 times, most recently
from
March 5, 2026 22:12
dbe4e51 to
2116f9b
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
March 10, 2026 15:19
cee9f7d to
e8a6811
Compare
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
3 times, most recently
from
March 16, 2026 18:37
6bf36ea to
3254541
Compare
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
2 times, most recently
from
April 6, 2026 18:39
ce88768 to
079b582
Compare
*P4/sidecar pipeline changes*:
* Unify route_result_t struct for Router4/Router6 to prevent PHV liverange divergence
* Unify nexthop: replace separate nexthop_ipv4/nexthop_ipv6 with single nexthop (ipv6_addr_t) plus nexthop_is_v6 flag
* Add TTL compound key (idx, route_ttl_is_1) to route tables for TTL offload
- This includes Rust updates and a larger routing table for the compound key
* Add @pa_no_init pragmas for metadata fields to guard against compiler init bugs
* Add @pa_container_type("normal") pragmas for all ingress booleans and wider metadata fields to prevent mocha container corruption (whole-container writes clobbering neighboring fields)
* Extend @pa_container_type to egress bridge header fields and drop_reason
* Zero sc_pad at every sc_code write site (was uninitialized before)
* Bridge header now includes is_mcast_routed for CPU copy detection
*EgressFilter*:
* Moves EgressFilter from ingress to egress pipeline to free one ingress stage for NatEgressFilter
* Adds nat_egress_hit to bridge header to carry NAT state across TM boundary
* Control Plane: updates table path and match key (egress_port instead of ucast_egress_port)
*Egress MacRewrite*:
* Creates separate unicast_mac_rewrite and mcast_mac_rewrite instances of MacRewrite
* Unicast instance: rewrites src_mac only (dst_mac already correct from ARP/NDP)
* Multicast instance: derives dst_mac from group address per RFC 1112/2464
*Multicast Egress changes*:
* Checks egress_rid != 0 first to identify PRE-replicated packets
* Validates is_mcast_routed for CPU copies (egress_rid == 0 but routed to multicast)
* Drops any CPU copies with DROP_MULTICAST_CPU_COPY reason
*Counters*:
* Moves Unicast, MulticastLL, EgressDropPort, EgressDropReason counters from multicast to base
* Adds a Forwarded counter (for every packet copy that egresses the pipeline)
* Removes Egress counter (from ingress pipe, replaced by per-port counters)
* Removes MulticastDrop from MULTICAST_COUNTERS, as it's covered in general drop w/ reason
* Handles link-local multicast counting in both MULTICAST and non-MULTICAST paths
*Build*:
* Configures TOFINO_STAGES where there's 15 for base, 18 for multicast (the win!)
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
from
April 6, 2026 19:04
079b582 to
808e4dd
Compare
Contributor
Author
|
Decided to target |
Contributor
Author
|
@taspelund + @FelixMcFelix, as promised, pinging you both here too. |
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
from
May 19, 2026 01:31
88d6b41 to
1df493a
Compare
The sidecar pipeline lands at 16 ingress stages without multicast and 19 with multicast, all without sacrificing unicast LPM capacity in either build. Includes: * Merges origin/main, picking up the RFD 619 dpd-api refactor (#259). * v6 unicast TTL=1 handling via per-prefix skip_ttl bit + inline check (saves a stage vs the v4-style compound TTL key). Service-port routes skip the TTL exception so userspace still receives the packet. This helped after the table-sizing changes on main (and making MCAST align). * Multicast tables are now sized per-workload (separate constants per table) rather than via a single global ceiling. * Lots of misc cleanup.
zeeshanlakhani
force-pushed
the
zl/p4-stages-18-cleanup
branch
from
May 19, 2026 01:57
1df493a to
4414b60
Compare
Contributor
Author
|
Note: If multicast is turned on, this will be needed because we're at 20 stages with unicast today. |
As part of conflict resolution, we fold TTL/service-port routing semantics into main's new `RouteTableOps` dispatch trait.
Contributor
Author
|
Closing, as we'll slowly bring this in piece-wise. |
zeeshanlakhani
added a commit
that referenced
this pull request
Sep 9, 2026
…multicast Removing the multicast feature gate got us over Tofino's 20-stage pipeline limit. The table resizing work in #268 matched the v6 LPM table size to what we had in v4, putting us at 20 stages without multicast. In #208, I explored a few different approaches to refactor our pipeline, across ingress and egress, to give us more breathing room (stage-wise). However, we want to be efficient and targeted when making changes to the sidecar. Reducing mcast table sizes was my first attempt. Route and replication tables fit in SRAM; SSM is TCAM. The result, sadly, was fewer TCAM blocks, but the same stage count. Instead, I went the direction of fulfilling a Nils'ism: > XXX: this control could be moved to the Egress pipeline if we need more space > in the Ingress pipeline. Currently unicast packets are able to bypass that > pipeline, which is why we've tacked it on here. We could probably also merge > it with the MacRewrite control, as they are both per-port settings, but that > would present some weird semantics to the control plane daemon. I moved the `EgressFilter` control to the egress pipeline to save a stage, carrying (or bridging) the NAT egress bit in the bridge header. I *did not* move the `MacRewrite` control over to handle both uni+multi(cast) in egress, but that was demonstrated in #208 for a follow-up (post-testing). With the move, 20 stages is achieved! However, the v4/v6 route tests still stopped below 8191 entries with a deficit. The exact-match route-target tables were losing capacity under four-way cuckoo placement (default). [`@ways(8)`][ways] was the trick I found that gives each key in the table eight placement choices, which now makes table tests fill all 8191 entries. Includes: - Removal of the `#ifdef MULTICAST` codegen guards. - Counter updates to match our move of the NAT egress filter - Ordering of the NAT egress filter before the CPU-copy (`egress_rid`) check, making sure NAT unicast is not treated as a multicast CPU copy. [ways]: https://github.com/p4lang/p4c/blob/a19f1c3d85a867a6288fd983f7bad505ac47d728/backends/tofino/bf-p4c/common/pragma/pragmas.cpp#L1172-L1183
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sidecar pipeline lands at 16 ingress stages without multicast and 19 with multicast, all without sacrificing unicast LPM capacity in either build, closing https://github.com /issues/269.
P4/sidecar pipeline changes
EgressFilter
Egress MacRewrite
Multicast Egress changes
Counters
Multicast table sizing
Build