Skip to content

Shard egress-lane VMs over several raw bridges - #59

Merged
CMGS merged 1 commit into
mainfrom
feat/bridges-shards
Jul 30, 2026
Merged

Shard egress-lane VMs over several raw bridges#59
CMGS merged 1 commit into
mainfrom
feat/bridges-shards

Conversation

@CMGS

@CMGS CMGS commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

#58 gave the egress lane its shard list, but over CNI conflists — and the CNI attachment is what it prices in: every clone pays a network namespace, a CNI plugin execution, a veth pair and a TC redirect. Measured on a 384-core host filling 1000 egress VMs to exec-probed warm (quiet console, udev exec queue stopped, refill_concurrency: 64, same harness for every row):

attachment rate note
networks ×4 (#58, CNI) 25.8/s spread even: 261/235/248/257
bridge ×1 (raw tap) 48.0/s the single-bridge ceiling story from #58 applies
bridges ×4 (this PR, raw tap) 83.3/s spread even: 245/233/250/272
4 separate daemons, one bridge each, meshed 82.2/s for context — this PR matches it in one process, one golden set

What it adds

bridges — a list of host bridge devices, sharded by the same stable VM-name hash networks uses, with the raw TAP-on-bridge attachment kept: one tap in the root netns, one RTM_SETLINK, no netns, no plugin process, no veth. Each entry is one Linux bridge, so N entries give N×BR_MAX_PORTS exactly like the conflist form.

Two consequences fall out of the taps staying in the root netns:

  • Guarded egress works over bridges. The nft NIC lock needs the tap reachable from the root netns, which is why guarded egress rejects networks at load; a bridges list satisfies it, so the guarded lane is no longer capped at one bridge.
  • A one-entry list is byte-for-byte the scalar bridge argv.

Shape

  • The scalar bridge key is retired the same way network was in Spread egress-lane VMs over several CNI networks #58: strict config decoding rejects it loudly, and the deploy docs carry the one-line rename ("bridge": "br0""bridges": ["br0"]). bridges and networks are mutually exclusive, with the same empty-name/duplicate validation.
  • engine: networkFor generalises to shardOf(shards, name) — same fnv hash, so a persisted record's attachment stays reproducible without process state, for either form.
  • lockEgress arms for any bridge form.
  • Docs: the deploy table row and the guarded-egress bridge-lane note now name both forms.

Verification

make go-lint (golangci, GOOS linux+darwin) 0 issues; go test -race green on config/engine/pool and e2e; gofmt clean. On hardware: the 1000-VM fill above, hash spread within 9% of even across four bridges, and the pool drained clean afterwards. Only refill_concurrency: 64 was measured for the new form.

@CMGS
CMGS force-pushed the feat/bridges-shards branch 3 times, most recently from 94247d2 to 3ecee7d Compare July 30, 2026 07:26
The networks list shards over CNI conflists, and the CNI attachment is what
it prices in: a per-VM network namespace, a CNI plugin execution, a veth
pair and a TC redirect on every clone. Measured on a 384-core host filling
1000 egress VMs (quiet console, udev exec queue stopped), the conflist
shards ran at 25.8/s where a single raw bridge ran 48/s — the sharding win
was smaller than the machinery cost.

bridges is the same shard list over host bridge devices with the raw
TAP-on-bridge attachment kept: one tap in the root netns, no netns, no
plugin, no veth. The same fill runs at 83.3/s — 3.2x the conflist shards,
level with four separate daemons sharding one bridge each without the
fourfold goldens — and the ceiling story is unchanged: each entry is one
Linux bridge, so N entries give N x BR_MAX_PORTS. Because the taps stay in
the root netns, guarded egress works over bridges, which the conflist form
cannot offer.

The shard pick is the same VM-name hash networks uses, so a record's
attachment stays reproducible without process state. The scalar bridge key
is retired like network before it — a one-entry bridges list is the same
argv, and strict config decoding fails the old key loudly.
@CMGS
CMGS force-pushed the feat/bridges-shards branch from 3ecee7d to 9059b83 Compare July 30, 2026 07:33
@CMGS
CMGS merged commit c2afb44 into main Jul 30, 2026
1 check passed
@CMGS
CMGS deleted the feat/bridges-shards branch July 30, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants