Skip to content

v0.22.0+ciris.1 — live-link mirror fix, event reclassification, drop-log ladder

Choose a tag to compare

@emooreatx emooreatx released this 20 Aug 00:18
· 852 commits to main since this release

First fixes from the live-canonical incident (CIRISEdge#508 → leviculum#56–#60). Two defects fixed, one flood cut off at the source, one diagnosis answered, one refactor argued against with measurements.

The completion mirror no longer evicts a live link (#56)

ESTABLISHED_MIRROR_CAP evicted FIFO over live entries, so the oldest link lost its completion first — and the oldest links are the long-lived canonical peers whose completions matter most. 130 live evictions were observed in the field, each silently degrading an await_link_established to "resolves only at LinkClosed or node stop".

The comment justifying that cap ("well above the realistic concurrent-link envelope") was falsified by production, and the trade was bad regardless: the mirror self-cleans on LinkClosed, so its occupancy is the live set, and a LinkId is 16 bytes. The cap is now an alarm, not an evictor — 80% warns, the envelope logs an error naming the lifecycle question, growth past it reports on a doubling ladder, and nothing is ever dropped.

Removing the FIFO VecDeque also deleted an O(n) scan taken under the registry lock on every link close.

Resource events classified by what they carry (#59)

ResourceTransferStarted + ResourceCompleted were 86% of everything dropped from the lossless plane (556 + 630), crowding out link and path liveness. The split now follows content, not variant name:

  • receiver-side ResourceCompleted stays lossless at every segment — its data field is the delivered payload, so dropping one would be silent data loss, strictly worse than the flood it relieves;
  • sender-side final-segment stays lossless — the transfer outcome, and what a completion await resolves on;
  • sender-side intermediate segments and ResourceTransferStarted become droppable — progress on our own upload, already covered by ResourceProgress and superseded by the final completion.

Completion futures are unaffected: the registry observes at dispatch, before the sink.

Control-plane drops log on a ladder (#60, property 1)

One WARN per dropped event produced 1014 near-identical lines in 2000, burying the CONTROL_PLANE_OVERFLOW marker 30:1. Drops now log at 1, 2, 4, 8 … each carrying the running total and naming the next report point; the ladder resets when the marker flushes. No drop goes uncounted.

Not changed, deliberately (#58)

The ask was to split or shard the registry lock. Measured first: a stand-in with the registry's exact hot-path shape sustains 9.6M events/sec/core, against an offered 0.76 events/sec — seven orders of magnitude of headroom. futex_wait_queue_me on tokio workers is the idle state, and the reported 6.1M nonvoluntary vs 993k voluntary context switches is the signature of CPU-bound work against a container ceiling, not of blocking (blocking yields voluntarily). The issue stays open because the symptom is real and unexplained; a profile would settle it, and the split gets built the moment evidence points there.

Also

Catch-up to upstream 3b52f66a (+2: nRF/LoRa CRC counters, tier-3 CI list gate). h2 0.4.16 re-applied for RUSTSEC-2026-0258 (the rebase takes upstream's lockfile, which reverts it).

Gate: core 1,649 / std 587 / scoped-transit 6/6 / destination-lifecycle 3/3 / request-response 3/3; clippy -D warnings; fmt; thumbv6m; full CI green.