feat: add gossip metrics benchmarks and CI integration

[swananan]

Summery

This PR adds baseline gossip observability and benchmark coverage before protocol optimization. #1087

It includes 4 parts:

1. new Criterion benchmarks (NetworkNode based)

Criterion Benchmarks (NetworkNode-based)

Added two new benches:

gossip_propagation:

- builds a multi-node synthetic topology,
- triggers channel updates,
- measures propagation to observer nodes,
- reports latency distribution (P50/P90/P99 summary).

GOSSIP_BENCH_NODE_COUNTS=10 GOSSIP_BENCH_MAX_OBSERVERS=8 cargo bench -p fnn --features bench --bench gossip_propagation -- --warm-up-time 1 --measurement-time 3 --sample-size 10
    Finished `bench` profile [optimized] target(s) in 0.31s
     Running benches/gossip_propagation.rs (target/release/deps/gossip_propagation-006779503b1d6065)
Gnuplot not found, using plotters backend
message_propagation_summary node_count=10 rounds=19 samples=152 min_ms=6.14 avg_ms=46.55 p50_ms=49.53 p90_ms=93.31 p99_ms=100.00 max_ms=100.07
Benchmarking gossip_channel_update_propagation_multi_node/update_to_observers/10: Warming up for 1.0000 s
Warning: Unable to complete 10 samples in 3.0s. You may wish to increase target time to 4.8s or enable flat sampling.
Benchmarking gossip_channel_update_propagation_multi_node/update_to_observers/10: Collecting 10 samples in estimated 4.7765 s (55 ite
gossip_channel_update_propagation_multi_node/update_to_observers/10
                        time:   [83.750 ms 89.134 ms 94.716 ms]
                        thrpt:  [10.558  elem/s 11.219  elem/s 11.940  elem/s]
                 change:
                        time:   [+4.3179% +16.130% +28.338%] (p = 0.01 < 0.05)
                        thrpt:  [-22.081% -13.890% -4.1392%]
                        Performance has regressed.

gossip_sync_recovery:

- takes one node offline,
- creates backlog updates while offline,
- reconnects the node,
- measures time to graph parity (recovery completion).

FNN_EVENT_WAIT_TIMEOUT_SECS=180 \
  GOSSIP_SYNC_RECOVERY_NODE_COUNTS=4 \
  GOSSIP_SYNC_RECOVERY_BACKLOG_UPDATES=20 \
  GOSSIP_SYNC_RECOVERY_RECONNECT_PEERS=1 \
  GOSSIP_SYNC_RECOVERY_PARITY_TIMEOUT_SECS=60 \
  GOSSIP_BENCH_NETWORK_INTERVAL_MS=50 \
  GOSSIP_BENCH_STORE_INTERVAL_MS=50 \
  cargo bench -p fnn --features bench --bench gossip_sync_recovery -- --warm-up-time 1 --measurement-time 3 --sample-size 10
    Finished `bench` profile [optimized] target(s) in 0.32s
     Running benches/gossip_sync_recovery.rs (target/release/deps/gossip_sync_recovery-8ba04823c18d7342)
Gnuplot not found, using plotters backend
Benchmarking gossip_sync_recovery/reconnect_to_parity/4: Warming up for 1.0000 s
Warning: Unable to complete 10 samples in 3.0s. You may wish to increase target time to 16.4s.
gossip_sync_recovery/reconnect_to_parity/4
                        time:   [1.5780 s 1.5879 s 1.5936 s]
                        thrpt:  [0.6275  elem/s 0.6298  elem/s 0.6337  elem/s]
                 change:
                        time:   [-1.0421% +0.0025% +1.0566%] (p = 1.00 > 0.05)
                        thrpt:  [-1.0456% -0.0025% +1.0531%]
                        No change in performance detected.
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) low severe
  1 (10.00%) low mild

2. new gossip metrics

Added the following metrics:

• fiber.gossip.received_broadcast_messages_total
• fiber.gossip.applied_broadcast_messages_total
• fiber.gossip.duplicate_broadcast_messages_total
• fiber.gossip.rejected_broadcast_messages_total
• fiber.gossip.received_bytes_total
• fiber.gossip.sent_bytes_total
• fiber.gossip.propagation_received_latency_ms
• fiber.gossip.propagation_applied_latency_ms
• fiber.gossip.propagation_samples_skipped_total
• fiber.gossip.active_sync_completion_ms
• fiber.gossip.active_sync_started_total
• fiber.gossip.active_sync_finished_total
• fiber.gossip.active_sync_timeout_total
• fiber.gossip.missing_dependency_messages_total
• fiber.gossip.dependency_query_requests_total
• fiber.gossip.dependency_query_items_total

Also configured dedicated histogram buckets for gossip latency metrics, using millisecond boundaries [1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 60000] to capture both fast propagation and long-tail sync delays with consistent percentile/coverage analysis.

3. new perf benchmark flow

Extended tests/perf with gossip-benchmark modes:

• steady: sends updates at a fixed interval (--interval-ms), so load is smooth and constant over time.
• burst: sends as many updates as needed to hit --burst-updates-per-window within each --burst-window-ms, so load is spiky (high short-term pressure, then wait for next window).

gossip-benchmark takes two metric snapshots (before / after) and computes deltas from new metrics.

How outputs are calculated from metrics:

• recv_msg / applied / dup / rejected / recv_bytes / sent_bytes:
  • delta of corresponding counters between snapshots.
• duplicate_rate:
  • duplicate_broadcast_messages_delta / received_broadcast_messages_delta.
• rejected_rate:
  • rejected_broadcast_messages_delta / received_broadcast_messages_delta.
• bytes_per_applied:
  • (received_bytes_delta + sent_bytes_delta) / applied_broadcast_messages_delta.
• latency avg/p50/p90/p99/max and <=50/100/200/500ms coverage:
  • computed from the delta histogram of fiber.gossip.propagation_applied_latency_ms (filtered to message_type=channel_update).
• full_convergence / avg_convergence:
  • derived from observer-side applied-latency histogram sample counts vs total updates sent.

Outputs include:

cargo run --bin run-test -- gossip-benchmark steady --duration-secs 60 --interval-ms 50 --result-mode base
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running `target/debug/run-test gossip-benchmark steady --duration-secs 60 --interval-ms 50 --result-mode base`
🎯 Starting gossip benchmark test...
📊 Parameters: load_mode=steady, duration=60s, interval=50ms, burst=0 updates/1000ms, result_mode=base

🚀 Gossip benchmark test:
  - source node: node2
  - observed nodes: node1,node3
  - workload channels: 2
    * node2 -> node1 | 0xa2f3f29e298ff0b771d8f3402d5712d921b3018380bb8b5f9a7577f232843d46 (0x97957619c4559625182ac0da0474a28e9615f507a0ad1c8e742e8b21fb6d2ab700000000)
    * node2 -> node3 | 0xc513fba9fa0abb6e662448f062379d0a0516967af325834a6093ace9b91cdd3d (0xd55f3bd0c2817a2338ce1e4b6020204fcea5951bcf05d3f23d82769504d93bd600000000)
  - duration: 60s
  - load mode: steady
  - update interval: 50ms
  - burst target: 0 updates per 1s
  - metrics endpoints: http://127.0.0.1:29114/metrics, http://127.0.0.1:29115/metrics, http://127.0.0.1:29116/metrics

🎯 Gossip benchmark results:
=====================================
Test duration: 60.00 seconds
Load profile: mode=steady, interval=50ms, burst_target=0 updates/1000ms, burst_windows=0
Scope: observer metrics = node1,node3 (combined)
Scope: network counters = node1,node2,node3 (sum delta)
Latency source: fiber.gossip.propagation_applied_latency_ms histogram (percentiles are estimates)
Total updates: 1069
Updates/sec: 17.82
Pressure send rate: peak=17.82 updates/sec, burst_updates_sent=0
Workload distribution:
  - node2 -> node1 | channel=0xa2f3f29e298ff0b771d8f3402d5712d921b3018380bb8b5f9a7577f232843d46 | updates=535
  - node2 -> node3 | channel=0xc513fba9fa0abb6e662448f062379d0a0516967af325834a6093ace9b91cdd3d | updates=534
Full convergence: 1062/1069 (99.35%)
Average convergence rate: 99.39%
Latency(ms): avg=522.70, p50=523.32, p90=919.78, p99=1475.00, max=2000.00
Latency coverage: <=50ms 2.16%, <=100ms 7.20%, <=200ms 17.88%, <=500ms 47.67%
Metrics deltas: recv_msg=7461, applied=2125, dup=501, rejected=4400, recv_bytes=1620740, sent_bytes=1636036
Derived: duplicate_rate=6.71%, rejected_rate=58.97%, bytes_per_applied=1532.60
=====================================

📊 Gossip baseline saved to: gossip_baseline.json
💾 Gossip baseline benchmark results saved to gossip_baseline.json

cargo run --bin run-test -- gossip-benchmark burst --duration-secs 60 --burst-updates-per-window 200 --burst-window-ms 1000  --result-mode base
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/run-test gossip-benchmark burst --duration-secs 60 --burst-updates-per-window 200 --burst-window-ms 1000 --result-mode base`
🎯 Starting gossip benchmark test...
📊 Parameters: load_mode=burst, duration=60s, interval=0ms, burst=200 updates/1000ms, result_mode=base
🚀 Gossip benchmark test:
  - source node: node2
  - observed nodes: node1,node3
  - workload channels: 2
    * node2 -> node1 | 0xa2f3f29e298ff0b771d8f3402d5712d921b3018380bb8b5f9a7577f232843d46 (0x97957619c4559625182ac0da0474a28e9615f507a0ad1c8e742e8b21fb6d2ab700000000)
    * node2 -> node3 | 0xc513fba9fa0abb6e662448f062379d0a0516967af325834a6093ace9b91cdd3d (0xd55f3bd0c2817a2338ce1e4b6020204fcea5951bcf05d3f23d82769504d93bd600000000)
  - duration: 60s
  - load mode: burst
  - update interval: 0ns
  - burst target: 200 updates per 1s
  - metrics endpoints: http://127.0.0.1:29114/metrics, http://127.0.0.1:29115/metrics, http://127.0.0.1:29116/metrics

🎯 Gossip benchmark results:
=====================================
Test duration: 60.00 seconds
Load profile: mode=burst, interval=0ms, burst_target=200 updates/1000ms, burst_windows=60
Scope: observer metrics = node1,node3 (combined)
Scope: network counters = node1,node2,node3 (sum delta)
Latency source: fiber.gossip.propagation_applied_latency_ms histogram (percentiles are estimates)
Total updates: 12000
Updates/sec: 200.00
Pressure send rate: peak=529.79 updates/sec, burst_updates_sent=12000
Workload distribution:
  - node2 -> node1 | channel=0xa2f3f29e298ff0b771d8f3402d5712d921b3018380bb8b5f9a7577f232843d46 | updates=6000
  - node2 -> node3 | channel=0xc513fba9fa0abb6e662448f062379d0a0516967af325834a6093ace9b91cdd3d | updates=6000
Full convergence: 12000/12000 (100.00%)
Average convergence rate: 100.00%
Latency(ms): avg=638.98, p50=716.04, p90=943.21, p99=994.32, max=1000.00
Latency coverage: <=50ms 0.00%, <=100ms 0.00%, <=200ms 0.00%, <=500ms 11.96%
Metrics deltas: recv_msg=83400, applied=24000, dup=477, rejected=58921, recv_bytes=18069600, sent_bytes=18387840
Derived: duplicate_rate=0.57%, rejected_rate=70.65%, bytes_per_applied=1519.06
=====================================

📊 Gossip baseline saved to: gossip_baseline.json
💾 Gossip baseline benchmark results saved to gossip_baseline.json

Supports baseline/compare report generation.

4. CI updates to run and compare benchmarks safely

Updated benchmark CI flow to include gossip benchmark in a safe bootstrap way:

• keeps baseline/current startup symmetric,
• only runs gossip compare when baseline branch supports required metrics flow,
• otherwise runs bootstrap mode and skips gossip regression gating,
• includes clear TODO/comments to switch to strict always-on compare later.

Notes

One open question is whether we should run these benchmarks with production-like gossip intervals (60s/20s) instead of test-friendly intervals. Current settings are chosen to keep runtime practical and feedback fast, but they may not fully reflect production behavior. I’d like feedback on whether we should switch to production values (or keep both fast-mode and prod-mode runs).

This PR is shared early to discuss and validate the overall direction first (metrics scope, benchmark design, and CI strategy) before moving to deeper optimization changes.

[chenyukang]

I prefer to run as production-like gossip intervals first, and gossip performance benchmark may don't need to be in CI, since we will not change this part frequently.

it's good if we can run benchmark for this part for every release, or we just track the performance change by running a cron job like this: https://sunchengzhu.github.io/fiber-ci-metrics/

@quake what's your opinions for metrics?

[swananan]

Sure, I’ve switched the benchmarks to production-like gossip intervals. For the perf tests, we can use macros to configure the interval settings. And I also removed the ci changes.

Gossip Benchmark result

FNN_EVENT_WAIT_TIMEOUT_SECS=180 \
  GOSSIP_SYNC_RECOVERY_NODE_COUNTS=4 \
  GOSSIP_SYNC_RECOVERY_BACKLOG_UPDATES=20 \
  GOSSIP_SYNC_RECOVERY_RECONNECT_PEERS=1 \
  GOSSIP_SYNC_RECOVERY_PARITY_TIMEOUT_SECS=1200 \
  cargo bench -p fnn --features bench --bench gossip_sync_recovery -- --warm-up-time 1 --measurement-time 3 --sample-size 10
    Finished `bench` profile [optimized] target(s) in 0.29s
     Running benches/gossip_sync_recovery.rs (target/release/deps/gossip_sync_recovery-8ba04823c18d7342)
Gnuplot not found, using plotters backend
Benchmarking gossip_sync_recovery/reconnect_to_parity/4: Warming up for 1.0000 s
Warning: Unable to complete 10 samples in 3.0s. You may wish to increase target time to 813.2s.
gossip_sync_recovery/reconnect_to_parity/4
                        time:   [81.257 s 81.272 s 81.286 s]
                        thrpt:  [0.0123  elem/s 0.0123  elem/s 0.0123  elem/s]
                 change:
                        time:   [-0.0337% -0.0101% +0.0140%] (p = 0.44 > 0.05)
                        thrpt:  [-0.0140% +0.0101% +0.0337%]
                        No change in performance detected.
Found 3 outliers among 10 measurements (30.00%)
  1 (10.00%) low severe
  1 (10.00%) low mild
  1 (10.00%) high mild

[chenyukang]

@swananan is this PR ready to be merged?
or do you want to continue to work on the optimization on this branch.

[swananan]

@swananan is this PR ready to be merged? or do you want to continue to work on the optimization on this branch.

This PR is ready for review. The main thing I’d still like to discuss is the metrics data design, but if that looks good, I’d be happy to merge this PR and continue the optimization work on a separate branch.

[chenyukang]

@quake
This PR adds baseline observability and benchmark coverage for Fiber gossip before optimization work.

I think it's ok to be merge (some trivial comments), you can also take a review.

[quake]

@quake This PR adds baseline observability and benchmark coverage for Fiber gossip before optimization work.

I think it's ok to be merge (some trivial comments), you can also take a review.

lgtm, only two trivial comments from my side.

[chenyukang]

link #1263, since this PR may have effect on gossip benchmarks.