Skip devices that are not calling the controller by martinsander00 · Pull Request #2769 · malbeclabs/doublezero

martinsander00 · 2026-01-30T20:49:30Z

Resolves: #2721

Summary of Changes

If Grafana env vars are set → filters out inactive devices
If Grafana query fails → logs warning and continues with all devices (graceful degradation)
If env vars not set → skips filtering entirely (backwards compatible)

Testing Verification

doublezero/e2e ms/2721 ✗ go test -tags=qa -v -timeout=10m github.com/malbeclabs/doublezero/e2e -run "TestQA_AllDevices_UnicastConnectivity" --args  -hosts=chi-dn-bm2,chi-dn-bm3,chi-dn-bm4 -env=devnet
=== RUN   TestQA_AllDevices_UnicastConnectivity
    qa_test.go:67: 2026-02-02T18:35:31.944Z INF Initializing client host=chi-dn-bm2 publicIP=137.174.145.145
    qa_test.go:67: 2026-02-02T18:35:32.040Z INF Initializing client host=chi-dn-bm3 publicIP=137.174.145.147
    qa_test.go:67: 2026-02-02T18:35:32.259Z INF Initializing client host=chi-dn-bm4 publicIP=137.174.145.149
    qa_test.go:67: 2026-02-02T18:35:32.259Z INF Skipping device with insufficient capacity device=chi-dn-dzd3-delete-me users=0 maxUsers=0
    qa_test.go:67: 2026-02-02T18:35:32.455Z INF Filtering devices by controller activity activeDeviceCount=101
    qa_test.go:67: 2026-02-02T18:35:32.455Z INF Collect `doublezero latency` for each client
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF Create a list of devices for each client.
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF     (If there are multiple clients with <25ms latency for that device, assign the device to the client with the fewest devices
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF     Otherwise, associate each device with the client with the lowest latency)
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF Assign devices to clients based on latency
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF Client device assignments:
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF 
        |         | chi-dn-bm2              | chi-dn-bm3               | chi-dn-bm4              |
        |---------|-------------------------|--------------------------|-------------------------|
        | Batch 1 | chi-dn-dzd1 (0.2ms) 🕒  | chi-dn-dzd2 (0.2ms) 🕒   | chi-dn-dzd3 (0.4ms) 🕒  |
        | Batch 2 | chi-dn-dzd1 (0.2ms) 🕒  | chi-dn-dzd4 (20.2ms) 🕒  | chi-dn-dzd3 (0.4ms) 🕒  |
        
    qa_test.go:67: 2026-02-02T18:35:33.122Z INF Planning to test deviceCount=4 clientCount=3 totalBatches=2
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_1
=== NAME  TestQA_AllDevices_UnicastConnectivity
    qa_test.go:67: 2026-02-02T18:35:34.043Z INF Connecting unicast user host=chi-dn-bm2 device=chi-dn-dzd1 allocateAddr=false
    qa_test.go:67: 2026-02-02T18:35:35.437Z INF Unicast user connected host=chi-dn-bm2 device=chi-dn-dzd1
    qa_test.go:67: 2026-02-02T18:35:36.134Z INF Connecting unicast user host=chi-dn-bm3 device=chi-dn-dzd2 allocateAddr=false
    qa_test.go:67: 2026-02-02T18:35:38.407Z INF Unicast user connected host=chi-dn-bm3 device=chi-dn-dzd2
    qa_test.go:67: 2026-02-02T18:35:39.733Z INF Connecting unicast user host=chi-dn-bm4 device=chi-dn-dzd3 allocateAddr=false
    qa_test.go:67: 2026-02-02T18:35:41.584Z INF Unicast user connected host=chi-dn-bm4 device=chi-dn-dzd3
    qa_test.go:67: 2026-02-02T18:36:06.271Z INF Waiting for routes to be installed host=chi-dn-bm2 expectedIPs="[137.174.145.147 137.174.145.149]"
    qa_test.go:67: 2026-02-02T18:36:06.296Z INF Waiting for routes to be installed host=chi-dn-bm3 expectedIPs="[137.174.145.145 137.174.145.149]"
    qa_test.go:67: 2026-02-02T18:36:06.309Z INF Waiting for routes to be installed host=chi-dn-bm4 expectedIPs="[137.174.145.145 137.174.145.147]"
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.145 target=137.174.145.149 iface=doublezero0 sourceExchange=xchi targetExchange=xlax
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.147 target=137.174.145.149 iface=doublezero0 sourceExchange=xpit targetExchange=xlax
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.149 target=137.174.145.147 iface=doublezero0 sourceExchange=xlax targetExchange=xpit
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.145 target=137.174.145.147 iface=doublezero0 sourceExchange=xchi targetExchange=xpit
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.147 target=137.174.145.145 iface=doublezero0 sourceExchange=xpit targetExchange=xchi
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:06.354Z INF Pinging source=137.174.145.149 target=137.174.145.145 iface=doublezero0 sourceExchange=xlax targetExchange=xchi
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:10.460Z INF Successfully pinged sourceHost=chi-dn-bm3 targetHost=chi-dn-bm2 iface=doublezero0 sourceDevice=chi-dn-dzd2 targetDevice=chi-dn-dzd1 packetsSent=5 packetsReceived=5 attempts=1
    qa_test.go:67: 2026-02-02T18:36:10.460Z INF Successfully pinged sourceHost=chi-dn-bm3 targetHost=chi-dn-bm4 iface=doublezero0 sourceDevice=chi-dn-dzd2 targetDevice=chi-dn-dzd3 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:10.461Z INF Successfully pinged sourceHost=chi-dn-bm2 targetHost=chi-dn-bm3 iface=doublezero0 sourceDevice=chi-dn-dzd1 targetDevice=chi-dn-dzd2 packetsSent=5 packetsReceived=5 attempts=1
    qa_test.go:67: 2026-02-02T18:36:10.483Z INF Successfully pinged sourceHost=chi-dn-bm2 targetHost=chi-dn-bm4 iface=doublezero0 sourceDevice=chi-dn-dzd1 targetDevice=chi-dn-dzd3 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:10.518Z INF Successfully pinged sourceHost=chi-dn-bm4 targetHost=chi-dn-bm2 iface=doublezero0 sourceDevice=chi-dn-dzd3 targetDevice=chi-dn-dzd1 packetsSent=5 packetsReceived=5 attempts=1
    qa_test.go:67: 2026-02-02T18:36:10.527Z INF Successfully pinged sourceHost=chi-dn-bm4 targetHost=chi-dn-bm3 iface=doublezero0 sourceDevice=chi-dn-dzd3 targetDevice=chi-dn-dzd2 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity
    qa_test.go:67: 2026-02-02T18:36:10.527Z INF 
        |         | chi-dn-bm2            | chi-dn-bm3            | chi-dn-bm4            |
        |---------|-----------------------|-----------------------|-----------------------|
        | Batch 1 | chi-dn-dzd1 10/10 ✅  | chi-dn-dzd2 10/10 ✅  | chi-dn-dzd3 10/10 ✅  |
        
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_2
=== NAME  TestQA_AllDevices_UnicastConnectivity
    qa_test.go:67: 2026-02-02T18:36:11.331Z INF Disconnecting user host=chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:12.846Z INF Connecting unicast user host=chi-dn-bm3 device=chi-dn-dzd4 allocateAddr=false
    qa_test.go:67: 2026-02-02T18:36:13.989Z INF Unicast user connected host=chi-dn-bm3 device=chi-dn-dzd4
    qa_test.go:67: 2026-02-02T18:36:34.961Z INF Waiting for routes to be installed host=chi-dn-bm2 expectedIPs=[137.174.145.149]
    qa_test.go:67: 2026-02-02T18:36:34.988Z INF Waiting for routes to be installed host=chi-dn-bm3 expectedIPs=[137.174.145.149]
    qa_test.go:67: 2026-02-02T18:36:35.008Z INF Waiting for routes to be installed host=chi-dn-bm4 expectedIPs="[137.174.145.145 137.174.145.147]"
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
=== PAUSE TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
=== CONT  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging source=137.174.145.145 target=137.174.145.149 iface=doublezero0 sourceExchange=xchi targetExchange=xlax
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging source=137.174.145.149 target=137.174.145.147 iface=doublezero0 sourceExchange=xlax targetExchange=xchi
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging source=137.174.145.147 target=137.174.145.149 iface=doublezero0 sourceExchange=xchi targetExchange=xlax
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging source=137.174.145.149 target=137.174.145.145 iface=doublezero0 sourceExchange=xlax targetExchange=xchi
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging (intra-exchange routing) source=137.174.145.147 target=137.174.145.145 exchange=xchi
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:35.054Z INF Pinging (intra-exchange routing) source=137.174.145.145 target=137.174.145.147 exchange=xchi
    qa_test.go:67: 2026-02-02T18:36:39.132Z INF Successfully pinged sourceHost=chi-dn-bm2 targetHost=chi-dn-bm3 sourceDevice=chi-dn-dzd1 targetDevice=chi-dn-dzd4 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:39.132Z INF Successfully pinged sourceHost=chi-dn-bm3 targetHost=chi-dn-bm4 iface=doublezero0 sourceDevice=chi-dn-dzd4 targetDevice=chi-dn-dzd3 packetsSent=5 packetsReceived=5 attempts=1
    qa_test.go:67: 2026-02-02T18:36:39.132Z INF Successfully pinged sourceHost=chi-dn-bm3 targetHost=chi-dn-bm2 sourceDevice=chi-dn-dzd4 targetDevice=chi-dn-dzd1 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:39.195Z INF Successfully pinged sourceHost=chi-dn-bm2 targetHost=chi-dn-bm4 iface=doublezero0 sourceDevice=chi-dn-dzd1 targetDevice=chi-dn-dzd3 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4
    qa_test.go:67: 2026-02-02T18:36:39.206Z INF Successfully pinged sourceHost=chi-dn-bm4 targetHost=chi-dn-bm2 iface=doublezero0 sourceDevice=chi-dn-dzd3 targetDevice=chi-dn-dzd1 packetsSent=5 packetsReceived=5 attempts=1
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF Successfully pinged sourceHost=chi-dn-bm4 targetHost=chi-dn-bm3 iface=doublezero0 sourceDevice=chi-dn-dzd3 targetDevice=chi-dn-dzd4 packetsSent=5 packetsReceived=5 attempts=1
=== NAME  TestQA_AllDevices_UnicastConnectivity
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF 
        |         | chi-dn-bm2            | chi-dn-bm3            | chi-dn-bm4            |
        |---------|-----------------------|-----------------------|-----------------------|
        | Batch 2 | chi-dn-dzd1 10/10 ✅  | chi-dn-dzd4 10/10 ✅  | chi-dn-dzd3 10/10 ✅  |
        
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF Test results:
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF 
        |         | chi-dn-bm2            | chi-dn-bm3            | chi-dn-bm4            |
        |---------|-----------------------|-----------------------|-----------------------|
        | Batch 1 | chi-dn-dzd1 10/10 ✅  | chi-dn-dzd2 10/10 ✅  | chi-dn-dzd3 10/10 ✅  |
        | Batch 2 | chi-dn-dzd1 10/10 ✅  | chi-dn-dzd4 10/10 ✅  | chi-dn-dzd3 10/10 ✅  |
        
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF Test summary packetsReceived=60 packetsSent=60 batchesWithLoss=0 totalBatches=2
    qa_test.go:67: 2026-02-02T18:36:39.231Z INF Metrics publishing skipped: no InfluxDB configuration
    qa_test.go:67: 2026-02-02T18:36:39.394Z INF Disconnecting user host=chi-dn-bm3
    qa_test.go:67: 2026-02-02T18:36:39.472Z INF Disconnecting user host=chi-dn-bm2
    qa_test.go:67: 2026-02-02T18:36:39.593Z INF Disconnecting user host=chi-dn-bm4
--- PASS: TestQA_AllDevices_UnicastConnectivity (71.29s)
    --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_1 (33.23s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd2__from_chi-dn-bm3 (4.11s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd1__from_chi-dn-bm2 (4.13s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_1/device_chi-dn-dzd3__from_chi-dn-bm4 (4.17s)
    --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_2 (23.90s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd4__from_chi-dn-bm3 (4.08s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd1__from_chi-dn-bm2 (4.14s)
        --- PASS: TestQA_AllDevices_UnicastConnectivity/batch_2/device_chi-dn-dzd3__from_chi-dn-bm4 (4.18s)
PASS
ok      github.com/malbeclabs/doublezero/e2e    71.292s

e2e/qa_alldevices_unicast_test.go

e2e/internal/qa/metrics.go

CHANGELOG.md

Resolves: #2721

martinsander00 force-pushed the ms/2721 branch from f4190e9 to e6f5f68 Compare January 30, 2026 20:54

martinsander00 requested a review from nikw9944 January 30, 2026 20:55

martinsander00 self-assigned this Feb 2, 2026

nikw9944 reviewed Feb 2, 2026

View reviewed changes

e2e/qa_alldevices_unicast_test.go Show resolved Hide resolved

martinsander00 force-pushed the ms/2721 branch from 81ebc02 to ee93edf Compare February 2, 2026 19:05

martinsander00 requested a review from nikw9944 February 2, 2026 19:05

nikw9944 requested changes Feb 2, 2026

View reviewed changes

e2e/internal/qa/metrics.go Show resolved Hide resolved

e2e/internal/qa/metrics.go Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

martinsander00 force-pushed the ms/2721 branch from ee93edf to 238d597 Compare February 2, 2026 19:17

martinsander00 requested a review from nikw9944 February 2, 2026 19:17

martinsander00 force-pushed the ms/2721 branch from 400d903 to b70c79b Compare February 2, 2026 23:05

Skip devices that are not calling the controller

a931fb8

Resolves: #2721

martinsander00 force-pushed the ms/2721 branch from b70c79b to a931fb8 Compare February 3, 2026 16:17

nikw9944 approved these changes Feb 3, 2026

View reviewed changes

martinsander00 merged commit a438ac5 into main Feb 3, 2026
25 of 26 checks passed

martinsander00 deleted the ms/2721 branch February 3, 2026 17:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip devices that are not calling the controller#2769

Skip devices that are not calling the controller#2769
martinsander00 merged 1 commit intomainfrom
ms/2721

martinsander00 commented Jan 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

martinsander00 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Testing Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martinsander00 commented Jan 30, 2026 •

edited

Loading