modem.py

[greatgitsby]

resolves: #37277
moved from: commaai/agnos-builder#556

Testing

Tested on mici (comma four, EG916Q-GL) and tizi (comma 3X, EG25-G) with Connect and Webbing eSIM profiles.

Boot & connect

$ ssh comma@10.0.0.22 "cat /dev/shm/modem | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d[\"state\"], d[\"iccid\"][-6:], d[\"operator\"], d[\"ip_address\"])'"
connected 463550 T-Mobile 10.26.216.207

Data connectivity through ppp0

$ ssh comma@10.0.0.22 "curl --interface ppp0 -s --connect-timeout 10 http://ifconfig.me"
38.86.196.156

Route management (wifi preferred, ppp0 fallback)

$ ssh comma@10.0.0.22 "ip route show default; echo ---; ip rule show | grep 1000"
default via 10.0.0.1 dev wlan0 proto dhcp src 10.0.0.22 metric 600
default via 10.64.64.64 dev ppp0 metric 1000
---
32765:  from 10.26.216.207 lookup 1000

APN change detection

$ ssh comma@10.0.0.22 "echo -n 'globaldata' > /data/params/d/GsmApn"
# detected in ~1s; full reconnect ~19s

Roaming toggle

$ ssh comma@10.0.0.22 "echo -n '0' > /data/params/d/GsmRoaming"
# pppd killed, state → registering with error=roaming_disabled within ~15s
$ ssh comma@10.0.0.22 "echo -n '1' > /data/params/d/GsmRoaming"
# reconnects in ~12s

Profile switching

$ ssh comma@10.0.0.22 "python3 system/hardware/esim.py --switch <iccid>"
# +QUSIM URC detected, pppd killed, full re-init on new SIM

Carrier error surfacing

{"state": "registering", "registration": "denied", "carrier_error": "EMM_CAUSE_PLMN_NOT_ALLOWED (check GsmApn)"}

Stress testing

See stress test comment and additional stress comment for the full matrix: pppd kills (single + 10× back-to-back + 4× rapid), AT-lock contention, APN/roaming thrash, USB re-enumeration mid-connection, sustained traffic with intermittent kills, SIGTERM cleanup, orphan pppd cleanup, 10-concurrent-reader churn (59,951 reads / 0 errors), garbage state file recovery, and 5-min idle soak.

Bonus: full-reboot → ping <60s

See bonus comment. Mean 49.1s across 7 trials (min 45.3s, max 52.0s), all under the 60s target.

[github-actions]

Process replay diff report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

✅ 0 changed, 66 passed, 0 errors

[greatgitsby]

Stress test results

Ran on mici (comma four, EG916Q-GL) with a Connect eSIM and APN=globaldata.

#	Test	Result	Metric
1	Baseline ping + HTTPS via ppp0	✅	5/5 ping, 138ms avg RTT; HTTPS 706ms
2	sudo killall -9 pppd (1×)	✅	Reconnected in ~6.5s
3	External flock /dev/shm/modem_lpa.lock for 30s	✅	No crash, stayed connected, AT polls degraded gracefully
5	GsmApn param change at runtime	✅	Detected in 1.3s; full reconnect ~19s
6	GsmRoaming off→on	✅	Off: pppd killed + error=roaming_disabled in ~15s. On: reconnect in ~12s
8	sudo killall -9 pppd ×10 back-to-back	✅	3-fails→RECONNECTING cycle triggers cleanly; no state file corruption
9	10 concurrent json.load readers of /dev/shm/modem for 30s	✅	43,374 reads, 0 errors
10	Download 1MB via ppp0, verify counters	✅	rx_bytes delta 1,045,070 (~4.5% TCP/TLS overhead); tx_bytes plausible
11	5-minute idle	✅	Stayed connected, counters kept updating
12	carrier_error field surfaces AT+CEER reason on denied registration	✅	E.g. EMM_CAUSE_PLMN_NOT_ALLOWED (check GsmApn)

Skipped: Test 4 (SIM-change URC injection — no safe way to synthesize externally; path exercised organically during eSIM profile switches). Test 7 (GPIO reset mid-connection — skipped for safety).

[greatgitsby]

Bonus target ($500): full reboot → ping < 60s

Result: 49.1s mean (n=7, min 45.3s, max 52.0s) on comma four with a Connect eSIM (APN=globaldata). All trials comfortably under 60s.

trial	uptime→ping	elapsed (script→ping)
1	50.02s	38.51s
2	48.38s	36.62s
3	45.25s	33.42s
4	49.59s	35.43s
5	48.95s	36.59s
6	51.97s	36.97s
7	49.25s	37.72s

Reproduction

/var is tmpfs on AGNOS (cron @reboot doesn't persist), and /etc is read-only, so the cleanest harness is a script in /data/ invoked from launch_chffrplus.sh at boot. Uses /proc/uptime (monotonic, unaffected by wall-clock sync).

# Install timing script on device
ssh comma@DEVICE "cat > /data/ping_timing.sh << 'SH'
#!/bin/bash
exec > /data/ping_timing.log 2>&1
t_script_start=\$(awk '{print \$1}' /proc/uptime)
echo \"script_start_uptime=\${t_script_start}s\"
while true; do
  up=\$(awk '{print \$1}' /proc/uptime)
  if sudo -n ping -I ppp0 -c 1 -W 2 -q 8.8.8.8 > /dev/null 2>&1; then
    echo \"PING_OK uptime=\${up}s elapsed=\$(awk -v a=\${up} -v b=\${t_script_start} 'BEGIN{print a-b}')s\"
    break
  fi
  [ \"\$(awk -v u=\${up} 'BEGIN{print (u>300)?1:0}')\" = 1 ] && { echo TIMEOUT; break; }
  sleep 1
done
cat /dev/shm/modem 2>/dev/null >> /data/ping_timing.log
SH
chmod +x /data/ping_timing.sh"

# Hook into openpilot launch (temporary — revert after measurement)
ssh comma@DEVICE "sed -i '/^    agnos_init\$/a\\    /data/ping_timing.sh &' /data/openpilot/launch_chffrplus.sh"

# Reboot, then read log
ssh comma@DEVICE "sudo reboot"
# ...wait for device, then:
ssh comma@DEVICE "cat /data/ping_timing.log"

Sample output

script_start_uptime=11.53s
PING_OK uptime=49.25s elapsed=37.72s
{"state": "connected", ..., "operator": "T-Mobile", "network_type": "lte", "registration": "roaming", ...}

[greatgitsby]

Additional stress testing

More chaos tests, all on mici (comma four, EG916Q-GL).

#	Test	Result	Notes
13	APN thrash (10 changes @ 5s)	✅	State machine collapses rapid flips; only latest APN read in final _do_init. Recovers to connected.
14	Roaming thrash (10 flips @ 3s)	✅	Same collapsing behavior; recovers to connected with error cleared.
15	4 pppd kills in 8s (within one retry window)	✅	Only PPP fail 1/3 logged; rapid kills hit an already-dead process. No re-entrancy issues.
16	USB re-enumeration mid-connection (lte.sh stop_blocking + start)	✅	State → reconnecting when port disappears, → waiting_port → port found → full re-init → connected. ~32s end-to-end.
17	60s sustained ppp0 traffic + pppd kill every 15s (×4)	✅	Traffic drops during reconnect window; counters persist; state stays consistent.
18	SIGTERM to modem.py	✅ (shutdown)	stop() runs clean: pppd killed, state file removed. Note: under openpilot manager there's no auto-respawn after SIGTERM; requires systemctl restart comma.
19	Orphan pppd from previous run	✅	_kill_ppp in _do_init clears it; fresh modem.py reaches connected cleanly.
20	10 concurrent readers for 40s while hammering APN param (20 flips @ 2s)	✅	59,951 successful reads, 0 errors. os.replace holds up under real concurrent churn.
21	Edge cases: empty GsmApn file + garbage bytes written to /dev/shm/modem	✅	Empty file treated as auto-APN. Garbage state file overwritten by next _update() within seconds.