Skip to content

lf_ftp.py — Robot/band-steering resilience to unresponsive devices + cleaner missing-CX logging - #326

Merged
goyalsaurabh06 merged 5 commits into
greearb:masterfrom
goyalsaurabh06:ftp_updates
Aug 3, 2026
Merged

lf_ftp.py — Robot/band-steering resilience to unresponsive devices + cleaner missing-CX logging#326
goyalsaurabh06 merged 5 commits into
greearb:masterfrom
goyalsaurabh06:ftp_updates

Conversation

@Sidartha-CT

@Sidartha-CT Sidartha-CT commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

PR: lf_ftp.py — Robot/band-steering resilience to unresponsive devices + cleaner missing-CX logging

Commits: bdeafed58982f268

What's handled

All devices go unresponsive during monitoring (bdeafed5)

Instead of hanging indefinitely, the monitor loop now retries for up to 40s (wait_for_any_cx_recovery), then gives up gracefully and stops the robot/band-steering run early, using whatever data was already collected.

Band-steering's coordinate loop and the plain robot coordinate/rotation loop both check all_devices_stopped and break out cleanly instead of continuing to move the robot.


Grace-period timing differs by test mode (3439f307)

Band-steering calls monitor_for_runtime_csv() repeatedly within one continuous session, so monitoring_start_time is set once.

Plain robot coordinate/rotation tests call it once per coordinate with CXs freshly restarted each time, so the grace period now resets on every call instead of only the first.


WebUI left in a stale "in progress" nav state (5ccf7acb)

When the 40s recovery wait times out on a robot test, update_nav_data_for_all_cxs_stopped() is now called so the WebUI shows the run as completed rather than stuck on a step that will never be reached.


Crash when band steering ends after all devices stop (2c1d6daa)

monitor_for_runtime_csv() now:

  • bails out immediately if band steering already gave up on a prior tick (before self.data gets reset, so the last good snapshot survives for stop() to read),
  • seeds start_time/end_time defensively in case traffic_duration <= 0,
  • uses the band-steering session's actual start time instead of each tick's timestamp,
  • records the real give-up time as end_time instead of the never-reached planned end.

Missing-CX warning dumped the entire nested API response (8982f268)

Now logs just the CX names present in the response, making the log readable instead of printing the full stats dict for every connection.


Net effect

Robot/band-steering FTP tests degrade gracefully (stop early, keep partial data, update WebUI state, no crash) when devices disconnect mid-test, and diagnostic logging for missing CXs is now scannable at a glance.

test logs : 

1785742108.137854 INFO     Test is about to start lf_ftp.py 222
1785742108.314358 INFO     Test is Initialized lf_ftp.py 351
1785742108.516930 INFO     Connected (version 2.0, client OpenSSH_8.8) transport.py 1938
1785742109.049552 INFO     Authentication (password) successful! transport.py 1938
1785742110.131999 INFO     ['True\n'] lf_ftp.py 980
1785742121.213474 INFO     File creation done 5MB lf_ftp.py 988
1785742124.285852 INFO     Upstream port IP 172.16.0.1 lf_ftp.py 611
1785742124.465448 WARNING  1.1.98ccc885 is in phantom state. Please make sure debugging is enabled in developer settings. DeviceConfig.py 393
1785742124.465659 WARNING  1.1.BYEQ6LSCDQOVO7GE is in phantom state. Please make sure debugging is enabled in developer settings. DeviceConfig.py 393
1785742125.379221 INFO     The laptop on port 1.105 is in phantom state. DeviceConfig.py 868
1785742125.823332 INFO     The laptop on port 1.128 is in phantom state. DeviceConfig.py 868
1785742125.823491 INFO     The laptop on port 1.151 is in phantom state. DeviceConfig.py 868
1785742126.444824 ERROR    Malformed response. Response does not have interfaces data. Setting the station name to default i.e., wlan0 DeviceConfig.py 953
1785742126.444959 INFO     The laptop on port 1.192 has no wiphy port available. DeviceConfig.py 875
1785742126.445001 INFO     The laptop on port 1.255 is in phantom state. DeviceConfig.py 868
1.10  1.10 Lin hp80-HP-ProBook-445R-G6
1.11  1.11 Lin user-ThinkPad-T450
1.15  1.15 android googlepixel
1.16  1.16 android Samsung_F41
1.18  1.18 android Oppo2375
1.22  1.22 android Pixel4a_1
1.25  1.25 android samsung
1.26  1.26 Lin lin-01
1.109  1.109 Lin user-HP-ProBook-445R-G6
1.113  1.113 Win DESKTOP-F0HLI17
1.153  1.153 Mac lanforges-Air.ctinb.candelatech
1785742127.464884 INFO     AVAILABLE DEVICES TO RUN TEST: ['1.10 Lin hp80-HP-ProBook-445R-G6', '1.11 Lin user-ThinkPad-T450', '1.15 android googlepixel', '1.16 android Samsung_F41', '1.18 android Oppo2375', '1.22 android Pixel4a_1', '1.25 android samsung', '1.26 Lin lin-01', '1.109 Lin user-HP-ProBook-445R-G6', '1.113 Win DESKTOP-F0HLI17', '1.153 Mac lanforges-Air.ctinb.candelatech'] lf_ftp.py 498
1785742127.465059 INFO     ['1.10 Lin hp80-HP-ProBook-445R-G6', '1.11 Lin user-ThinkPad-T450', '1.15 android googlepixel', '1.16 android Samsung_F41', '1.18 android Oppo2375', '1.22 android Pixel4a_1', '1.25 android samsung', '1.26 Lin lin-01', '1.109 Lin user-HP-ProBook-445R-G6', '1.113 Win DESKTOP-F0HLI17', '1.153 Mac lanforges-Air.ctinb.candelatech'] lf_ftp.py 499
1785742127.465134 INFO     Test is initiated on devices: ['1.26', '1.109'] lf_ftp.py 522
1785742127.465198 INFO     device got from webui are: 1.26,1.109 lf_ftp.py 539
1785742127.465271 INFO     INPUT DEVICES LIST ['1.26.wlan0', '1.109.wlan0'] lf_ftp.py 557
REAL CLIENT LIST ['1.26 Lin lin-01', '1.109 Lin user-HP-ProBook-445R-G6']
1785742127.465342 INFO     MAC ID LIST ['ac:ed:5c:36:a6:1e', 'd8:12:65:0d:74:59'] lf_ftp.py 576
1785742127.465541 INFO     Cleaning up cxs and endpoints http_profile.py 77
1785742127.465658 INFO     Cleaning up stations station_profile.py 474
1785742127.465706 INFO     INFO: StationProfile cleanup, list is empty station_profile.py 480
1785742128.466100 INFO     precleanup done lf_ftp.py 677
1785742129.264034 INFO     Create HTTP CXs...py-json.http_profile http_profile.py 119
1785742129.510039 INFO     ###### url:dl ftp://lanforge:lanforge@172.16.0.1/ftp_test.txt /dev/null http_profile.py 184
1785742130.430204 INFO     ###### url:dl ftp://lanforge:lanforge@172.16.0.1/ftp_test.txt /dev/null http_profile.py 184
1785742132.777391 INFO     Test Build done lf_ftp.py 842
1785742132.777791 ERROR    passes: zero test results, Chamberview tests will show zero test results so not an error lfcli_base.py 530
1785742133.506250 INFO     cross connections found for all devices lf_ftp.py 3197
1785742133.506574 INFO     Test started on the devices : ['1.26.wlan0', '1.109.wlan0'] lf_ftp.py 4075
1785742133.506699 INFO     Traffic started running at 2026-08-03 12:58:53.506653 lf_ftp.py 4078
1785742133.507301 INFO     Starting CXs... http_profile.py 59
1785742134.020154 INFO     Test Started lf_ftp.py 865
1785742162.759352 WARNING  CX 'wlan0_ftp109_l4' status changed: Run -> Stopped lf_ftp.py 1120
1785742168.832036 INFO     CX 'wlan0_ftp109_l4' recovered: Stopped -> Run lf_ftp.py 1123
1785742180.096454 WARNING  CX 'wlan0_ftp109_l4' is missing from the monitoring data, the device may have disconnected or its connection was not created. Continuing the test with the remaining devices.
URL          : layer4/wlan0_ftp26_l4,wlan0_ftp109_l4/list?fields=uc-avg,uc-max,uc-min,total-urls,rx rate (1m),bytes-rd,total-err,status
Response CXs : ['wlan0_ftp26_l4'] lf_ftp.py 1457
1785742196.479403 ERROR    All l4 data not found lf_ftp.py 1406
1785742196.895932 INFO     Monitoring complete lf_ftp.py 1714
1785742196.897068 INFO     Stopping CXs... http_profile.py 68
1785742197.100678 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/set_cx_state
  X-Error-00: unable to execute set_cx_state rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/set_cx_state
 = = = = = = = = = = = = = = = = LFRequest.py 396
1785742197.299331 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/set_cx_state
  X-Error-00: unable to execute set_cx_state rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/set_cx_state
 = = = = = = = = = = = = = = = = LFRequest.py 396
Traffic stopped running
1785742197.301829 INFO     Cleaning up cxs and endpoints http_profile.py 77
1785742197.461776 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/rm_cx
  X-Error-00: unable to execute rm_cx rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/rm_cx
 = = = = = = = = = = = = = = = = LFRequest.py 396
1785742197.708680 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/rm_endp
  X-Error-00: unable to execute rm_endp rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/rm_endp
 = = = = = = = = = = = = = = = = LFRequest.py 396
1785742197.914173 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/rm_cx
  X-Error-00: unable to execute rm_cx rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/rm_cx
 = = = = = = = = = = = = = = = = LFRequest.py 396
1785742198.123818 ERROR    
 = = LANforge Error Messages = =
 = = URL: http://192.168.245.117:8080/cli-json/rm_endp
  X-Error-00: unable to execute rm_endp rv: 22 (22  Invalid argument)
  X-Error-01: URL: /cli-json/rm_endp
 = = = = = = = = = = = = = = = = LFRequest.py 396
1785742198.124161 INFO     Cleaning up stations station_profile.py 474
1785742198.124259 INFO     INFO: StationProfile cleanup, list is empty station_profile.py 480
1785742198.124361 INFO     Test ended at 2026-08-03 12:59:58.124329 lf_ftp.py 4107
1785742198.126399 INFO     path set:  lf_report.py 85
1785742198.126628 INFO     path_date_time 2026-08-03-12-59-58_ftp_test lf_report.py 239
1785742198.127806 INFO     report path : 2026-08-03-12-59-58_ftp_test lf_report.py 248
1785742198.130478 INFO     path set: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf lf_report.py 85
1785742198.130757 INFO     path_date_time /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test lf_report.py 239
1785742198.131922 INFO     report path : /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test lf_report.py 248
1785742198.135152 ERROR    failed to create all layer 4 csv lf_ftp.py 2520
                        Client names  Download
0                    1.26 Lin lin-01         0
1  1.109 Lin user-HP-ProBook-445R-G6         4
graph name Total-url_ftp.png
1785742198.509482 INFO     graph_src_file: Total-url_ftp.png lf_report.py 212
1785742198.509673 INFO     graph_dst_file: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/Total-url_ftp.png lf_report.py 213
1785742198.509800 INFO     csv_src_file: Total-url_ftp.csv lf_report.py 219
1785742198.509838 INFO     csv_dst_file: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/Total-url_ftp.csv lf_report.py 220
                        Client names  Download
0                    1.26 Lin lin-01      0.00
1  1.109 Lin user-HP-ProBook-445R-G6    197.25
graph name ucg-avg_ftp.png
1785742198.769928 INFO     graph_src_file: ucg-avg_ftp.png lf_report.py 212
1785742198.770284 INFO     graph_dst_file: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/ucg-avg_ftp.png lf_report.py 213
1785742198.770510 INFO     csv_src_file: ucg-avg_ftp.csv lf_report.py 219
1785742198.770596 INFO     csv_dst_file: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/ucg-avg_ftp.csv lf_report.py 220
1785742209.778145 INFO     write_output_html: /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/ftp_test.html lf_report.py 368
1785742209.778360 INFO     returned file /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/ftp_test.html lf_ftp.py 2810
1785742209.778409 INFO     /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/ftp_test.html lf_ftp.py 2811
1785742210.339556 INFO     Monitoring Duration: 1m 2s lf_ftp.py 2813
1785742210.339726 INFO     output file /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/_2026-08-03-13-00-10-test_l4_ftp.csv lf_report.py 350
1785742210.339787 INFO     csv output file : /home/Sidartha/myhome/local/interop-webGUI/results/safsaf/2026-08-03-12-59-58_ftp_test/_2026-08-03-13-00-10-test_l4_ftp.csv lf_ftp.py 2999

…sive

Signed-off-by: Sidartha-CT <neelapu.sidartha@candelatech.com>
Comment thread py-scripts/lf_ftp.py Outdated
Comment thread py-scripts/lf_ftp.py Outdated
Comment thread py-scripts/lf_ftp.py Outdated

@haricharan-candela haricharan-candela left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, suggested nits and update the description with summary of changes made

@Sidartha-CT Sidartha-CT changed the title Enhance logging for unresponsive device CXs lf_ftp.py — Robot/band-steering resilience to unresponsive devices + cleaner missing-CX logging Aug 3, 2026
…nate

Signed-off-by: Sidartha-CT <neelapu.sidartha@candelatech.com>
…esponding

Signed-off-by: Sidartha-CT <neelapu.sidartha@candelatech.com>
…esponding

Signed-off-by: Sidartha-CT <neelapu.sidartha@candelatech.com>
…ssing

Verified CLI  : python3 lf_ftp.py --ssid "SSID" --passwd sharedsecret  --file_sizes 5MB --mgr "192.168.245.117" --traffic_duration 1M  --security wpa2 --directions Download --clients_type Real --bands 5G --upstream_port eth2 --twog_radio wiphy0 --device_list 1.26,1.109

Signed-off-by: Sidartha-CT <neelapu.sidartha@candelatech.com>
@goyalsaurabh06
goyalsaurabh06 merged commit 9263146 into greearb:master Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants