Agents show EXITED in supervisorctl

[nancyc12]

Issue

Some agents in TEL agent hosts were found to be EXITED and could not recover themselves.

ubuntu@juju-462031-0:~$ sudo supervisorctl status | grep EXITED
hp-eliteone800-g6-all-in-one-pc-c28167                        EXITED    Jan 06 11:01 AM
rpi3bp009                                                     EXITED    Jan 06 11:12 AM
rpi4b1g001                                                    EXITED    Jan 06 11:09 AM
rpi4b2g001                                                    EXITED    Jan 06 02:30 PM
rpi4b2g002                                                    EXITED    Jan 06 11:06 AM
rpi4b4g001                                                    EXITED    Jan 06 02:29 PM
rpi4b4g002                                                    EXITED    Jan 06 11:04 AM
rpi4b8g001                                                    EXITED    Jan 06 02:22 PM
rpi4b8g002                                                    EXITED    Jan 06 11:08 AM

ubuntu@juju-47ff0e-1:~$ sudo supervisorctl status | grep EXITED
rpi400001                                                      EXITED    Jan 06 02:32 PM

ubuntu@juju-ccf3c0-0:~$ sudo supervisorctl status | grep EXITED
cm3p001                                                        EXITED    Jan 06 11:21 AM
dell-precision7550-c27818                                      EXITED    Jan 09 08:29 AM
rpi400002                                                      EXITED    Jan 06 11:09 AM

Log

ubuntu@juju-462031-0:~$ sudo supervisorctl tail -f rpi4b8g002
==> Press Ctrl-C to exit <==
68c75fc38c/events (Caused by ResponseError('too many 503 error responses'))
[25-01-06 11:08:04]   ERROR: (cmd.py:32)| 'NoneType' object has no attribute 'status_code'
Traceback (most recent call last):
  File "/srv/testflinger-venv/lib/python3.10/site-packages/testflinger_agent/cmd.py", line 27, in main
    start_agent()
  File "/srv/testflinger-venv/lib/python3.10/site-packages/testflinger_agent/__init__.py", line 139, in start_agent
    agent.process_jobs()
  File "/srv/testflinger-venv/lib/python3.10/site-packages/testflinger_agent/agent.py", line 328, in process_jobs
    event_emitter.emit_event(TestEvent.CLEANUP_START)
  File "/srv/testflinger-venv/lib/python3.10/site-packages/testflinger_agent/event_emitter.py", line 55, in emit_event
    self.client.post_status_update(
  File "/srv/testflinger-venv/lib/python3.10/site-packages/testflinger_agent/client.py", line 448, in post_status_update
    % (status_update_uri, job_request.status_code)
AttributeError: 'NoneType' object has no attribute 'status_code

Workaround

Manually start each agent.

sudo supervisorctl start rpi4b8g002

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CERTTF-474.

This message was autogenerated

[gajeshbhat]

This is an interesting one! I remember implementing a cron job that used a bash script to track the supervisorctl status of agents to do manual restarts when they crashed. The issue was caused by an AttributeError when the requests retry mechanism returned None after exhausting retries during server connectivity issues (like repeated 503 errors). The agent code was checking if not job_request: but then trying to access job_request.status_code, which crashed when job_request was None.

I've put up a pull request to address this: #787

The fix handles the None case explicitly in 5 client methods, so agents will log errors gracefully instead of crashing and showing EXITED in supervisorctl. CC: @nancyc12 @pedro-avalos Please let me know if you have any questions. I have explained the underlying cause in more detail in the PR description. Cheers.