ERRO uncaptured python exception, closing channel <supervisor.http.supervisor_af_inet_http_server> <class 'RuntimeError'>:dictionary changed size during iteration

[trickmeyer]

supervisord 4.0.0.dev0
python 3.66

since this relates just to inet_http_server, the config is minimal:

[inet_http_server]
port={somelocalserver}:9003

full trace - issue in the dispatcher.handle_read_event() section of supervisord's select loop.

2018-12-05 09:27:11,739 ERRO ch1-prod-02:uncaptured python exception, closing channel <supervisor.http.supervisor_af_inet_http_server listening ch1-prod-02:9003 at 0x7f87a9e0fda0> (<class 'RuntimeError'>:dictionary changed size during iteration [supervisord.py|runforever|218] [python3.6/site-packages/supervisor/medusa/asyncore_25.py|handle_read_event|385] [python3.6/site-packages/supervisor/medusa/http_server.py|handle_accept|745] [python3.6/site-packages/supervisor/medusa/http_server.py|__init__|484] [python3.6/site-packages/supervisor/medusa/http_server.py|check_maintenance|499] [python3.6/site-packages/supervisor/medusa/http_server.py|maintenance|502] [python3.6/site-packages/supervisor/medusa/http_server.py|kill_zombies|509])

this supervisord instance had been running for about a month. below are the logs. last activity at 9:11am is just fine, then at 9:27am a user requests restart of a process via the web ui. at that point we get the error message, but the program restarts just fine. everything is now running excapt the inet_http_server.

2018-12-05 09:11:32,926 INFO success: {process name} entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2018-12-05 09:27:11,739 ERRO ch1-prod-02:uncaptured python exception, closing channel <supervisor.http.supervisor_af_inet_http_server listening ch1-prod-02:9003 at 0x7f87a9e0fda0> (<class 'RuntimeError'>:dictionary changed size during iteration [python3.6/site-packages/supervisor/supervisord.py|runforever|218] [python3.6/site-packages/supervisor/medusa/asyncore_25.py|handle_read_event|385] [python3.6/site-packages/supervisor/medusa/http_server.py|handle_accept|745] [python3.6/site-packages/supervisor/medusa/http_server.py|__init__|484] [python3.6/site-packages/supervisor/medusa/http_server.py|check_maintenance|499] [python3.6/site-packages/supervisor/medusa/http_server.py|maintenance|502] [python3.6/site-packages/supervisor/medusa/http_server.py|kill_zombies|509])
2018-12-05 09:27:15,668 INFO waiting for {process name} to stop
2018-12-05 09:27:15,675 INFO stopped: {process name} (terminated by SIGTERM)
2018-12-05 09:27:16,678 INFO spawned: '{process name}' with pid 26799
2018-12-05 09:27:18,682 INFO success: {process name} entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)

[trickmeyer]

@mnaberez i'm happy to help on this, but not sure how to try to reliably recreate and test - any ideas?

[mnaberez]

Thanks, any help would be appreciated. Since the log shows the error occurred around the time a process was stopped, I'd probably try to reproduce it by doing something like running /bin/cat as a subprocess under supervisord and repeatedly trying supervisorctl stop cat or similar to see if that can make the error occur. Beyond that, I'm not sure. This is the first time I have seen this error.