IncompleteRead exception when running supervisorctl status

[r14c]

I see the following error when I run supervisorctl status

error: <class 'httplib.IncompleteRead'>, IncompleteRead(3973 bytes read, 2081 more expected): file: /usr/lib/python2.7/httplib.py line: 666

This is the corresponding log entry:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/supervisor/xmlrpc.py", line 75, in more
    return self.getresponse(body)
  File "/usr/lib/python2.7/dist-packages/supervisor/xmlrpc.py", line 142, in getresponse
    self.request.log
  File "/usr/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py", line 163, in push_with_producer
    self.initiate_send()
  File "/usr/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py", line 213, in initiate_send
    self.refill_buffer()
  File "/usr/lib/python2.7/dist-packages/supervisor/http.py", line 382, in refill_buffer
    data = p.more()
  File "/usr/lib/python2.7/dist-packages/supervisor/medusa/producers.py", line 179, in more
    data = self.producer.more()
  File "/usr/lib/python2.7/dist-packages/supervisor/medusa/producers.py", line 206, in more
    self.function (self.bytes)
  File "/usr/lib/python2.7/dist-packages/supervisor/http.py", line 256, in log
    bytes
TypeError: %d format: a number is required, not str
error: uncaptured python exception, closing channel <supervisor.http.deferring_http_channel connected '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' at 0x7ffffd90dea8 channel#: 6 requests:1> (<type 'exceptions.TypeError'>:%d format: a number is required, not str [/usr/lib/python2.7/dist-packages/supervisor/supervisord.py|runforever|246] [/usr/lib/python2.7/dist-packages/supervisor/medusa/asyncore_25.py|handle_write_event|397] [/usr/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py|handle_write|152] [/usr/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py|initiate_send|213] [/usr/lib/python2.7/dist-packages/supervisor/http.py|refill_buffer|382] [/usr/lib/python2.7/dist-packages/supervisor/http.py|more|93] [/usr/lib/python2.7/dist-packages/supervisor/http.py|more|125] [/usr/lib/python2.7/dist-packages/supervisor/http.py|log|256])

I am running supervisord in a docker container as PID 1 with an image based on ubuntu:14.04

[mnaberez]

What version of Supervisor are you using?

[r14c]

$ supervisord --version
3.0b2
$ python --version
Python 2.7.6

[mnaberez]

Are you able to send some instructions that would allow us to reproduce this? I tried that version but did not see this error occur.

[r14c]

This image is throwing exceptions pretty reliably.

>>> ETCD_DISCOVERY=$(curl -s "https://discovery.etcd.io/new?size=2")
>>> docker run -d -e ETCD_DISCOVERY=$ETCD_DISCOVERY zulumarketing/app-server
$CONTAINER_1_ID
>>> docker run -d -e ETCD_DISCOVERY=$ETCD_DISCOVERY zulumarketing/app-server
$CONTAINER_2_ID

This should allow all of the daemons to start correctly. Running docker exec -it $CONTAINER_x_ID /bin/bash will allow you to poke around inside the running container.

At this point running supervisorctl status should reproduce the error.

[mnaberez]

I tried your container with this result:

$ docker exec -it 57f7b1e4089b /bin/bash
root@57f7b1e4089b:/# for i in {1..5000}; do supervisorctl status >> output.txt; done 
root@57f7b1e4089b:/# head output.txt
etcd                             RUNNING   pid 9, uptime 0:11:58
etcd-proxy                       RUNNING   pid 10, uptime 0:11:58
nginx                            RUNNING   pid 7, uptime 0:11:58
nginx-log-forwarder              RUNNING   pid 26, uptime 0:11:47
etcd                             RUNNING   pid 9, uptime 0:11:58
etcd-proxy                       RUNNING   pid 10, uptime 0:11:58
nginx                            RUNNING   pid 7, uptime 0:11:58
nginx-log-forwarder              RUNNING   pid 26, uptime 0:11:47
etcd                             RUNNING   pid 9, uptime 0:11:58
etcd-proxy                       RUNNING   pid 10, uptime 0:11:58
root@57f7b1e4089b:/# tail output.txt
nginx                            RUNNING   pid 7, uptime 0:18:09
nginx-log-forwarder              RUNNING   pid 26, uptime 0:17:58
etcd                             RUNNING   pid 9, uptime 0:18:09
etcd-proxy                       RUNNING   pid 10, uptime 0:18:09
nginx                            RUNNING   pid 7, uptime 0:18:09
nginx-log-forwarder              RUNNING   pid 26, uptime 0:17:58
etcd                             RUNNING   pid 9, uptime 0:18:09
etcd-proxy                       RUNNING   pid 10, uptime 0:18:09
nginx                            RUNNING   pid 7, uptime 0:18:09
nginx-log-forwarder              RUNNING   pid 26, uptime 0:17:58
root@57f7b1e4089b:/# grep -v RUNNING output.txt 
root@57f7b1e4089b:/# grep -i error output.txt 
root@57f7b1e4089b:/#

I couldn't find the IncompleteRead in the output, and didn't see exceptions from supervisord in docker logs. Is there anything else I should do to reproduce it?

[mnaberez]

Closing due to inactivity. I wasn't able to reproduce this. If you are still experiencing this issue and can help us reproduce it, please let us know and we can reopen.

[TRManderson]

Having the exact same problem.

error: <class 'httplib.IncompleteRead'>, IncompleteRead(3973 bytes read, 1066 more expected): file: /usr/lib/python2.7/httplib.py line: 666

$ supervisord --version
3.2.0
$ python --version
Python 2.7.6

[TRManderson]

The traceback is effectively the same

2016-02-10 21:47:04,362 CRIT Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/supervisor/xmlrpc.py", line 401, in continue_request
    request.done()
  File "/usr/local/lib/python2.7/dist-packages/supervisor/http.py", line 238, in done
    self.channel.push_with_producer(outgoing_producer)
  File "/usr/local/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py", line 163, in push_with_producer
    self.initiate_send()
  File "/usr/local/lib/python2.7/dist-packages/supervisor/medusa/asynchat_25.py", line 213, in initiate_send
    self.refill_buffer()
  File "/usr/local/lib/python2.7/dist-packages/supervisor/http.py", line 387, in refill_buffer
    data = p.more()
  File "/usr/local/lib/python2.7/dist-packages/supervisor/http.py", line 98, in more
    data = self.producer.more()
  File "/usr/local/lib/python2.7/dist-packages/supervisor/http.py", line 130, in more
    self.function (self.bytes)
  File "/usr/local/lib/python2.7/dist-packages/supervisor/http.py", line 261, in log
    bytes
TypeError: %d format: a number is required, not str