The first treq request after post_shutdown_to_master() fails

[guidow]

I have been trying to change the Agent.stop() method so that it will set itself to offline first and then stop running assignments (because otherwise the master would assign new tasks to the agent in the short time inbetween).

When I did this, though, the state update for the first task in the assignment failed with this error:

2015-03-26 12:20:29 ERROR    - pf.jobtypes.core - Error while posting state update for task, [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.RequestTransmissionFailed'>: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]                                                                            
], retrying                                                                                                                                                                            
2015-03-26 12:20:29 ERROR    - root            - Unexpected exception from twisted.web.client.FileBodyProducer.stopProducing
Traceback (most recent call last):                                                                                                                                                     
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/internet/endpoints.py", line 128, in connectionLost         
    return self._wrappedProtocol.connectionLost(reason)                                                                                                                                
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 911, in dispatcher                 
    return func(*args, **kwargs)                                                                                                                                                       
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 1564, in _connectionLost_TRANSMITTING                                                                                                                                                                                     
    self._currentRequest.stopWriting()                                                                                                                                                 
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 812, in stopWriting                
    _callAppFunction(self.bodyProducer.stopProducing)                                                                                                                                  
--- <exception caught here> ---                                                                                                                                                        
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 192, in _callAppFunction           
    function()                                                                                                                                                                         
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/client.py", line 1020, in stopProducing                 
    self._task.stop()                                                                                                                                                                  
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 464, in stop                        
    self._checkFinish()                                                                                                                                                                
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 474, in _checkFinish                
    raise self._completionState                                                                                                                                                        
twisted.internet.task.TaskStopped:                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                     
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 192, in _callAppFunction           
    function()                                                                                                                                                                         
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/web/client.py", line 1020, in stopProducing                 
    self._task.stop()                                                                                                                                                                  
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 464, in stop                        
    self._checkFinish()                                                                                                                                                                
  File "/var/tmp/virtualenv/pyfarm-agent-python2/lib/python2.7/site-packages/Twisted-15.0.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 474, in _checkFinish                
    raise self._completionState                                                                                                                                                        
TaskStopped

The updates for the other tasks went through with no problem. I have looked at the traffic between the agent and the master with wireshark, and there was no problem visible there, either, except that the request to change the state for the first task was just never done.

[opalmer]

Could you post the debug log output and the code change you made that seems to trigger this error? It almost seems like the reactor may be stopping before the update is complete but that's hard to tell for sure without more information.