New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rare quads-daemon traceback #188

Closed
sadsfae opened this Issue Sep 13, 2018 · 1 comment

Comments

Projects
None yet
1 participant
@sadsfae
Member

sadsfae commented Sep 13, 2018

Very rarely (but now twice over the past year) we see this pop up, restarting the daemon "fixes" it. I'm not sure what's causing it but it seems load / connection related.

Traceback (most recent call last):
  File "/opt/quads/bin/quads-cli", line 504, in <module>
    main(sys.argv[1:])
  File "/opt/quads/bin/quads-cli", line 493, in main
    r = requests.post(url, data, headers=headers, timeout=120)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line
475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line
585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line
453, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.',
BadStatusLine("''",))

@sadsfae sadsfae added bug quads-daemon and removed bug labels Sep 13, 2018

@sadsfae

This comment has been minimized.

Show comment
Hide comment
@sadsfae

sadsfae Sep 13, 2018

Member

After some investigation there was nothing wrong with the code nor anything evident in the tcpdumps taken.

This seemed to be system related, we had prior XFS corruption on the hypervisor as well as CPU soft lockup on the VM which was most likely contributing to the timeouts experienced (and subsequent tracebacks). I am closing this now but we can re-open.

Member

sadsfae commented Sep 13, 2018

After some investigation there was nothing wrong with the code nor anything evident in the tcpdumps taken.

This seemed to be system related, we had prior XFS corruption on the hypervisor as well as CPU soft lockup on the VM which was most likely contributing to the timeouts experienced (and subsequent tracebacks). I am closing this now but we can re-open.

@sadsfae sadsfae closed this Sep 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment