Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle Rollbar API 50x errors, had a 502 Bad Gateway and didn't fail gracefully #223

Closed
GabLeRoux opened this issue Dec 12, 2017 · 1 comment
Closed

Handle Rollbar API 50x errors, had a 502 Bad Gateway and didn't fail gracefully #223

GabLeRoux opened this issue Dec 12, 2017 · 1 comment
Labels
bug
Milestone
v0.13.18

Comments

@GabLeRoux
Copy link
Contributor

GabLeRoux commented Dec 12, 2017 •
edited
Loading

Hi there, I just introduced Rollbar to our team recently and as soon as I had the rollbar setup on our QA server, rollbar failed to deliver a payload and the exception was raised to django's exception handling.

Rollbar returned a 502 error (maybe this is an isolated case).

Initial error

('Request to %s returned invalid JSON response', 'item/')
Django Version:	1.9.5
Exception Type:	ApiException
Exception Value:	
('Request to %s returned invalid JSON response', 'item/')
Exception Location:	<project_path>/lib/python3.4/site-packages/rollbar/__init__.py in _parse_response, line 1390
Python Version:	3.4.2
Traceback

/usr/lib/python3.4/json/decoder.py in raw_decode
            obj, end = self.scan_once(s, idx)
Local Vars

Variable	Value
s	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
self	
<json.decoder.JSONDecoder object at 0x7fac21074358>
idx	
0
During handling of the above exception (0), another exception occurred:
<project_path>/lib/python3.4/site-packages/rollbar/__init__.py in _parse_response
        json_data = json.loads(data)
Local Vars

Variable	Value
data	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
endpoint	
None
access_token	
'<snip>'
params	
'{"data": <snip>}'
resp	
<Response [502]>
path	
'item/'
/usr/lib/python3.4/json/__init__.py in loads
        return _default_decoder.decode(s)
Local Vars

Variable	Value
parse_constant	
None
parse_int	
None
object_hook	
None
encoding	
None
object_pairs_hook	
None
s	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
kw	
{}
cls	
None
parse_float	
None
/usr/lib/python3.4/json/decoder.py in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Local Vars

Variable	Value
s	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
self	
<json.decoder.JSONDecoder object at 0x7fac21074358>
_w	
<built-in method match of _sre.SRE_Pattern object at 0x7fac210e8378>
/usr/lib/python3.4/json/decoder.py in raw_decode
            raise ValueError(errmsg("Expecting value", s, err.value)) from None
Local Vars

Variable	Value
s	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
self	
<json.decoder.JSONDecoder object at 0x7fac21074358>
idx	
0
During handling of the above exception (Expecting value: line 1 column 1 (char 0)), another exception occurred:
<project_path>/lib/python3.4/site-packages/rollbar/__init__.py in _send_payload
        _post_api('item/', payload, access_token=access_token)
Local Vars

Variable	Value
payload	
'{"data": <snip>}'
access_token	
None
e	
ApiException('Request to %s returned invalid JSON response', 'item/')
<project_path>/lib/python3.4/site-packages/rollbar/__init__.py in _post_api
    return _parse_response(path, SETTINGS['access_token'], payload, resp)
Local Vars

Variable	Value
url	
'https://api.rollbar.com/api/1/item/'
access_token	
None
payload	
'{"data": <snip>}'
resp	
<Response [502]>
path	
'item/'
headers	
{'Content-Type': 'application/json'}
<project_path>/lib/python3.4/site-packages/rollbar/__init__.py in _parse_response
        raise ApiException('Request to %s returned invalid JSON response', path)
Local Vars

Variable	Value
data	
'<html>\r\n'
'<head><title>502 Bad Gateway</title></head>\r\n'
'<body bgcolor="white">\r\n'
'<center><h1>502 Bad Gateway</h1></center>\r\n'
'<hr><center>nginx</center>\r\n'
'</body>\r\n'
'</html>\r\n'
endpoint	
None
access_token	
'<snip>'
params	
'{"data"<snip>}'
resp	
<Response [502]>
path	
'item/'

I tried to reproduce this and actually wrote a test to demonstrate what actually happens in #224 :

test_502_failsafe (rollbar.test.test_rollbar.RollbarTest) ... WARNING:rollbar:Got unexpected status code from Rollbar api: 502
Response:
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

ERROR:rollbar:Could not decode Rollbar api response:
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
Traceback (most recent call last):
  File "<snip>/pyrollbar/rollbar/__init__.py", line 1387, in _parse_response
    json_data = json.loads(data)
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
ERROR:rollbar:Exception while posting item ApiException('Request to %s returned invalid JSON response', 'item/')
Traceback (most recent call last):
  File "<snip>/pyrollbar/rollbar/__init__.py", line 1387, in _parse_response
    json_data = json.loads(data)
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/opt/pyenv/versions/3.4.2/lib/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<snip>/pyrollbar/rollbar/__init__.py", line 1209, in _send_payload
    _post_api('item/', payload, access_token=access_token)
  File "<snip>/pyrollbar/rollbar/__init__.py", line 1257, in _post_api
    return _parse_response(path, SETTINGS['access_token'], payload, resp)
  File "<snip>/pyrollbar/rollbar/__init__.py", line 1390, in _parse_response
    raise ApiException('Request to %s returned invalid JSON response', path)
rollbar.ApiException: ('Request to %s returned invalid JSON response', 'item/')
WARNING:rollbar:Got unexpected status code from Rollbar api: 502
GabLeRoux added a commit to GabLeRoux/pyrollbar that referenced this issue Dec 12, 2017
@GabLeRoux
Adds failing test example for rollbar#223
57b3580
@GabLeRoux GabLeRoux changed the title Handle Rollbar API 500+ errors Handle Rollbar API 50x errors, had a 502 Bad Gateway and didn't fail gracefully Dec 12, 2017
@coryvirok
Copy link
Contributor

Hi @GabLeRoux,

Thanks for raising this. We are looking into it now.

@rokob rokob mentioned this issue Dec 13, 2017
Closed
@rivkahstandig3636 rivkahstandig3636 added this to the v0.13.18 milestone Dec 24, 2017
rokob pushed a commit that referenced this issue Feb 16, 2018
@GabLeRoux @rokob
Adds failing test example for #223 502 bad gateway (#224)
87ba05b 
* Adds failing test example for #223

* special case 502 errors
@rokob rokob closed this as completed Feb 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
v0.13.18
Development

No branches or pull requests
4 participants
@coryvirok @rokob @GabLeRoux @rivkahstandig3636