Skip to content

Commit

Permalink
update query.py: Errors during JSON serialization are catched.
Browse files Browse the repository at this point in the history
If a request which is supposed to be non-html (JSON) returns an empty response or a html response,
it serialization leads to an error. This error is catched with a try / except.

This fixes issue taspinar/twitterscraper#93
  • Loading branch information
meticulousfan committed Mar 21, 2018
1 parent f1b2c9d commit 21d857c
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions twitterscraper/query.py
Expand Up @@ -33,10 +33,14 @@ def query_single_page(url, html_response=True, retry=10):
try:
response = requests.get(url, headers=headers)
if html_response:
html = response.text
html = response.text or ''
else:
json_resp = response.json()
html = json_resp['items_html']
html = ''
try:
json_resp = json.loads(response.text)
html = json_resp['items_html'] or ''
except ValueError as e:
logging.exception('Failed to parse JSON "{}" while requesting "{}"'.format(e, url))

tweets = list(Tweet.from_html(html))

Expand Down

0 comments on commit 21d857c

Please sign in to comment.