Backend Error

[neonknight]

I'm running gglsbl under gglsbl-rest, but the traceback indicates an issue on gglsbl side. If my assumption is wrong I'll open this bug report again for gglsbl-rest.

At random I see the following error:

ERROR in app: exception handling [http://example.com/index.php]
Traceback (most recent call last):
File "/root/gglsbl-rest/app.py", line 28, in app_lookup
sbl = SafeBrowsingList(gsb_api_key, active['name'], True)
File "build/bdist.linux-x86_64/egg/gglsbl/client.py", line 31, in __init__
self.api_client = SafeBrowsingApiClient(api_key, discard_fair_use_policy=discard_fair_use_policy)
File "build/bdist.linux-x86_64/egg/gglsbl/protocol.py", line 68, in __init__
self.service = build('safebrowsing', 'v4', developerKey=developer_key)
File "build/bdist.linux-x86_64/egg/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/googleapiclient/discovery.py", line 237, in build
raise e
HttpError: <HttpError 503 when requesting https://www.googleapis.com/discovery/v1/apis/safebrowsing/v4/rest returned "Backend Error">

Environment:

cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

python --version
Python 2.7.5

cat /usr/lib/python2.7/site-packages/easy-install.pth
<snip>
./gglsbl-v1.4.5-py2.7.egg
./google_api_python_client-1.6.3-py2.7.egg
./oauth2client-4.1.2-py2.7.egg
</snip>

Is this something that can be fixed or is it Google's fault?

[afilipovich]

According to docs, the issue appears to be on the remote end:
https://developers.google.com/analytics/devguides/reporting/core/v3/errors

Just tried to run similar query and it works okay, perhaps only your geo is affected.

[neonknight]

Thank you for your feedback. The error is indeed very random and not reliably reproducible.

As the issue appears on Google side it's probably best to handle the error in the final application. I therefore suggested a workaround to gglsbl-rest.

[afilipovich]

Thanks for confirmation.

[asieira]

Addressed this in gglsbl-rest (mlsecproject/gglsbl-rest#5). Thanks for bringing this up. 🙇

[afilipovich]

I'm concerned that in case of insufficient disk space lookup will be performed on incomplete cache DB and may miss blacklisted URLs. If the root cause was lack of disk space of course...

[asieira]

Can gglsbl check for that condition and generate a clear exception if and when it occurs?

[asieira]

Or is there a specific exception generated by the sqlite driver in that scenario that I could look for in gglsbl-rest?

[afilipovich]

Current exception is obscure, that's for sure, but even if this exception does not happen there is no guarantee that cache DB is complete. E.g. when there is enough disk space to fit half of cache this exception won't happen, but lookups will be missing bad URLs.

I am going to add some integrity check to prevent this. Thank you for the info.