Threadstats: 413 - Request Entity Too Large

[trygvebw]

When threadstats receives too many metrics within a flush_interval window, it receives a 413 error from the Datadog server upon submitting the metrics, resulting in the following exception being thrown:

2019-10-12T21:36:00+0200 ERROR:datadog.threadstats:Error flushing metrics and events
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/datadog/threadstats/base.py", line 312, in flush
self.reporter.flush_metrics(metrics)
File "/usr/local/lib/python3.7/site-packages/datadog/threadstats/reporters.py", line 21, in flush_metrics
api.Metric.send(metrics)
File "/usr/local/lib/python3.7/site-packages/datadog/api/metrics.py", line 91, in send
return super(Metric, cls).send(attach_host_name=attach_host_name, **metrics_dict)
File "/usr/local/lib/python3.7/site-packages/datadog/api/resources.py", line 78, in send
attach_host_name=attach_host_name)
File "/usr/local/lib/python3.7/site-packages/datadog/api/api_client.py", line 133, in submit
proxies=_proxies, verify=_cacert
File "/usr/local/lib/python3.7/site-packages/datadog/api/http_client.py", line 99, in request
raise _remove_context(HTTPError(e.response.status_code, result.reason))
datadog.api.exceptions.HTTPError: Datadog returned a bad HTTP response code: 413 - Request Entity Too Large. Please try again later. If the problem persists, please contact support@datadoghq.com

Is this behaviour new? This has happened several times in the past two weeks or so, but I cannot find any occurences of HTTP 413 errors in my logs from before that.

Whether or not this is new, is there a better way of handling large variances in the volume of metrics than turning flush_interval way down and hoping that the metric load won't get too high again? Ideally the flush_interval would have an "auto" mode that let it be automatically adjusted when the metric load increases and decreases, but as far as I can tell this is not available.

[zippolyte]

Hello @trygvebw
I've opened a PR that add the ability to send compressed payload (#466) , which should help with your issue. Would you be willing to give it a try and see if it does help ?

You should be able to instantiate your threadstats object with the new param compress_payload set to True by checking out the branch in the attached PR.

[trygvebw]

Hi @zippolyte, that PR seems promising. I don't have time to check if it solves the problem this week (the issue was fixed for the time being by lowering the flush_interval), unfortunately, but I might have time later.

[trygvebw]

Another issue: It would be very useful to be able to handle any exceptions thrown by threadstats, including HTTPError but also other exceptions, instead of threadstats just silently swallowing most exceptions. Currently this doesn't seem to be possible; correct me if I'm wrong. Ideally I'd be able to provide threadstats with an error handler function that is invoked every time threadstats throws an exception.

[dabcoder]

Closing this issue as this was addressed in #466 (now merged).

@trygvebw mind opening a separate issue for the above? Thanks!