Large traces hit Request Entity Too Large errors

[rtpg]

On some of our larger APM traces (background tasks) we've been hitting a lot of the following kind of error:

failed_to_send traces to Agent: HTTP error status 413, reason Request Entity Too Large, message Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 07 May 2018 04:40:12 GMT
Content-Length: 18

I believe that this is a similar issue to this issue over on the ruby library issues.

The ruby library talks about partial flushing, but I didn't see anything in the python library that would allow for this.

[palazzem]

Correct! Actually we're implementing this behavior on other libraries (Go for example) and Python will be next. I'll keep you updated once we'll start working on it!

[alandotcom]

Hey @palazzem any updates? Just started seeing this issue

[davvilla]

Hello, this just happened to us and brought down our stack as machines started failing one by one. Any updates here?

[brettlangdon]

Sorry for the delay in updating everyone here.

We are looking into this issue to see if we can stop any exceptions from bubbling up from our code.

There is a long term solution for supporting partial traces flushes so we can break up and batch send large traces, however we are waiting on changes to the trace agent before we can move forward with that.

We will keep everyone here posted as we make progress towards an incremental improvement.

[brettlangdon]

We have released a partial/experimental fix for this issue in 0.16.0.

Now if you set an environment variable DD_TRACER_PARTIAL_FLUSH_ENABLED=true we will try to break up traces with more than 500 spans and send them in chunks.

While this solution works and will resolve this issue, it is not fully supported by our entire stack which means that traces reported this way may have their stats calculated incorrectly (since we'll report 2-3 traces instead of 1) and there is a chance that some parts may get sampled out (you will only see part of the entire trace).

I am going to leave this issue open for now until there is an official fix for this issue.

[luvpreetsingh]

Using Django 2.0.6 to send traces to datadog (ddtrace==0.16.0).

Still getting this issue. DD_TRACER_PARTIAL_FLUSH_ENABLED option is not available in the datadog.yaml file and not in django-datadog-docs.

failed_to_send traces to Agent: HTTP error status 413, reason Request Entity Too Large, message Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 30 Nov 2018 03:50:40 GMT
Content-Length: 18

[brettlangdon]

@luvpreetsingh DD_TRACER_PARTIAL_FLUSH_ENABLED=true is meant to be set as an environment variable, not as a configuration value in datadog.yaml or in the django configuration.

[galonsky]

@brettlangdon has there been an official fix released for this yet? and if not, is the partial flush feature still "experimental" / not fully supported?

[brettlangdon]

@galonsky there has not, but #801 is intended to resolve this issue, hoping to get it out with the next release... maybe the one after that if something comes up.

[felippemr]

Hey folks, any news on this?

[galonsky]

hi @brettlangdon any updates?

[brettlangdon]

@galonsky we have #941 ready to merge, we just want to do some testing on it internally before we ship it.

[galonsky]

@brettlangdon @jd I see this fix was merged. when will the new version of the package be published?

[jd]

@galonsky It's on its way out. Should be a couple of days now I think.

[mrmachine]

@jd @brettlangdon is this fixed by #941 which is included in the 0.27.0 release? Is the DD_TRACER_PARTIAL_FLUSH_ENABLED variable (or any other) still required to activate the new behaviour?

[jd]

@mrmachine Yes, #941 should fix this with 0.27.0 and you don't need to do anything. Just enjoy it! ;)

[brettlangdon]

Closing this card since version 0.27.0 should address this issue for most customers.

Please feel free to re-open if the latest version of the tracer does not fix this issue for you.