Error after upgrade to 108.1

[jendib]

I upgraded from an old version to the latest 108.1 and I'm getting these warning every few minutes:

WARN  [dat.tra.com.wri.dda.DDAgentApi] (dd-trace-processor) msgp: too few bytes left to read object while sending 2 (size=1KB) traces to the DD agent. Total: 63244, Received: 63244, Sent: 56667, Failed: 6577. Status: 400 Bad Request (Will not log errors for 5 minutes)

[bantonsson]

Hi @jendib , thank you for reporting this. Which version of the Datadog agent are you running on the other side?

[jendib]

Thanks for your quick answer. My agent is running gcr.io/datadoghq/agent:7.35.2
Should I upgrade it as well?

[bantonsson]

It would be great if you could try with and upgraded agent and report back.

[jendib]

I still have the same issue after upgrading to the latest agent gcr.io/datadoghq/agent:7.38.2

[bantonsson]

Thanks for trying this out.
Could I ask some more questions about your setup? Is this being run in a kubernetes setup, and is it using a domain socket, or is it plain TCP? Also, which old version of dd-java-agent were you using before the upgrade?

One thing to try would be to enable a newer version of the Java Tracer to Datadog Agent protocol by setting the environment variable DD_TRACE_AGENT_V0_5_ENABLED=true or the property -Ddd.trace.agent.v0.5.enabled=true.

[jendib]

The agent is running in Kubernetes with the official Helm chart. I was using com.datadoghq:dd-trace-ot:0.98.1 before the upgrade.

I added DD_TRACE_AGENT_V0_5_ENABLED=true and it seems to fix the errors. Shouldn't this be enabled by default or have some kind of negotiation between the tracer and the agent to define the protocol version?

[bantonsson]

@jendib Thanks for confirming that the v0.5 version of the protocol works better. It will be enabled by default in a future release, and there is a negotiation to downgrade if the Datadog agent doesn't support it.

[jendib]

Oops I spoke too soon, the errors came back after a few minutes.

[bantonsson]

Hi @jendib I'm sorry to hear that. We are trying to figure out what might be causing this issue.

[jclark123123]

[dd.trace 2022-12-10 14:04:06:420 +0000] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - msgp: too few bytes left to read object while sending 3 (size=4KB) traces to the DD agent. Total: 4978, Received: 4978, Sent: 4975, Failed: 3. Status: 400 Bad Request (Will not log errors for 5 minutes)

Ditto, with https://github.com/DataDog/dd-trace-java/releases/download/v1.2.0/dd-java-agent-1.2.0.jar in our docker image and running -javaagent:/dd-java-agent.jar alongside our app.

From the message, it sounds like no error logs at all (even ones unrelated to this bytes reading problem) would be created - is that true?

[alejacadavid]

Is there any news or updates you can share regarding the status of this issue?

[maksymgendin]

We're also getting a similar message:

msgp: too few bytes left to read object while sending 2 (size=2KB) traces. Total: 567742, Received: 567742, Sent: 566918, Failed: 824. Status: 400 Bad Request (Will not log errors for 5 minutes)

What could cause the 400 Bad Request?

[pellizzetti]

Hi! We are experiencing this issue with a Kotlin application. Any tips on how to troubleshoot this issue?

dd-trace-java version: 0.110.0~a26ee1d794
Agent image: gcr.io/datadoghq/agent:7.40.1-jmx

[schmidti159]

Same issue with dd-java-agent: 1.20.1

The java agent logs:
[dd.trace 2024-03-04 08:42:48:610 +0000] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - msgp: too few bytes left to read object while sending 1 (size=2KB) traces. Total: 108442, Received: 108442, Sent: 108439, Failed: 3. Status: 400 Bad Request (Will not log errors for 5 minutes)

On the datadog agent side we get this error message:
2024-03-04 08:42:48 UTC | TRACE | ERROR | (pkg/trace/api/api.go:503 in handleTraces) | Cannot decode v0.4 traces payload: msgp: too few bytes left to read object

There is an open issue for datadog agent for this error message, but also not much information from there: DataDog/datadog-agent#9555