Unable to read payload context deadline exceeded

[strowk]

Describe what happened:
Recently (since May ~10 I think) we started to periodically get this message in logs of datadog-agent:

Error code "400 Bad Request" received while sending transaction to "https://7-35-2-app.agent.datadoghq.eu/api/v1/series?api_key=***************************": "Unable to read payload context deadline exceeded", dropping it

Now I read this that agent was sending too big a payload, which caused server to timeout reading the payload. Or maybe network was too slow. Is it some sort of wide spread issue on servers or some change in backend? Or should I adjust something in agent config? We are using latest-jmx image, so it is always the newest.
These errors cause our alerts to go off and this is very noisy. I am used (a bit) to datadog-agent spamming periodically some error messages about timeouts and such, but this was not so often before and this message is relatively new. Before we maybe could have 2 errors a month, usually about timeouts between AWS and agent. Now we have this message every 2-3 days.

Describe what you expected:

• no errors

Steps to reproduce the issue:

• Use datadog-agent in JVM service
• Send metrics (I guess)

Additional environment details (Operating System, Cloud provider, etc):
AWS, Fargate container, we are attaching dd-trace-java too, which I have updated in April, but it does not seem to be directly connected (errors seem to start appearing couple of weeks later).

[delenamalan]

We started to get this error recently very sporadically when pushing metrics to Datadog using the API. There doesn't seem to be a discernable pattern as to when this error is returned.

[don41382]

Same issue here. We noticed an increase since 18th of November. In addition, the forwarder seems to have problems too:

2023-11-23 04:48:58 UTC | CORE | ERROR | (comp/forwarder/defaultforwarder/transaction/transaction.go:359 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "https://7-49-1-app.agent.datadoghq.eu/api/v1/check_run": "Unable to read payload context deadline exceeded", dropping it

[nabillarahmanizhafira]

do we have any update on this? because I also have similar issue with this log:

May 18 15:04:01 datadog-agent agent[14557]: 2024-05-18 15:04:01 WIB | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "https://7-43-0-app.agent.datadoghq.com/api/v2/series": "\n\x1cUnable to decomp>
May 18 15:04:20 datadog-agent agent[14557]: 2024-05-18 15:04:20 WIB | CORE | INFO | (pkg/serializer/serializer.go:413 in sendMetadata) | Sent metadata payload, size (raw/compressed): 750/357 bytes.

Thanks