Seeing read limit exceeded error

[manojm321]

I have telegraf writing about 100 points every 15 seconds and I'm seeing follow error:
level=info caller=errors.go:64 orgID=fake path=/api/v1/push/influx/write msg="problem reading body" response_code=400 err="read limit exceeded"
Any Idea what's going on?

[alexgreenbank]

Sorry we didn't notice this earlier.

tl;dr: Add -max.request.size.bytes 10485760 to the proxy command line and that should stop the errors.

To avoid this error you need to up the -max.request.size.bytes argument from the default of 102400. What value you need to increase it to is dependent on your data but we have just run in to the read limit exceeded errors on the proxies we run and so we've upped the value to 10485760 (which is 10MB).

When reading/parsing the influx requests the influx libraries used require a maximum number of bytes it will read from a network connection (as a general safety protection against DOS style attacks sending more data that a process can handle). Some clients are obviously sending more than the default of 102400 and so this error is logged before all of the data on the socket is processed. We're going to do some more investigation internally to analyse the size of requests coming in and then we may decide to up the default that's used in the influx2cortex proxy code, but in the mean time moving it to a bigger number should mitigate the problem.

[alexgreenbank]

#81 now gives logging such as:

level=info ts=2023-03-09T14:15:22.182461323Z caller=api.go:87 orgID=fake path=/api/v1/push/influx/write bytesRead=81953 nosMetrics=2814 response_code=204

We'll leave this to run for a bit and then start to gather some stats for it. Eventually we'd like metrics/histograms for this info but we'll make do with scraping logs for now.

Biggest so far I've seen in the hour since deployment is: level=info ts=2023-03-09T14:20:28.891319869Z caller=api.go:87 orgID=**ELIDED** path=/api/v1/push/influx/write bytesRead=838880 nosMetrics=6927 response_code=204

That was quite easily over the original default of 100KB.

[alexgreenbank]

2023-03-16 23:02:07.595 | level=info ts=2023-03-16T23:02:07.595827063Z caller=api.go:87 orgID=**ELIDED** path=/api/v1/push/influx/write bytesRead=1112191 nosMetrics=19976 response_code=204

[alexgreenbank]

We monitored our proxies for the largest batches seen coming from telegraf agents and found some just over 1MB in size, so I'm bumping the default to 10MB to give plenty of leeway.

#85

[alexgreenbank]

OK, going to close this in a couple of weeks (15th October 2023) if there's no further action.

[alexgreenbank]

October? I meant 15th August 2023.