Agent: Support sending large execution logs / debug info in several POST requests

[petertrr]

500 Server Error for HTTP POST "/executionLogs"
org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
	at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:98)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ? HTTP POST "/executionLogs" [ExceptionHandlingWebHandler]
Stack trace:

[petertrr]

It might make sense to change request type from plain JSON with text to multipart/form-data

[orchestr7]

It might make sense to change request type from plain JSON with text to multipart/form-data

You think we need to put logs into the compressed file?
Or you suggest to use binary format @petertrr?

[orchestr7]

I do not think that multipart data will work good with a data in a serializable data class:
https://stackoverflow.com/questions/69628364/kotlin-ktor-request-class-for-multipart-form-data

[petertrr]

You think we need to put logs into the compressed file? Or you suggest to use binary format @petertrr

I think, at first we can try to send them as multipart instead of application/json, because for multipart Spring has different receiving logic without data size limits. ktor-client should be able to do this: https://ktor.io/docs/request.html#upload_file.

Compression of text data should also help, but it can be added later and be sent as multipart/form-data as well

[kgevorkyan]

You think we need to put logs into the compressed file? Or you suggest to use binary format @petertrr

I think, at first we can try to send them as multipart instead of application/json, because for multipart Spring has different receiving logic without data size limits. ktor-client should be able to do this: https://ktor.io/docs/request.html#upload_file.

Compression of text data should also help, but it can be added later and be sent as multipart/form-data as well

Why do not simply increase the limits for now by spring.codec.max-in-memory-size?

[petertrr]

Why do not simply increase the limits for now by spring.codec.max-in-memory-size?

It will still lead to possibly undetermined behavior, some new combination of tools and test suites might introduce even greater amounts of data.