Transparent HTTP source request timeout

[dlvenable]

Is your feature request related to a problem? Please describe.

The HTTP Source plugin provides a configurable request_timeout parameter. As a user, I'd expect that this is the actual timeout for HTTP requests.

However, the HTTP request timeout is doubled from that parameter.

data-prepper/data-prepper-plugins/http-source/src/main/java/com/amazon/dataprepper/plugins/source/loghttp/HTTPSource.java

Line 99 in 0a5910b

sb.requestTimeout(Duration.ofMillis(2*requestTimeoutInMillis));

Describe the solution you'd like

I propose the following changes:

• No longer double the HTTP request timeout
• Set the buffer timeout to a value derived from the request_timeout. My initial thought is that it can be 80% of the request_timeout. e.g. int bufferTimeoutMillis = (int)(0.8 * requestTimeoutMillis). Some additional logic may be necessary to ensure it is a valid time.

Additionally, the following can optionally be included:

• A configuration for users to set their own buffer timeout. For example, buffer_timeout. This should have a default value so that user's don't have to configure it.

Describe alternatives you've considered (Optional)

There could be other approaches to deriving the buffer timeout from the request timeout. Perhaps subtracting some time rather than using a percentage. But, these would be more complicated because they would not work for smaller values. A future improvement could consider using a percentage when request_timeout is small and a subtraction approach when request_timeout is large enough. But, this seems unnecessary.