Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GELF/HTTP timeout with http keep-alive #5720

Closed
jalogisch opened this Issue Feb 25, 2019 · 1 comment

Comments

Projects
None yet
3 participants
@jalogisch
Copy link
Member

jalogisch commented Feb 25, 2019

Current Behavior

Create a HTTP GELF Input and use the following curl to post a message:

curl -v -i -XPOST http://graylog.lan:12202/gelf -d '{"_testing_number": 12524, "_testing_string": "string", "host": "graylog.lan", "version": "1.1", "message": "perfmon dump", "short_message": "first"}' http://graylog.lan:12202/gelf

The second request will run into a timeout because the server does not respond.

Context

With 3.0.beta.2 that worked, but beta.3 breaks this, what makes it related to:

REFS: #5143
REFS: #5503

In beta2 and before, after Graylog responds with a 202 it closes the socket. This forces the client to start a new TCP socket and all is good.
In beta3+, the socket remains open. The client tries to re-use the socket, but Graylog doesn't respond to the request. This eventually forces either the client to fail (or app case) or in the case of CURL to re-try after a timeout.

[ FD#2392 ]

Your Environment

  • Graylog Version: 3.0

@jalogisch jalogisch added the bug label Feb 25, 2019

@mpfz0r mpfz0r self-assigned this Feb 26, 2019

@mpfz0r mpfz0r changed the title GELF/HTTP drops silent messages GELF/HTTP timeout with http keep-alive Feb 27, 2019

@mpfz0r

This comment has been minimized.

Copy link
Member

mpfz0r commented Feb 27, 2019

It seems that http keep-alive was always broken.
The fix from #5503 only exposed that. Before it would always close the connection after the error,
thus not doing keep-alive at all :-)

mpfz0r added a commit that referenced this issue Feb 27, 2019

Fix http input keep-alive handling
We were only responding to the first request on an http keep-alive connection
with a 202.
Any following requests would not be answered, leading to a timeout in the client.

Configuring a ChannelFuture for CLOSE_ON_FAILURE did expose the reason
for the bug:

`io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException: unexpected message type:
DefaultHttpResponse, state: 1`

We ought to use `DefaultFullHttpResponse` instead of `DefaultHttpResponse`.
Also keep the `CLOSE_ON_FAILURE` to make this more robust for future bugs.

Fixes #5720

@deeshe deeshe added the triaged label Mar 5, 2019

dennisoelkers added a commit that referenced this issue Mar 13, 2019

Fix http input keep-alive handling (#5728)
We were only responding to the first request on an http keep-alive connection
with a 202.
Any following requests would not be answered, leading to a timeout in the client.

Configuring a ChannelFuture for CLOSE_ON_FAILURE did expose the reason
for the bug:

`io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException: unexpected message type:
DefaultHttpResponse, state: 1`

We ought to use `DefaultFullHttpResponse` instead of `DefaultHttpResponse`.
Also keep the `CLOSE_ON_FAILURE` to make this more robust for future bugs.

Fixes #5720

mpfz0r added a commit that referenced this issue Mar 13, 2019

Fix http input keep-alive handling (#5728)
We were only responding to the first request on an http keep-alive connection
with a 202.
Any following requests would not be answered, leading to a timeout in the client.

Configuring a ChannelFuture for CLOSE_ON_FAILURE did expose the reason
for the bug:

`io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException: unexpected message type:
DefaultHttpResponse, state: 1`

We ought to use `DefaultFullHttpResponse` instead of `DefaultHttpResponse`.
Also keep the `CLOSE_ON_FAILURE` to make this more robust for future bugs.

Fixes #5720

(cherry picked from commit 9f34e09)

dennisoelkers added a commit that referenced this issue Mar 13, 2019

Fix http input keep-alive handling (#5728) (#5769)
We were only responding to the first request on an http keep-alive connection
with a 202.
Any following requests would not be answered, leading to a timeout in the client.

Configuring a ChannelFuture for CLOSE_ON_FAILURE did expose the reason
for the bug:

`io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException: unexpected message type:
DefaultHttpResponse, state: 1`

We ought to use `DefaultFullHttpResponse` instead of `DefaultHttpResponse`.
Also keep the `CLOSE_ON_FAILURE` to make this more robust for future bugs.

Fixes #5720

(cherry picked from commit 9f34e09)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.