New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RAW TCP input loses the data after the last 0x0a #4105

Closed
jtkkex opened this Issue Aug 23, 2017 · 2 comments

Comments

Projects
None yet
4 participants
@jtkkex

jtkkex commented Aug 23, 2017

Expected Behavior

Setting up a RAW TCP input, Graylog uses all lines that are written to the input.

Current Behavior

Raw TCP input reads a message in whenever there is a newline (0x0a) coming from the input. The characters between the last 0x0a and closing the TCP connection are discarded. If a sender opens a separate connection for each log line and does not terminate the connections with a 0x0a, Graylog will not process the log lines.

The total byte count on the inputs page shows the incoming bytes, but they are not processed as messages.

Possible Solution

When connection closes, read the remaining bytes and process them normally.

Steps to Reproduce (for bugs)

  1. create a RAW TCP input
  2. create a script that sends a string to the input, such as "See me\nSee me not" (verify with TCPdump that your script does not silently add a \n to the end of the string).

Context

Tried to send log lines from Windows with Powershell.

Your Environment

  • Graylog Version: 2.3.0
  • Elasticsearch Version: 5.5
  • MongoDB Version: 3.4
  • Operating System: CentOS 7

@bernd bernd added the bug label Aug 23, 2017

@bernd bernd added this to the 2.3.1 milestone Aug 23, 2017

@joschi joschi self-assigned this Aug 23, 2017

@joschi

This comment has been minimized.

Contributor

joschi commented Aug 23, 2017

@jtkkex Thanks for reporting this!

The Raw/Plaintext TCP input expects messages to end with the configured delimiter (\0, \n, or \r\n).

We'll see if we can change that with little effort.

Reference:

joschi added a commit that referenced this issue Aug 23, 2017

Use lenient delimiter based frame decoder in TcpTransport
In contrast to the standard `DelimiterBasedFrameDecoder` from Netty, the derivative
`LenientDelimiterBasedFrameDecoder` emits the last bytes received on the channel
even if it doesn't end with the configured delimiter.

Fixes #4105
@bernd

This comment has been minimized.

Member

bernd commented Aug 24, 2017

Moving this to 2.4.0 so we have a chance to test the fix in #4106 more thoroughly.

@bernd bernd modified the milestones: 2.4.0, 2.3.1 Aug 24, 2017

@jalogisch jalogisch added the triaged label Sep 5, 2017

@bernd bernd closed this in #4106 Sep 26, 2017

bernd added a commit that referenced this issue Sep 26, 2017

Use lenient delimiter based frame decoder in TcpTransport (#4106)
* Use lenient delimiter based frame decoder in TcpTransport

In contrast to the standard `DelimiterBasedFrameDecoder` from Netty, the derivative
`LenientDelimiterBasedFrameDecoder` emits the last bytes received on the channel
even if it doesn't end with the configured delimiter.

Fixes #4105

* Retain original copyright notices

[ci skip]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment