-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FLUME-3133 Add client IP / hostname headers to Syslog sources. #234
Conversation
In the newer version of the Syslog message format (RFC-5424) the hostname is not a mandatory header anymore so the Syslog client might not send it. On the Flume side it would be a useful information that could be used in interceptors or for event routing. To keep this information, two new properties have been added to the Syslog sources: clientIPHeader and clientHostnameHeader. Flume users can define custom event header names through these parameters for storing the IP address / hostname of the Syslog client in the Flume event as headers. The IP address / hostname are retrieved from the underlying network sockets, not from the Syslog message. This change is based on the patch submitted by Jinjiang Ling which has been rebased onto the current trunk and the review comments have been implemented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @turcsanyip for the contribution,
it looks good, however, the tests are failing:
2018-11-08 23:52:05,824 (main) [INFO - org.apache.flume.source.MultiportSyslogTCPSource.start(MultiportSyslogTCPSource.java:214)] Multiport Syslog TCP source null started.
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at org.apache.flume.source.TestMultiportSyslogTCPSource.testClientHeaders(TestMultiportSyslogTCPSource.java:559)
I will try to get more details.
are not the same. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tests are ok.
+1
LGTM, +1, ship it. |
In the newer version of the Syslog message format (RFC-5424) the hostname is not a mandatory header anymore so the Syslog client might not send it. On the Flume side it would be a useful information that could be used in interceptors or for event routing. To keep this information, two new properties have been added to the Syslog sources: clientIPHeader and clientHostnameHeader. Flume users can define custom event header names through these parameters for storing the IP address / hostname of the Syslog client in the Flume event as headers. The IP address / hostname are retrieved from the underlying network sockets, not from the Syslog message. This change is based on the patch submitted by Jinjiang Ling which has been rebased onto the current trunk and the review comments have been implemented. This closes apache#234 Reviewers: Ferenc Szabo, Endre Major (Peter Turcsanyi via Ferenc Szabo)
use gopkg.in/mgo.v2 package for mgoutil supporting mongodb auth conne…
In the newer version of the Syslog message format (RFC-5424) the hostname
is not a mandatory header anymore so the Syslog client might not send it.
On the Flume side it would be a useful information that could be used
in interceptors or for event routing.
To keep this information, two new properties have been added to the Syslog
sources: clientIPHeader and clientHostnameHeader.
Flume users can define custom event header names through these parameters
for storing the IP address / hostname of the Syslog client in the Flume
event as headers.
The IP address / hostname are retrieved from the underlying network sockets,
not from the Syslog message.
This change is based on the patch submitted by Jinjiang Ling which has been
rebased onto the current trunk and the review comments have been implemented.