Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLUME-3133 Add client IP / hostname headers to Syslog sources. #234

Closed
wants to merge 2 commits into from

Conversation

turcsanyip
Copy link
Contributor

In the newer version of the Syslog message format (RFC-5424) the hostname
is not a mandatory header anymore so the Syslog client might not send it.
On the Flume side it would be a useful information that could be used
in interceptors or for event routing.
To keep this information, two new properties have been added to the Syslog
sources: clientIPHeader and clientHostnameHeader.
Flume users can define custom event header names through these parameters
for storing the IP address / hostname of the Syslog client in the Flume
event as headers.
The IP address / hostname are retrieved from the underlying network sockets,
not from the Syslog message.

This change is based on the patch submitted by Jinjiang Ling which has been
rebased onto the current trunk and the review comments have been implemented.

In the newer version of the Syslog message format (RFC-5424) the hostname
is not a mandatory header anymore so the Syslog client might not send it.
On the Flume side it would be a useful information that could be used
in interceptors or for event routing.
To keep this information, two new properties have been added to the Syslog
sources: clientIPHeader and clientHostnameHeader.
Flume users can define custom event header names through these parameters
for storing the IP address / hostname of the Syslog client in the Flume
event as headers.
The IP address / hostname are retrieved from the underlying network sockets,
not from the Syslog message.

This change is based on the patch submitted by Jinjiang Ling which has been
rebased onto the current trunk and the review comments have been implemented.
Copy link
Contributor

@szaboferee szaboferee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @turcsanyip for the contribution,

it looks good, however, the tests are failing:

2018-11-08 23:52:05,824 (main) [INFO - org.apache.flume.source.MultiportSyslogTCPSource.start(MultiportSyslogTCPSource.java:214)] Multiport Syslog TCP source null started.

java.net.ConnectException: Connection refused (Connection refused)

	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:244)
	at org.apache.flume.source.TestMultiportSyslogTCPSource.testClientHeaders(TestMultiportSyslogTCPSource.java:559)

I will try to get more details.

@szaboferee
Copy link
Contributor

context.put("host", InetAddress.getLoopbackAddress().getHostAddress());
and

InetAddress localHost = InetAddress.getLocalHost();
Socket syslogSocket = new Socket(localHost, port);

are not the same. getLocalHost() is a different IP

Copy link
Contributor

@szaboferee szaboferee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are ok.
+1

@majorendre
Copy link
Contributor

LGTM, +1, ship it.

@asfgit asfgit closed this in 671fc3f Nov 14, 2018
szaboferee pushed a commit to szaboferee/flume that referenced this pull request Jan 16, 2019
In the newer version of the Syslog message format (RFC-5424) the hostname
is not a mandatory header anymore so the Syslog client might not send it.
On the Flume side it would be a useful information that could be used
in interceptors or for event routing.
To keep this information, two new properties have been added to the Syslog
sources: clientIPHeader and clientHostnameHeader.
Flume users can define custom event header names through these parameters
for storing the IP address / hostname of the Syslog client in the Flume
event as headers.
The IP address / hostname are retrieved from the underlying network sockets,
not from the Syslog message.

This change is based on the patch submitted by Jinjiang Ling which has been
rebased onto the current trunk and the review comments have been implemented.

This closes apache#234

Reviewers: Ferenc Szabo, Endre Major

(Peter Turcsanyi via Ferenc Szabo)
waidr pushed a commit to waidr/flume that referenced this pull request Jul 24, 2019
use gopkg.in/mgo.v2 package for mgoutil supporting mongodb auth conne…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants