Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add system socket MetricSet #3246

Merged
merged 3 commits into from Jan 2, 2017

Conversation

Projects
None yet
3 participants
@andrewkroh
Copy link
Member

commented Dec 23, 2016

The system.socket metricset reports an event for each new TCP socket that it
sees. It does this by polling the kernel to get a dump of all sockets.
So using a short polling interval with this metricset is important to
not miss short lived connections.

The metricset reports the process that has the socket open. It does this
by associating the socket's inode to the process that has a file
descriptor open pointing to the socket's inode. It reads /proc and
/proc//fd just prior to polling the kernel to get all sockets.

A reverse lookup can be performed by the metricset on the remote IP and the
returned hostname will be added to the event and cached. The is disabled by
default and can be enabled through the configuration. If a hostname is found
then the eTLD+1 (effective top-level domain plus one level) value will also be
added to the event.

For the IP address fields the index template for Elasticsearch 5.x uses
the ip field type. But for Elasticsearch 2.x it uses string because the
ip field type in 2.x does not support IPv6 addresses.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Dec 23, 2016

Don't forget the CHANGELOG ;-)

@andrewkroh andrewkroh force-pushed the andrewkroh:feature/system-socket-metricset branch from 78119ca to 6abd7c9 Dec 23, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Dec 26, 2016

I added a changelog entry and modified the mapping for ES 5.x to use ip instead of keyword for the IP address fields.

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Dec 29, 2016

Fixes #3257

@tsg

tsg approved these changes Jan 2, 2017

Copy link
Collaborator

left a comment

LGTM. Nice work!

return nil, err
}
if os.Geteuid() != 0 {
debugf("socket process info will only be available for metricbeat " +

This comment has been minimized.

Copy link
@tsg

tsg Jan 2, 2017

Collaborator

This would make sense as a warning, maybe?

This comment has been minimized.

Copy link
@andrewkroh

andrewkroh Jan 2, 2017

Author Member

How about INFO?

This comment has been minimized.

Copy link
@tsg

tsg Jan 2, 2017

Collaborator

Deal :)


// IsEnabled returns true if enabled is not defined or is set to true.
func (c ReverseLookupConfig) IsEnabled() bool {
return c.Enabled == nil || *c.Enabled == true

This comment has been minimized.

Copy link
@tsg

tsg Jan 2, 2017

Collaborator

This mean that by default reverse lookup is enabled, right? I'm not sure about enabling that by default since it can cause performance issues, especially since there's no configurable timeout on the lookup, right?

This comment has been minimized.

Copy link
@andrewkroh

andrewkroh Jan 2, 2017

Author Member

Correct, there is no configurable lookup timeout and the requests are executed serially. It's not an optimal implementation and it can slow down fetching. I'll change it to disabled by default.

andrewkroh added some commits Dec 22, 2016

Add system socket MetricSet
The system.socket metricset reports an event for each new TCP socket that it
sees. It does this by polling the kernel to get a dump of all sockets.
So using a short polling interval with this metricset is important to
not miss short lived connections.

The metricset reports the process that has the socket open. It does this
by associating the socket's inode to the process that has a file
descriptor open pointing to the socket's inode. It reads /proc and
/proc/<pid>/fd just prior to polling the kernel to get all sockets.

A reverse lookup can be performed by the metricset on the remote IP and the
returned hostname will be added to the event and cached. The is disabled by
default and can be enabled through the configuration.  If a hostname is found
then the eTLD+1 (effective top-level domain plus one level) value will also be
added to the event.

For the IP address fields the index template for Elasticsearch 5.x uses
the ip field type. But for Elasticsearch 2.x it uses string because the
ip field type in 2.x does not support IPv6 addresses.

@andrewkroh andrewkroh force-pushed the andrewkroh:feature/system-socket-metricset branch from 0a3fc90 to c5a6c97 Jan 2, 2017

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2017

Rebased. I recommend using the "Rebase and merge" option which will add my 3 separate commits, but will not create a merge commit.

@tsg tsg merged commit 3ab08d1 into elastic:master Jan 2, 2017

4 checks passed

CLA Commit author has signed the CLA
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Build finished.
Details

@monicasarbu monicasarbu deleted the andrewkroh:feature/system-socket-metricset branch Jan 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.