Packetbeat process monitor enhancements #7135

adriansr · 2018-05-18T11:26:08Z

This patch extends the functionality of the processes monitor in packetbeat:

report the full command-line for all processes (not only those configured as monitored).
Add support for Windows process monitoring.
Disable the monitor when using file input.
Get rid of polling for processes and sockets.

Closes #541

Updated functionality

The original monitor will only track those processes that match a pattern specified in the configuration:

packetbeat.procs:
  enabled: true
  monitored:
    - process: mysqld
      cmdline_grep: mysqld

    - process: pgsql
      cmdline_grep: postgres

    - process: nginx
      cmdline_grep: nginx

    - process: app
      cmdline_grep: gunicorn

Will result in the following fields being added to generated events:

"proc": "mysqld" when the process is detected to be on the server-side of a stream, and
"client_proc": "mysqld" when the process is detected on the client-side.

While maintaining this behavior, this patch will add:

"cmdline": "/usr/bin/mysqld --bind-address=0.0.0.0 --data-dir=/var/lib/mysql"

and the same for client_cmdline when the process is on the client-side of a connection.

The cmdline and client_cmdline fields will be added to events generated by any running process. The proc and client_proc fields will be added only to those processes in the monitored section, so the original behavior is maintained.

Windows support

Added support for Windows using IP Helper functions (see issue #541).

Gosigar dependency

The original feature was tailored to Linux. To make it multiplatform, gosigar is used to get process information. This also means that it is no longer required to run packetbeat as root to access the command-line from processes belonging to other users.

Get rid of processes/sockets polling

The original implementation would refresh the process list with a specified frequency (refreshPidsFreq). The mapping of TCP ports to processes would also be updated after a specified interval (maxReadFreq). This caused the feature to frequently miss process information for an event, specially with short-lived processes and connections.

In this implementation, such delays are removed. The list of processes and active sockets is updated as necessary. While this is a somewhat expensive operation, ranging from ~10ms under Linux and ~100ms in Windows, it is only performed when an event is being populated, and the source or destination of this traffic is a local port for which we don't have any information yet.

As the processes monitor is an opt-in feature, disabled by default, I find it more important to aim for more reliable behavior in the expense of some delays.

Missing features

Some limitations of the original implementation have not been addressed by this PR:

Support UDP sockets.
Handle processes bound to a subset of the local network addresses, and by extension two or more processes using the same port on different local addresses.
Fetch Linux sockets using elastic/gosigar/sys/linux/inetdiag (faster than the current proc-based access).
Move Windows socket querying to gosigar/sys/windows and use it in metricbeat's system/socket module.
Support macOS.

packetbeat/procs/zsyscall_windows.go

+	procGetExtendedTcpTable = modiphlpapi.NewProc("GetExtendedTcpTable")
+)
+
+func _GetExtendedTcpTable(pTcpTable uintptr, pdwSize *uint32, bOrder bool, ulAf uint32, tableClass uint32, reserved uint32) (code syscall.Errno, err error) {


packetbeat/procs/zsyscall_windows.go

+var (
+	modiphlpapi = windows.NewLazySystemDLL("iphlpapi.dll")
+
+	procGetExtendedTcpTable = modiphlpapi.NewProc("GetExtendedTcpTable")


packetbeat/procs/zsyscall_windows.go

+)
+
+var (
+	errERROR_IO_PENDING error = syscall.Errno(errnoERROR_IO_PENDING)


packetbeat/procs/zsyscall_windows.go

+// Do the interface allocations only once for common
+// Errno values.
+const (
+	errnoERROR_IO_PENDING = 997


houndci-bot · 2018-05-18T11:26:27Z

packetbeat/procs/syscall_windows.go

+	localScopeId  uint32
+	localPort     uint32
+	remoteAddr    [16]byte
+	remoteScopeId uint32


struct field remoteScopeId should be remoteScopeID

houndci-bot · 2018-05-18T11:26:28Z

packetbeat/procs/procs_windows.go

+	e(uint32FieldToPort(row.localPort), int(row.owningPID))
+}
+
+func (_ TCPRowOwnerPIDExtractor) Size() int {


exported method TCPRowOwnerPIDExtractor.Size should have comment or be unexported
receiver name should not be an underscore, omit the name if it is unused

houndci-bot · 2018-05-18T11:26:28Z

packetbeat/procs/procs_windows.go

+	return TCP6RowOwnerPIDExtractor(fn)
+}
+
+func (e TCPRowOwnerPIDExtractor) Extract(ptr unsafe.Pointer) {


exported method TCPRowOwnerPIDExtractor.Extract should have comment or be unexported

houndci-bot · 2018-05-18T11:26:28Z

packetbeat/procs/procs_windows.go

+	{windows.AF_INET6, _GetExtendedTcpTable, TCP_TABLE_OWNER_PID_ALL, extractTCP6RowOwnerPID},
+}
+
+func (proc *ProcessesWatcher) GetLocalPortToPIDMapping() (ports map[uint16]int, err error) {


exported method ProcessesWatcher.GetLocalPortToPIDMapping should have comment or be unexported

houndci-bot · 2018-05-18T11:26:28Z

packetbeat/procs/procs_windows.go

+type extractorFactory func(fn callbackFn) extractor
+
+type TCPRowOwnerPIDExtractor callbackFn
+type TCP6RowOwnerPIDExtractor callbackFn


exported type TCP6RowOwnerPIDExtractor should have comment or be unexported

houndci-bot · 2018-05-18T11:26:28Z

packetbeat/procs/procs_windows.go

+type callbackFn func(uint16, int)
+type extractorFactory func(fn callbackFn) extractor
+
+type TCPRowOwnerPIDExtractor callbackFn


exported type TCPRowOwnerPIDExtractor should have comment or be unexported

houndci-bot · 2018-05-22T03:54:21Z

packetbeat/procs/procs_other.go

+
+package procs
+
+func (proc *ProcessesWatcher) GetLocalPortToPIDMapping() (ports map[uint16]int, err error) {


exported method ProcessesWatcher.GetLocalPortToPIDMapping should have comment or be unexported

houndci-bot · 2018-05-22T03:54:21Z

packetbeat/procs/procs_linux.go

+	inode uint64
+}
+
+func (proc *ProcessesWatcher) GetLocalPortToPIDMapping() (ports map[uint16]int, err error) {


exported method ProcessesWatcher.GetLocalPortToPIDMapping should have comment or be unexported

houndci-bot · 2018-05-22T03:54:21Z

packetbeat/procs/procs.go


-	return false
+func (proc *ProcessesWatcher) GetLocalIPs() ([]net.IP, error) {


exported method ProcessesWatcher.GetLocalIPs should have comment or be unexported

houndci-bot · 2018-05-22T03:54:21Z

packetbeat/procs/procs.go

-		if ip.Equal(addr) {
-			return true
-		}
+func (proc *ProcessesWatcher) GetProcessCommandLine(pid int) (cmdLine string) {


exported method ProcessesWatcher.GetProcessCommandLine should have comment or be unexported

libbeat/common/tuples.go

 	SrcIP, DstIP     net.IP
 	SrcPort, DstPort uint16
+}
+
+type IPPortTuple struct {


libbeat/common/tuples.go

@@ -15,10 +15,15 @@ const MaxIPPortTupleRawSize = 16 + 16 + 2 + 2

 type HashableIPPortTuple [MaxIPPortTupleRawSize]byte

-type IPPortTuple struct {
-	IPLength         int
+type BaseTuple struct {


andrewkroh · 2018-05-22T01:26:00Z

packetbeat/procs/procs.go


-	logp.Debug("procsdetailed", "UpdateMappingEntry(): port=%d pid=%d", port, p.name)
+	logp.Debug("procsdetailed", "UpdateMappingEntry(): port=%d pid=%d process='%s' name=%s",


Looks like that should “updateMappingEntry()”.

ruflin · 2018-05-22T12:24:10Z

I wonder if we should make this an autodiscovery provider instead so we can reuse it in all Beats?

andrewkroh

LGTM. The PR description is very helpful. I'm looking forward to the future enhancements too.

andrewkroh · 2018-05-22T12:59:09Z

packetbeat/procs/procs_windows.go

+	class     uint32
+	extractor extractorFactory
+}{
+	{windows.AF_INET, _GetExtendedTcpTable, TCP_TABLE_OWNER_PID_ALL, extractTCPRowOwnerPID},


These TCP table structures look fun to work with [read with sarcastic tone].

This patch extends the functionality of the processes monitor in packetbeat: - report the full command-line for all processes (not only those configured as `monitored`. - Add support for Windows process monitoring. - Disable the monitor when using file input. - Get rid of refresh delays. Closes elastic#541

adriansr · 2018-05-25T13:31:50Z

Updated CHANGELOG

andrewkroh · 2018-05-25T13:41:26Z

jenkins, test it

adriansr requested a review from andrewkroh May 18, 2018 11:26

adriansr added Packetbeat discuss Issue needs further discussion. labels May 18, 2018

houndci-bot reviewed May 18, 2018

View reviewed changes

adriansr added the enhancement label May 18, 2018

adriansr force-pushed the packetbeat/feature/procs branch 3 times, most recently from 070f64b to 8a519bf Compare May 22, 2018 03:54

houndci-bot reviewed May 22, 2018

View reviewed changes

adriansr force-pushed the packetbeat/feature/procs branch from 8a519bf to 76d7080 Compare May 22, 2018 04:09

andrewkroh reviewed May 22, 2018

View reviewed changes

andrewkroh approved these changes May 22, 2018

View reviewed changes

adriansr force-pushed the packetbeat/feature/procs branch from 76d7080 to 5d9b954 Compare May 23, 2018 05:58

adriansr force-pushed the packetbeat/feature/procs branch from 5d9b954 to c5c3a1a Compare May 25, 2018 13:03

andrewkroh merged commit cf19da7 into elastic:master May 26, 2018

adriansr mentioned this pull request Jul 13, 2018

Adapt packetbeat process monitor to ECS #7595

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Packetbeat process monitor enhancements #7135

Packetbeat process monitor enhancements #7135

adriansr commented May 18, 2018

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

houndci-bot May 18, 2018

houndci-bot May 18, 2018

houndci-bot May 18, 2018

houndci-bot May 18, 2018

houndci-bot May 18, 2018

houndci-bot May 18, 2018

houndci-bot May 22, 2018

houndci-bot May 22, 2018

houndci-bot May 22, 2018

houndci-bot May 22, 2018

This comment was marked as resolved.

This comment was marked as resolved.

andrewkroh May 22, 2018

ruflin commented May 22, 2018

andrewkroh left a comment •

edited

Loading

andrewkroh May 22, 2018

adriansr commented May 25, 2018

andrewkroh commented May 25, 2018


		package procs

		func (proc *ProcessesWatcher) GetLocalPortToPIDMapping() (ports map[uint16]int, err error) {


		return false
		func (proc *ProcessesWatcher) GetLocalIPs() ([]net.IP, error) {


		logp.Debug("procsdetailed", "UpdateMappingEntry(): port=%d pid=%d", port, p.name)
		logp.Debug("procsdetailed", "UpdateMappingEntry(): port=%d pid=%d process='%s' name=%s",

Packetbeat process monitor enhancements #7135

Packetbeat process monitor enhancements #7135

Conversation

adriansr commented May 18, 2018

Updated functionality

Windows support

Gosigar dependency

Get rid of processes/sockets polling

Missing features

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

Choose a reason for hiding this comment

ruflin commented May 22, 2018

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr commented May 25, 2018

andrewkroh commented May 25, 2018

andrewkroh left a comment •

edited

Loading