[Auditbeat] Recover from errors in audit monitoring routine

[adriansr]

The auditd module spawns a monitoring goroutine that fetches auditd status every 15s. Due to this routine using a single audit client, if an update fails (because a netlink message is late or other causes), the audit client can get out of sync with the stream, failing in all subsequent requests.

For reasons that aren't 100% clear to me at the moment, this error condition leads to a lot of [audit_send_repl] (2.6.x) / [audit_send_reply] (3.x+) kernel threads being created. (Reproduced in 2.6.32, no other versions tested).

The following error will appear every 15s:

ERROR [auditd] auditd/audit_linux.go:183 get status request failed:failed to get audit status ack: unexpected sequence number for reply (expected 6286 but got 6285)

ps -ef will show a lot of audit_send_repl threads:

[...]
root     27790     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27791     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27792     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27793     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27794     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27795     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27796     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27797     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27798     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27799     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27800     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27801     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27802     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27803     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27804     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27805     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27806     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27807     2  0 12:52 ?        00:00:00 [audit_send_repl]
root     27808     2  0 12:52 ?        00:00:00 [audit_send_repl]
[...]

This patch updates the error-handling logic to create a new audit client when a status update fails, allowing to recover and preventing the proliferation of audit_send_repl kernel threads.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

It's easy to reproduce this issue by modifying the code at

beats/auditbeat/module/auditd/audit_linux.go

Lines 159 to 182 in bb973c4

	if ms.kernelLost.enabled {
	client, err := libaudit.NewAuditClient(nil)
	if err != nil {
	reporter.Error(err)
	ms.log.Errorw("Failure creating audit monitoring client", "error", err)
	}
	go func() {
	defer client.Close()
	timer := time.NewTicker(lostEventsUpdateInterval)
	defer timer.Stop()
	for {
	select {
	case <-reporter.Done():
	return
	case <-timer.C:
	if status, err := client.GetStatus(); err == nil {
	ms.updateKernelLostMetric(status.Lost)
	} else {
	ms.log.Error("get status request failed:", err)
	}
	}
	}
	}()
	}

• Set a lower update interval (milliseconds)
• call client.GetStatusAsync(false) outside of the polling loop.

Similar can be used to validate this fix. Ideally sending an async getstatus every few iterations of the loop.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22673 updated]

• Start Time: 2020-11-19T15:58:29.778+0000

• Duration: 23 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	455
Skipped	61
Total	516

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	455
Skipped	61
Total	516

[andrewkroh]

If this one fails then I think client == nil which will cause problems on subsequent iterations.

[adriansr]

D'oh! Good catch