[Auditbeat] Fix up socket dataset runaway CPU usage

[andrewstucki]

What does this PR do?

Fix for auditbeat runaway CPU usage: #19141

So, here's the explanation, basically everything was pretty much as described in the previous PR (#19033), the only additional things that I found were that:

1. When a *socket is terminated by another socket with a different kernel tid it's moved to the closing LRU list.
2. The new *socket is added to the state socks map with the ptr reference pointing to it
3. The reaper comes along and hits the following code path:

	for item := s.closing.peek(); item != nil && item.Timestamp().Before(deadline); {
		if sock, ok := item.(*socket); ok {
			s.onSockTerminated(sock)
		} else {
			s.closing.get()
		}
		item = s.closing.peek()
	}

4. The old "terminated" socket is now in a "closing" state, so onSockTerminated is called again
5. In onSockTerminated the socket is pruned again from the socks map with the call to delete(s.socks, sock.sock)
6. The problem is that the socks map now refers to the new *socket rather than the old one
7. Eventually if the new *socket times out onSockDestroyed is called on it with the code that's doing the peek on the socketLRU in the reaper code
8. That was taking a reference to the socket pointer that had been deleted from the socks map in step 5
9. onSockDestroyed was running the following code:

	sock, found = s.socks[ptr]
	if !found {
		return nil
	}

10. found was returning false and the function was returning
11. Because of the call to s.socketLRU.peek() the same socket was getting returned over and over, resulting in the reaper routine getting wedged in a tight for loop (hence the high CPU usage).

The fix

Basically we pass a reference to the *socket object in the reaper's onSockDestroyed call, that way we don't have to look up the socket in s.socks and, instead handle the socket closure directly.

Related issues

• Closes Auditbeat 7.7.x Poor Performance: 100%+ CPU Usage with System Module Socket Dataset Enabled #19141

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #19764 updated]

• Start Time: 2020-07-09T11:17:41.326+0000

• Duration: 31 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	230
Skipped	49
Total	279

[adriansr]

Great investigation @andrewstucki, thanks for fixing this!

It explains the profiles shared in the discuss thread:

With onSockDestroyed() taking most of the CPU and only doing a map lookup.

[adriansr]

LGTM, needs a changelog entry

[andrewstucki]

Just added on to your previous changelog entry and rebased from master to hopefully fix the weird CI mage issues

[adriansr]

I've shared a custom 7.8.1 build with this fix on the discuss thread.