monitor: Initialize agent in deamon early

[gandro]

The pkg/monitor/agent package mainly performs three tasks:

• Managing subscribers of monitor events (such as cilium monitor
  clients or the Hubble observer)
• Receiving and broadcasting user-space notifications ("agent events")
  to the above subscribers.
• Polling the perf event ring buffer for datapath notifications and
  broadcasting them to subscribers.

The third task can only be performed once the perf event ring buffer map
has been set up by the daemon. However, some subsystems will try to send
user-space notifications before the events map has been set up. This
often causes an unnecessary log warning to be emitted, which regularly
confuses users (see #14146). Fundamentally however, there is no reason
to drop user-space events if the datapath is not set up yet.

Therefore, this commit splits the monitor agent constructor into two
functions: The first one, NewAgent initializes the subscriber
management and allows user-space notifications to be received early, the
second function AttachToEventsMap opens the perf event ring buffer to
poll datapath events.

By splitting up the initialization phase and invoking the constructor
earlier, we can avoid the data race in #17167 and we also avoid emitting
a unhelpful warning if user-space events are created early.

Note: Because subscribers (i.e. Hubble or cilium monitor clients) are
still attached relatively late, we will still drop early events if there
is no one listening for them. This commit does not address that - but
instead of emitting a confusing warning that the user cannot do anything
about, it will drop the event the same way Cilium up to version 1.8 did.

Fixes: #17167

[gandro]

test-me-please

Edit: ConfirmanceGKE hit known flake #17337

[joamaki]

LGTM, just one comment.

[joamaki]

shouldn't this check be before we call LoadPinnedMap?

[gandro]

Yes, good catch, we should. Especially since the code doesn't close the map fd if this case is hit in the current code. Will fix.

[gandro]

Moved the call LoadPinnedMap below the check. Unfortunately this means that we need to keep the lock while loading the map (to avoid a TOCTOW issue with a.events), but I think that should be fine.

[gandro]

test-me-please

Job 'Cilium-PR-K8s-GKE' has 1 failure but they might be new flake since it also hit 1 known flake: #17403 (79.85)

[gandro]

Marking this ready to merge. Both GKE failures are known flakes and this PR fixes some other unit test flakes, so it should be exempt from the zero flake merge policy.

[joestringer]

This can mitigate issues on v1.10 where Cilium can erroneously print warnings in the log on startup like level=warning msg="Failed to send policy update as monitor notification" error="monitor agent is not set up". I raised the topic of backporting with @gandro and it seems low-risk and will help to filter out some of the noise on startup.