hubble: Add support for SockLB tracing

[gandro]

This PR adds a Hubble parser for TraceSockNotify events. Those
events are emitted whenever SockLB is performed, see #20492
for details about the datapath tracing.

This parser is similar to the L3/L4 event parser. But because the events
are traced on the socket level, we do not know the source endpoint id,
source endpoint IP address or source TCP/UDP port of the event. We only
know the source cgroup ID. From the cgroup, we can derive which local
pod emitted the event (via GetPodMetadataForContainer). From the pod
IP we can then perform an IP-based lookup in either the local endpoint
list or the IPCache to determine the source endpoint metadata (endpoint
id, numeric identity, labels, etc). The lookup for destination IP is
almost identical to the threefour parser, with the exception that we do
not have any datapath identity available for fallback.

The resulting flow looks similar to regular FlowType_L3_L4, with a
couple of key differences:

• The cgroup id of the container is available
• Information if the event is pre or post translation point is
  available
• The socket cookie is available
• The TCP/UDP source port is not known
• TCP flags are not known
• Ethernet headers are not known

Unfortunately, sock events can also be performed by non-pod entities
(e.g. the host cgroup). In that case, we cannot perform a mapping to any
local endpoint and therefore the source IP and endpoint data are left
empty. The cgroup id may also not be available on older kernels or
systems with the systemd cgroup driver (though this will likely be fixed
in a follow-up PR). This fact that the source IP may be empty/unknown is
also key difference to the flows which have been produced by Hubble so
far.

---

Here's an example of a pod performing a DNS request and response. The first three events are the DNS request: We first see two the SockLB events (pre and post translation, meaning before and after service backend selection):

And here are the events for the response: We see the response packet arriving at the endpoint, but before it is delivered over the socket, reverse NAT being applied, i.e. the pod IP is once again translated to the service IP (which is where the userspace sent the packet to):

[gandro]

/test

[aditighag]

🚀 I only looked at the commit "hubble: Add support for SockLB tracing" that pertains to parsing socket-lb trace events. The parsing logic looks good. Let's follow up on pruning events from non-pod entities.

Can we print cgroup id like metadata in the verbose json output? Will you be adding json summary in a follow-up?

[gandro]

Can we print cgroup id like metadata in the verbose json output? Will you be adding json summary in a follow-up?

This already done. The JSON schema is derived from the protobuf schema, which is updated in this PR to contain the cgroup id. Once the schema is synced to the Hubble CLI (which we can only do once this PR is merged), hubble observe -o jsonpb will print it too.

[kaworu]

Awesome work @gandro! I have a few comments, mostly cosmetic. Feel free to ignore nits and comments on code that has only moved (I only noticed after review).

[kaworu]

At first I thought decodeRevNat returning a *wrapperspb.BoolValue was a left-over from a previous version of the patch, because isRevNat is never added to the decoded flow and this line is the only call site. We check for isRevNat.GetValue() a few lines below, but we only need a true/false signal at that point.

Looking at the screenshots of the Hubble CLI output, I noticed that we're missing the arrows for TraceSock flows (i.e. we see <>), so related question: would setting IsReply and TrafficDirection based on isRevNat make sense for TraceSock events?

[gandro]

Yes, your suspicion is correct, an earlier version of the PR used the result of decodeRevNat as the reply flag. I've changed it to a bool in the most recent version.

Unfortunately, trying to derive the traffic direction / reply state from the observation point does not work: IsReply cannot be meaningfully populated for TraceSock events because their occur on the socket granularity level, not a packet level.

The screenshot in the PR description only shows the traces from the node where pod-to-a is hosted, where rev nat is indeed only used for the reply. But if we include also the events emitted by the kube-dns host, then the picture looks slightly different, we also see rev nat being used for the non-reply packet:

The screenshot above is using xwing, but it's a single UDP request packet being sent from xwing to kubedns, with a single reply packet being sent back. Please note the timetsamps - as you can see, kind-worker2 (where coredns is hostd) emits a pre-xlate-rev for the initial incoming packet (there is no post event, as there is no translation). The reply packet (the last 4 events) happens half a second later, and does not emit any SockLB event, probably because it's sent back on the same "connected UDP socket".

[gandro]

Another way to formulate this: I think (but I might be mistaken, cc @aditighag) that pre/post-xlate-rev is only executed on recvmsg and getpeername, both system calls don't really imply anything of overall the "traffic direction" (e.g. if it was an ingress or egress connection in policy lingo).

pre/post-xlate-fwd on the other hand is executed for connect and sendmsg. connect is typically used to initiate an egress connection, but the same cannot necessary be said for sendmsg.

[kaworu]

@gandro ah that make sense, thanks for the detailed explanation 👍 one more question about the example:

The reply packet (the last 4 events) happens half a second later, and does not emit any SockLB event, probably because it's sent back on the same "connected UDP socket".

To make sure I understand correctly: the reply packet does not emit any SockLB event on kind-worker2 (i.e. coredns host), but it does in kind-worker (i.e. xwing host) correct? The screenshot's last two events are a couple of pre/post-xlate-rev analogous to the request packet pre/post-xlate-fwd events (which make sense to me), hence the question.

[gandro]

That's correct, yes. My guess here is that coredns is using Golang's net.Conn to send the reply packet, meaning it's using the write system call on a "connected UDP socket". Thus, none of the above cgroup hooks is hit. dig (used inside xwing) on the other hand might be using readmsg/sendmsg syscalls, thus causing a Hubble event for each syscall.

Maybe @aditighag has deeper insight here.

[gandro]

On second thought, connected UDP sockets are only useful for clients. Checking a basic UDP echo server, it seems Go is using recvfrom and sendto, so not sure why the sendto does not cause a Hubble event (it should for unconnected udp sockets, according to https://manpages.debian.org/experimental/bpftool/bpftool-cgroup.8.en.html)

[aditighag]

The recvmsg hook is executed only for regular UDP. TCP (and connected UDP) sockets are connected, so TCP clients send and receive data using send/recv socket calls.
@gandro Your observation is correct. Golang uses a connected UDP client by default. Regarding, sendto for connected UDP, here is a snippet from the man page -

If sendto() is used on a connection-mode (SOCK_STREAM, SOCK_SEQPACKET) socket, the arguments dest_addr and addrlen are ignored (and the error EISCONN may be returned when they are not NULL and 0)

[aditighag]

Another way to formulate this: I think (but I might be mistaken, cc @aditighag) that pre/post-xlate-rev is only executed on recvmsg and getpeername, both system calls don't really imply anything of overall the "traffic direction" (e.g. if it was an ingress or egress connection in policy lingo).

When podA on nodeA talks to podB on nodeB, both fwd and rev translation happen on the nodeA (for UDP). For TCP and connected UDP, it would just be fwd translation.

The {pre, post}-fwd/rev markers already convey the direction. We shouldn't try to interpret this with the same context as policy terminology (ingress/egress).

fwd: svc vip -> backend ip
rev: backend ip -> svc vip

Hope that makes sense.

[gandro]

Thanks for the feedback @aditighag and @kaworu - I think I addressed or commented on all of it.

[gandro]

/test

Job 'Cilium-PR-K8s-1.24-kernel-4.19' has 2 failures but they might be new flakes since it also hit 1 known flakes: #17628 (92.24)

Job 'Cilium-PR-K8s-1.16-kernel-4.9' failed:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: Timed out after 30.000s.

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.16-kernel-4.9 so I can create one.

[gandro]

/test

Job 'Cilium-PR-K8s-1.24-kernel-4.19' has 2 failures but they might be new flakes since it also hit 1 known flakes: #17628 (92.24)

[gandro]

• k8s-1.24-kernel-4.19 hit known flake CI: K8sDatapathConfig Host firewall With VXLAN on K8s 1.24 / Kernel 4.19: host EP is not ready #21979
• CI Multicluster seems to have hit a variant of CI: Multicluster / Cluster mesh (ci-multicluster-1.11, ci-multicluster-1.12, ci-multicluster-1.13): allow-all-except-world connectivity test #21655 (in any case, it's a datapath flake which this PR does not touch).

[aditighag]

Commit "hubble: Add support for SockLB tracing" looks good to me. Nice work!

One minor comment around setting the verdict field.

[aditighag]

The recvmsg hook is executed only for regular UDP. TCP (and connected UDP) sockets are connected, so TCP clients send and receive data using send/recv socket calls.
@gandro Your observation is correct. Golang uses a connected UDP client by default. Regarding, sendto for connected UDP, here is a snippet from the man page -

If sendto() is used on a connection-mode (SOCK_STREAM, SOCK_SEQPACKET) socket, the arguments dest_addr and addrlen are ignored (and the error EISCONN may be returned when they are not NULL and 0)

[aditighag]

Another way to formulate this: I think (but I might be mistaken, cc @aditighag) that pre/post-xlate-rev is only executed on recvmsg and getpeername, both system calls don't really imply anything of overall the "traffic direction" (e.g. if it was an ingress or egress connection in policy lingo).

When podA on nodeA talks to podB on nodeB, both fwd and rev translation happen on the nodeA (for UDP). For TCP and connected UDP, it would just be fwd translation.

The {pre, post}-fwd/rev markers already convey the direction. We shouldn't try to interpret this with the same context as policy terminology (ingress/egress).

fwd: svc vip -> backend ip
rev: backend ip -> svc vip

Hope that makes sense.

[gandro]

/test

[squeed]

Looks good for my files!

[rolinh]

[gandro]

Removed api code owner, as it has been superseeded by sig-hubble-api and is thus not required anymore (if it was required, I could not have removed the review request).

Marking ready to merge.