eBPF control plane signals #3237

NDStrahilevitz · 2023-06-14T12:05:58Z

Add a control plane package which attaches specialized shorter eBPF probes to guarantee lifecycle events reach tracee.
Currently, only container lifecycle events are handled.

Author: Nadav Strahilevitz <nadav.strahilevitz@aquasec.com>
Date:   Sun Jun 4 09:04:16 2023 +0000

    ebpf: refactor arg buffer to type

Author: Nadav Strahilevitz <nadav.strahilevitz@aquasec.com>
Date:   Wed Jun 7 12:14:55 2023 +0000

    bufferdecoder: add DecodeArguments method

Author: Nadav Strahilevitz <nadav.strahilevitz@aquasec.com>
Date:   Wed Jun 7 12:29:37 2023 +0000

    events/parse: refactor parse.ArgVal
    
    Take arguments array directly instead of the entire event.

Author: Nadav Strahilevitz <nadav.strahilevitz@aquasec.com>
Date:   Wed Jun 7 12:31:05 2023 +0000

    ebpf: add control plane
    
    The Control Plane's intention is to process "signal programs" which
    would previously depend on the event submission success.
    
    Instead, a separate buffer and slimmer type is used for these critical
    event processes.

Resolve #3086

NDStrahilevitz · 2023-06-14T12:12:17Z

@AlonZivony FYI, if this is merged before the process tree, it might be a good idea to connect it to this instead of the regular event buffer. If you see any limitations for that LMK here so I can address them.

@yanivagman @rafaeldtinoco Currently we automatically pass cgroup_mkdir/rmdir in all scopes with no filter check, WDYT about removing the bypass in this PR?

yanivagman · 2023-06-19T10:41:22Z

@yanivagman @rafaeldtinoco Currently we automatically pass cgroup_mkdir/rmdir in all scopes with no filter check, WDYT about removing the bypass in this PR?

Yes please.
Going to review it next.

yanivagman · 2023-06-19T10:59:39Z

pkg/bufferdecoder/eventsreader.go

-			pathDecoder := New(trimmedPath)
-			sunPath, err = readStringVarFromBuff(pathDecoder, 108)
-		}
+		sunPath, err := readStringVarFromBuff(ebpfMsgDecoder, 108)


There were reasons for the above additions like having the extra NULL bytes, why do you think it is ok to remove those?

As I saw it doing it in another buffer was unnecessary overhead (in terms of readability mostly). By moving the cursor it has the same effect in the buffer.

@yanivagman About this and the below comment, this is a one time used function for this specific flow, I don't think it's necessary to formalize it for this PR. Maybe in another one, but it's a very niche function not used for anything else.

yanivagman · 2023-06-19T11:08:08Z

pkg/bufferdecoder/eventsreader.go

 		res = append(res, byte(char))
 		err = decoder.DecodeInt8(&char)
 		if err != nil {
 			return "", errfmt.Errorf("error reading null terminated string: %v", err)
 		}
 	}
 	res = bytes.TrimLeft(res[:], "\000")
+	decoder.cursor += max - count // move cursor to end of buffer


We never access the cursor directly in this file but always using the DecodeXXX decoder methods.
Maybe we should consider converting this function to be a decoder method?

Well then the package boundary may be wrong. I saw these function as utils which shouldn't be in the decoder itself but do have access to it. But maybe it should be added to the decoder as you suggest.

pkg/bufferdecoder/protocol_test.go

pkg/bufferdecoder/protocol.go

pkg/ebpf/c/common/buffer.h

pkg/ebpf/c/tracee.bpf.c

pkg/ebpf/c/types.h

yanivagman · 2023-06-20T12:56:28Z

pkg/ebpf/events_enrich.go

@@ -89,7 +89,7 @@ func (t *Tracee) enrichContainerEvents(ctx gocontext.Context, in <-chan *trace.E
 				cgroupId := uint64(event.CgroupID)
 				// CgroupMkdir: pick EventID from the event itself
 				if eventID == events.CgroupMkdir {
-					cgroupId, _ = parse.ArgVal[uint64](event, "cgroup_id")
+					cgroupId, _ = parse.ArgVal[uint64](event.Args, "cgroup_id")
 				}
 				// CgroupRmdir: clean up remaining events and maps
 				if eventID == events.CgroupRmdir {


Is the below code related to event enrichment of cgroup mkdir/rmdir or should we move it to be handled in the control plane?

We could add an enrich call at the end of the control plane logic so the code here could take less time (since enrichment would already be done), that's a good idea.
But the change done here is just to align with the parse.ArgVal function signature change.

By starting the EnrichCgroupInfo sooner (in the control plane), we will help the pipeline (that might receive same related cgroupid events a bit later because of its channel buffer). I think its a good idea.

Added now, will test to see if it works right, please review it as well @rafaeldtinoco.

So if we now enrich using the control plane messages - do we still need to perform the code below (currently the lines below are not changed in this PR)

pkg/ebpf/events_processor.go

pkg/events/parse/params_test.go

rafaeldtinoco · 2023-06-21T07:21:28Z

I want to review this one pls.

pkg/ebpf/controlplane/controller.go

pkg/ebpf/tracee.go

pkg/ebpf/c/maps.h

rafaeldtinoco

Left some minor comments but LGTM, its a nice change, thank you!

NDStrahilevitz · 2023-06-25T10:40:29Z

@yanivagman @rafaeldtinoco I believe i've addressed the problems raised in the PR relevant to its scope, please re-review.

yanivagman

LGTM,
we can potentially remove more logic from the events pipeline

yanivagman · 2023-06-25T14:10:30Z

pkg/ebpf/events_enrich.go

@@ -89,7 +89,7 @@ func (t *Tracee) enrichContainerEvents(ctx gocontext.Context, in <-chan *trace.E
 				cgroupId := uint64(event.CgroupID)
 				// CgroupMkdir: pick EventID from the event itself
 				if eventID == events.CgroupMkdir {
-					cgroupId, _ = parse.ArgVal[uint64](event, "cgroup_id")
+					cgroupId, _ = parse.ArgVal[uint64](event.Args, "cgroup_id")
 				}
 				// CgroupRmdir: clean up remaining events and maps
 				if eventID == events.CgroupRmdir {


So if we now enrich using the control plane messages - do we still need to perform the code below (currently the lines below are not changed in this PR)

yanivagman · 2023-06-25T14:17:10Z

Thanks @NDStrahilevitz, well done!
I think the control plane will be a very good addition to Tracee

rafaeldtinoco · 2023-06-26T13:18:32Z

@NDStrahilevitz Im re-checking with your changes, and will +1 soon.

rafaeldtinoco · 2023-06-26T15:31:02Z

@NDStrahilevitz needs rebasing.

Take arguments array directly instead of the entire event.

The Control Plane's intention is to process "signal programs" which would previously depend on the event submission success. Instead, a separate buffer and slimmer type is used for these critical event processes.

itaysk · 2023-06-27T06:16:01Z

Is there a reason the last commit is separate (fix id override)?

NDStrahilevitz · 2023-06-27T08:19:26Z

Is there a reason the last commit is separate (fix id override)?

Because it was a bug that I think existed before the features introduced here, and I just found it while testing it with container enrichment. So I decided to include it in a separate commit and fix it opportunistically as part of the PR.

itaysk · 2023-06-27T09:28:23Z

makes sense, thanks

itaysk · 2023-06-27T09:33:55Z

In this case the PR should have closed the bug report issue in addition to the feature issue (ok to close tow issues with one PR) and that bug issue should have been included in the milestone. Not a huge deal just taking the learning opportunity
cc @yanivagman

NDStrahilevitz added area/ebpf kind/feature backport/v0.17.0 labels Jun 14, 2023

NDStrahilevitz requested review from rafaeldtinoco and yanivagman June 14, 2023 12:05

NDStrahilevitz self-assigned this Jun 14, 2023

github-actions bot added area/events area/testing labels Jun 14, 2023

NDStrahilevitz force-pushed the control_plane_events branch 4 times, most recently from 7d1ddd4 to 29a8fed Compare June 15, 2023 12:28

yanivagman requested changes Jun 20, 2023

View reviewed changes