Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize init program #3923

Merged
merged 5 commits into from
Mar 24, 2024
Merged

Conversation

yanivagman
Copy link
Collaborator

1. Explain what the PR does

perf: use lazy program data initialization

We currently initialize all task context related fields on the beginning
of our BPF programs. Reading all the relevant fields takes some CPU
cycles although these fields are not always used. In many cases, for
example when filters are applied, the event is not sent to userspace and
is just dropped without any side effects.

Reading only the essential fields and leaving all other fields
initialization to the submit stage will avoid wasting these cycles, and
will also allow us to perform some of the event enrichment in userspace
instead (not part of this PR).

To do this change, we have to remove the matched_policies which were
cached in task_info since we can't check if "context changed" anymore.
Considering that the average user will only use one or two scope filters
(e.g. container id or binary name), computing the scope on every run is
not a big overhead - running BPF statistics with and without the
change verified this is indeed the case, and with the change the
performance is even slightly better.

Although this change does not introduce a visible performance gain for
most of the events, it is clear for some other specific event that it
does. For example, the hidden_inode event which attach a program to the
filldir64 function has a 50% performance gain.

2. Explain how to test it

3. Other comments

Copy link
Member

@geyslan geyslan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great!

I've done the first pass and did some questions.

pkg/ebpf/c/common/buffer.h Outdated Show resolved Hide resolved
pkg/ebpf/c/common/context.h Outdated Show resolved Hide resolved
pkg/ebpf/c/common/context.h Show resolved Hide resolved
pkg/ebpf/c/common/context.h Show resolved Hide resolved
pkg/ebpf/c/common/context.h Show resolved Hide resolved
pkg/ebpf/c/common/filtering.h Outdated Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Show resolved Hide resolved
Copy link
Collaborator

@NDStrahilevitz NDStrahilevitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will run this full branch on my cluster before giving a +1. Had some questions too.

Have to note on the filldir64 case in particular: I believe the case there isn't this optimization in particular. Note that we have a quick terminating conditional based on a kprobe context argument (the inode_number == 0 check). We could have just as well moved it to the head of the function before the init,trace,submit function trio, and had just the same effect.

pkg/ebpf/c/maps.h Show resolved Hide resolved
pkg/ebpf/c/common/context.h Show resolved Hide resolved
pkg/ebpf/c/common/buffer.h Show resolved Hide resolved
pkg/ebpf/c/common/context.h Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Outdated Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Outdated Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Show resolved Hide resolved
pkg/ebpf/c/tracee.bpf.c Show resolved Hide resolved
@yanivagman yanivagman force-pushed the optimize_init_program branch 2 times, most recently from d27f636 to c2ef413 Compare March 20, 2024 14:30
@geyslan
Copy link
Member

geyslan commented Mar 20, 2024

It passed locally, I can't see why it failed when triggered by the action: https://github.com/aquasecurity/tracee/actions/runs/8360692120/job/22887161311.

Anyhoo, I'm gonna replay this test in Github.

@yanivagman yanivagman force-pushed the optimize_init_program branch 4 times, most recently from b8cb2ee to 27f803f Compare March 21, 2024 20:21
init_tailcall_program_data() function was missing the initialization of
proc_info and args_buf - add them
socket_dup event depends on the arguments of the matching dup syscall.
Add this missing dependency.
@yanivagman yanivagman force-pushed the optimize_init_program branch from 27f803f to 0422a39 Compare March 21, 2024 21:34
Copy link
Member

@geyslan geyslan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great.

We currently initialize all task context related fields on the beginning
of our BPF programs. Reading all the relevant fields takes some CPU
cycles although these fields are not always used. In many cases, for
example when filters are applied, the event is not sent to userspace and
is just dropped without any side effects.

Reading only the essential fields and leaving all other fields
initialization to the submit stage will avoid wasting these cycles, and
will also allow us to perform some of the event enrichment in userspace
instead (not part of this PR).

To do this change, we have to remove the matched_policies which were
cached in task_info since we can't check if "context changed" anymore.
Considering that the average user will only use one or two scope filters
(e.g. container id or binary name), computing the scope on every run is
not a big overhead - running BPF statistics with and without the
change verified this is indeed the case, and with the change the
performance is even slightly better.

Although this change does not introduce a visible performance gain for
most of the events, it is clear for some other specific event that it
does. For example, the hidden_inode event which attach a program to the
filldir64 function has a 50% performance gain.
@yanivagman yanivagman force-pushed the optimize_init_program branch from 0422a39 to fe5f097 Compare March 23, 2024 10:21
Copy link
Collaborator

@NDStrahilevitz NDStrahilevitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance test turned out good, let's merge.

@yanivagman yanivagman merged commit f806cb4 into aquasecurity:main Mar 24, 2024
32 checks passed
@yanivagman yanivagman deleted the optimize_init_program branch March 24, 2024 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants