Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tetragon: Add buffer between perf reader and events processing code #593

Merged
merged 2 commits into from Sep 6, 2023

Conversation

olsajiri
Copy link
Contributor

@olsajiri olsajiri commented Dec 13, 2022

Adding extra layer between perf ring buffer reader and event
processing by sensor code.

This way we keep the perf reader fast and we won't get lost events
if sensor processing code have a hiccup and takes longer.

@olsajiri olsajiri force-pushed the events_buffer branch 6 times, most recently from 8c44927 to cd655e6 Compare December 19, 2022 18:11
@olsajiri olsajiri force-pushed the events_buffer branch 4 times, most recently from f5e3f16 to af73cf1 Compare August 16, 2023 13:05
@olsajiri
Copy link
Contributor Author

olsajiri commented Aug 17, 2023

this change makes big difference in being able to get and process more events from kernel

events buffer stats - results-1.pdf

@olsajiri olsajiri marked this pull request as ready for review August 17, 2023 07:44
@olsajiri olsajiri requested a review from a team as a code owner August 17, 2023 07:44
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! LGTM!

The only concern I have is about naming. How about using Perf in the identifiers (gave a few examples) to make it more clear?

pkg/option/config.go Outdated Show resolved Hide resolved
pkg/observer/observer.go Show resolved Hide resolved
cmd/tetragon/flags.go Outdated Show resolved Hide resolved
Adding extra layer between perf ring buffer reader and event
processing by sensor code.

Basically it's new channel (default size 65k) that is written by
perf ringbuffer reader and read by new go routine that processes
these events.

This way we keep the perf reader fast and we won't get lost events
if sensor processing code have a hiccup and takes longer.

When the eventsQueue channel is full we drop the event. It's configurable
by new --rb-queue-size option, the default is 65k.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Adding following metric counters for events buffer queue:

  ringbufqueuemetrics.Received - number of records received from perf event reader go routine
  ringbufqueuemetrics.Lost     - number of records lost because the RB Queue channel was full

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
@olsajiri olsajiri added the release-note/minor This PR introduces a minor user-visible change label Sep 1, 2023
@jrfastab
Copy link
Contributor

jrfastab commented Sep 6, 2023

OK lets try it.

@jrfastab jrfastab merged commit 9ed09fa into cilium:main Sep 6, 2023
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/minor This PR introduces a minor user-visible change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants