ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

rootfs · 2023-07-24T15:21:28Z

this reduces the bpf call overhead from 1352ns to 99ns

without this fix

# sysctl -w kernel.bpf_stats_enabled=1
# bpftool prog show |grep kepler_trace |awk '{print $(NF-2)/$NF}'
1352.07

with this fix

# sysctl -w kernel.bpf_stats_enabled=1
# bpftool prog show |grep kepler_trace |awk '{print $(NF-2)/$NF}'
99.0167

make bcc work
make libbpf able to set sampling rate by calling InitilizeGlobalVar
evaluate trade-off between overhead and prediction accuracy

rootfs · 2023-07-24T18:58:08Z

depend on #824

marceloamaral

Looks great, I only concerned about the sample rate.

marceloamaral · 2023-07-25T07:17:49Z

bpfassets/libbpf/src/kepler.bpf.c

@@ -44,6 +44,9 @@ BPF_ARRAY(cache_miss, u64, NUM_CPUS);
 // cpu freq counters
 BPF_ARRAY(cpu_freq_array, u32, NUM_CPUS);

+int sample_rate = 1000;
+int counter = 1000;


Shouldn't it be:

int sample_rate = SAMPLE_RATE; int counter = SAMPLE_RATE;

libbpf is pre-compiled, we cannot use compilation flags. In this case, we have to global variable to set it. I opened #824 so I can use InitGlobalVar function

Got it!
If we need to hard code now, let's use a smaller value.

marceloamaral · 2023-07-25T07:18:08Z

bpfassets/perf_event/perf_event.c

@@ -71,6 +71,11 @@ BPF_ARRAY(cache_miss, u64, NUM_CPUS);
 // cpu freq counters
 BPF_ARRAY(cpu_freq_array, u32, NUM_CPUS);

+#ifndef SAMPLE_RATE
+#define SAMPLE_RATE 1000


Isn't skipping 1000 sample to extreme?
Did you try 10 and 100?

yes, 10 or 100 does have some reduction from 1000ns to 300ns.

marceloamaral · 2023-07-25T07:19:06Z

pkg/config/config.go

@@ -80,6 +80,7 @@ var (
 	BindAddressKey               = "BIND_ADDRESS"
 	CPUArchOverride              = getConfig("CPU_ARCH_OVERRIDE", "")
 	MaxLookupRetry               = getIntConfig("MAX_LOOKUP_RETRY", defaultMaxLookupRetry)
+	BPFSampleRate                = getIntConfig("BPF_SAMPLE_RATE", 1000)


Shouldn't we use smaller values?

sure, let's have some stats first so users know what to choose and start from small values as default.

rootfs · 2023-07-26T14:14:05Z

test environment

RHEL 8.6
Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz

kepler command

BPF_SAMPLE_RATE=1000 _output/bin/linux_amd64/kepler

ebpf calculation

bpftool prog show |grep kepler_trace |awk '{print $(NF-2)/$NF}'

result

sample frequency	per call time (ns)
0	1239
1	797
10	300
50	152
100	129
1000	93

@marceloamaral @sunya-ch what's your recommended default sample rate?

eklee15 · 2023-07-26T14:31:02Z

Thanks for sharing it. This is great!

marceloamaral · 2023-07-27T04:33:09Z

@rootfs, for now, let's set the default value to 10 since we're uncertain about the consequences of skipping samples. Moreover, using 10 seems to bring significant improvements. Once we conduct further analysis, we can increasing this value.

rootfs · 2023-07-27T12:15:58Z

@marceloamaral sure, let default to 10. One more question, when we sample the ebpf calls, should we also extrapolate the metrics (cpu time, cpu instructions, etc) as well?

eklee15 · 2023-07-27T12:42:18Z

Just created a discussion
#836

rootfs · 2023-07-28T19:25:57Z

@eklee15 @marceloamaral @sunya-ch Let's disable sampling for now until we have a resolution on #836

sthaha · 2023-07-31T04:59:41Z

bpfassets/perf_event/perf_event.c

+        if (sample_counter_value > 0) {
+            if (*sample_counter_value > 0) {
+                (*sample_counter_value)--;
+                return 0;
+            }
+        }


New to this so please excuse if this suggestion looks stupid . Would this work?

Suggested change

if (sample_counter_value > 0) {

if (*sample_counter_value > 0) {

(*sample_counter_value)--;

return 0;

}

}

if (sample_counter_value && *sample_counter_value > 0) {

(*sample_counter_value)--;

return 0;

}

sthaha · 2023-07-31T23:09:43Z

pkg/collector/metric/container_metric.go

+	if c == nil {
+		return
+	}


This shouldn't be done we must expect the receiver to be initialised at all time. If it is not the case, then it is a programming error and must panic. By returning we are only shadowing a logical error.

sthaha · 2023-07-31T23:11:21Z

pkg/model/estimator/local/ratio_process.go

@@ -74,6 +74,10 @@ func getProcessResUsage(process *collector_metric.ProcessMetrics, usageMetric st
 // UpdateProcessComponentEnergyByRatioPowerModel calculates the process energy consumption based on the energy consumption of the container that contains all the processes
 func UpdateProcessComponentEnergyByRatioPowerModel(processMetrics map[uint64]*collector_metric.ProcessMetrics, containerMetrics *collector_metric.ContainerMetrics, component, usageMetric string, wg *sync.WaitGroup) {
 	defer wg.Done()
+	if containerMetrics == nil || processMetrics == nil {
+		klog.V(5).Infoln("containerMetrics or processMetrics is nil")


How about we return an error if the arguments aren't initialised properly?

marceloamaral · 2023-08-25T09:57:15Z

bpfassets/libbpf/src/kepler.bpf.c

    u32 next_pid = ctx->next_pid; // the new pid that is to be scheduled
+=======
+>>>>>>> 2c38bc2dc7b9aca85374c16c96db555f16784169


merge left over

the libbpf module conflict is hard to merge, now my git log gets quite messy. Will open a different PR.

author Huamin Chen <hchen@redhat.com> 1690211838 -0400 committer Huamin Chen <hchen@redhat.com> 1692968086 -0400 ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls Signed-off-by: Huamin Chen <hchen@redhat.com>

Signed-off-by: Huamin Chen <hchen@redhat.com>

marceloamaral

@rootfs, could you also modify the approach for dropping samples? Instead of calculating the percentage, could we implement a method that aggregates a counter and drops samples accordingly?

For instance, after collecting 99 samples, we could skip the next 1 sample, effectively skipping 1% of the total. Then, if we decide to skip 10 samples, it would translate to a 10% reduction. To achieve this, we would need a mechanism to skip 'y' samples after gathering 'x' samples. This would provide us with the flexibility to adjust the dropout rate as needed.

rootfs · 2023-09-13T13:58:01Z

the rebase was not successful, will reopen another PR

rootfs requested review from marceloamaral and sunya-ch July 24, 2023 15:22

This was referenced Jul 24, 2023

Latency increase on kernel rt #671

Closed

eBPF scalability improvement #668

Closed

rootfs added this to the kepler-release-0.6 milestone Jul 24, 2023

marceloamaral requested changes Jul 25, 2023

View reviewed changes

rootfs force-pushed the ebpf-sampling branch from 030d8ce to f94f610 Compare July 26, 2023 13:49

rootfs changed the title ~~[WIP] ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls~~ ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls Jul 26, 2023

rootfs force-pushed the ebpf-sampling branch from 85e9d60 to 9cb4307 Compare July 28, 2023 19:12

sthaha reviewed Jul 31, 2023

View reviewed changes

rootfs force-pushed the ebpf-sampling branch from 2c38bc2 to 197d0ab Compare August 23, 2023 15:05

marceloamaral reviewed Aug 25, 2023

View reviewed changes

rootfs added 3 commits August 25, 2023 08:55

parent 9b82f25

dc036ec

author Huamin Chen <hchen@redhat.com> 1690211838 -0400 committer Huamin Chen <hchen@redhat.com> 1692968086 -0400 ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls Signed-off-by: Huamin Chen <hchen@redhat.com>

parent 9b82f25

49bb3fb

author Huamin Chen <hchen@redhat.com> 1690211838 -0400 committer Huamin Chen <hchen@redhat.com> 1692968086 -0400 ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls Signed-off-by: Huamin Chen <hchen@redhat.com>

rebase

2a7a37d

Signed-off-by: Huamin Chen <hchen@redhat.com>

rootfs force-pushed the ebpf-sampling branch from 825e7ec to 2a7a37d Compare August 25, 2023 12:58

marceloamaral requested changes Aug 28, 2023

View reviewed changes

rootfs mentioned this pull request Sep 13, 2023

ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #928

Merged

rootfs closed this Sep 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

rootfs commented Jul 24, 2023 •

edited

Loading

rootfs commented Jul 24, 2023

marceloamaral left a comment

marceloamaral Jul 25, 2023

rootfs Jul 25, 2023

marceloamaral Jul 26, 2023

marceloamaral Jul 25, 2023

rootfs Jul 25, 2023

marceloamaral Jul 25, 2023

rootfs Jul 25, 2023

rootfs commented Jul 26, 2023 •

edited

Loading

eklee15 commented Jul 26, 2023

marceloamaral commented Jul 27, 2023

rootfs commented Jul 27, 2023

eklee15 commented Jul 27, 2023

rootfs commented Jul 28, 2023

sthaha Jul 31, 2023

sthaha Jul 31, 2023

sthaha Jul 31, 2023

marceloamaral Aug 25, 2023

rootfs Aug 25, 2023

marceloamaral left a comment •

edited

Loading

rootfs commented Sep 13, 2023

ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

ebpf: reducing ebpf call overhead by using sampling instead of tracing every calls #823

Conversation

rootfs commented Jul 24, 2023 • edited Loading

without this fix

with this fix

rootfs commented Jul 24, 2023

marceloamaral left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootfs commented Jul 26, 2023 • edited Loading

test environment

kepler command

ebpf calculation

result

eklee15 commented Jul 26, 2023

marceloamaral commented Jul 27, 2023

rootfs commented Jul 27, 2023

eklee15 commented Jul 27, 2023

rootfs commented Jul 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marceloamaral left a comment • edited Loading

Choose a reason for hiding this comment

rootfs commented Sep 13, 2023

rootfs commented Jul 24, 2023 •

edited

Loading

rootfs commented Jul 26, 2023 •

edited

Loading

marceloamaral left a comment •

edited

Loading