Perf event ring buffer full, dropped 16 samples

[BurlyLuo]

root@bpf2:~# ./pwru --filter-proto icmp  --output-tuple
2022/01/21 04:44:15 Attaching kprobes...
577 / 577 [---------------------------------------------------------------------------------------------------------------------------------] 100.00% 34 p/s
2022/01/21 04:44:32 Attached (ignored 13)
2022/01/21 04:44:32 Listening for events..
               SKB         PROCESS                     FUNC        TIMESTAMP
0xffff96f353734300          [ping]           __ip_local_out   52052971636335 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]                ip_output   52052971655120 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]             nf_hook_slow   52052971658827 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping] __cgroup_bpf_run_filter_skb   52052971662013 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]     neigh_resolve_output   52052971666902 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]       __neigh_event_send   52052971669217 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]               eth_header   52052971673094 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]                 skb_push   52052971676681 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]           dev_queue_xmit   52052971678785 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      netdev_core_pick_tx   52052971681881 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]       netif_skb_features   52052971684686 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]  passthru_features_check   52052971686700 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]     skb_network_protocol   52052971689855 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]       validate_xmit_xfrm   52052971693733 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      dev_hard_start_xmit   52052971695646 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]   skb_clone_tx_timestamp   52052971699343 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]        __dev_forward_skb   52052971701788 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]         skb_scrub_packet   52052971704002 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]           eth_type_trans   52052971706256 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]                 netif_rx   52052971708560 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]     tcf_classify_ingress   52052971727115 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      skb_ensure_writable   52052971788730 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      skb_ensure_writable   52052971790414 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      skb_ensure_writable   52052971791676 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      skb_ensure_writable   52052971792948 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]          skb_do_redirect   52052971820830 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]                   ip_rcv   52052971825740 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]               sock_wfree   52052971829246 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]     ip_route_input_noref   52052971832592 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]       ip_route_input_rcu   52052971834256 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]      fib_validate_source   52052971839766 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]         ip_local_deliver   52052971843844 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]  ip_protocol_deliver_rcu   52052971846338 10.0.1.30:0->10.0.1.23:0(icmp)
0xffff96f353734300          [ping]        raw_local_deliver   52052971848362 10.0.1.30:0->10.0.1.23:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 3 samples
0xffff96f353734300          [ping]     fib_compute_spec_dst   52052971858761 10.0.1.30:0->10.0.1.23:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 5 samples
0xffff96f353734b00          [ping]     neigh_resolve_output   52052971871024 10.0.1.23:0->10.0.1.30:0(icmp)
### 2022/01/21 04:44:35 Perf event ring buffer full, dropped 16 samples
0xffff96f353734300       [<empty>]   skb_release_head_state   52052971885111 10.0.1.30:0->10.0.1.23:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 1 samples
0xffff96f353734b00       [<empty>]      skb_ensure_writable   52052971919565 10.0.1.23:0->10.0.1.30:0(icmp)
0xffff96f353734b00       [<empty>]      skb_ensure_writable   52052971921188 10.0.1.23:0->10.0.1.30:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 2 samples
0xffff96f353734b00       [<empty>]          skb_do_redirect   52052971938721 10.0.1.23:0->10.0.1.30:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 13 samples
0xffff96f353734300       [<empty>]       sk_filter_trim_cap   52052971953108 10.0.1.23:0->10.0.1.30:0(icmp)
2022/01/21 04:44:35 Perf event ring buffer full, dropped 9 samples
0xffff96f353734300       [<empty>]        skb_free_datagram   52052971971122 10.0.1.23:0->10.0.1.30:0(icmp)
^C2022/01/21 04:44:43 Received signal, exiting program..
root@bpf2:~#


root@bpf1:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
cni-bpz2q   1/1     Running   0          14h   10.0.1.23    bpf2   <none>           <none>
cni-hx8gz   1/1     Running   0          14h   10.0.0.159   bpf1   <none>           <none>
same        1/1     Running   0          79s   10.0.1.30    bpf2   <none>           <none>
root@bpf1:~# kubectl exec -it same -- ping -c 1 10.0.1.23
PING 10.0.1.23 (10.0.1.23): 56 data bytes
64 bytes from 10.0.1.23: seq=0 ttl=63 time=0.269 ms

--- 10.0.1.23 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.269/0.269/0.269 ms

How about the issue when monitor the two pods communicate to each other on the same node.

2022/01/21 04:44:35 Perf event ring buffer full, dropped 3 samples

which is it means that the ring buffer is full？
ENV：

root@bpf2:~# uname -a
Linux bpf2 5.10.0-051000-generic #202012132330 SMP Sun Dec 13 23:33:36 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@bpf2:~#

[aditighag]

You can try increasing the per cpu buffer size - http://github.com/brb/pwru/blob/70487475cbc00e66375b724febd223290bc686f9/main.go#L118-L118, depending on the available memory. Please note that this will drive up the memory usage on the host though. Alternatively, you can try tweaking the filters to minimize the number of events.

[BurlyLuo]

depending on the available memory. Please note that this will drive up the memory usage on the host though. Alternatively, you can try tweaking the filters to minimize the number of events.

yes. with 8G memory in this node. is it enough?
and when i run the pwru, watch the free -mh , it is not increase so mush.
but i run this test in the vmware workstation env.

[brb]

yes. with 8G memory in this node. is it enough?

Yes, it should be more than enough.

Have you tried to increase the buffer size as suggested above?

[BurlyLuo]

Have you tried to increase the buffer size as suggested above?

Yes.Unfortunately, the address can't be accessed now.

[brb]

Could you elaborate more? What error did you get?

[BurlyLuo]

Could you elaborate more? What error did you get?

404 now.

[brb]

404 now.

Em? I'm not following.

[BurlyLuo]

Yes, but i really get this 404 error here.
@brb would you please share the web snapshot or share the methord here to increase the buffer size?
thanks a lot.

[brb]

Ah, got you. The actual link supposed to be https://github.com/cilium/pwru/blob/master/main.go#L145.

[vincentmli]

https://github.com/cilium/pwru/blob/master/main.go#L145

@brb, that line is rd, err := perf.NewReader(events, os.Getpagesize()), you mean increase the page size and how?

[vincentmli]

never mind, I got what you mean, should have read the code :-)

// NewReader creates a new reader with default options.
//      
// array must be a PerfEventArray. perCPUBuffer gives the size of the
// per CPU buffer in bytes. It is rounded up to the nearest multiple
// of the current page size.
func NewReader(array *ebpf.Map, perCPUBuffer int) (*Reader, error) {
        return NewReaderWithOptions(array, perCPUBuffer, ReaderOptions{})
} 

[BurlyLuo]

Yes.
main.go

        bar.Finish()
        log.Printf("Attached (ignored %d)\n", ignored)
        // modify syscall to customize
        rd, err := perf.NewReader(events, 16384)   
        if err != nil {
                log.Fatalf("Creating perf event reader: %s", err)
        }
        defer rd.Close()

[brb]

@vincentmli @BurlyLuo maybe you could make it configurable via a flag and send a PR?

[BurlyLuo]

@vincentmli @BurlyLuo maybe you could make it configurable via a flag and send a PR?

Yeah, i will sned a PR .