RFC: Add skboutput function

[olsajiri]

heya,
please take as RFC, it's by no means final code ;-)
I just need to hear some feedback on wether you guys think this is useful

I tried to put together something we discussed with Arnaldo and Jiri Benc
on LPC - storing skb_buff via bpf helper from any probe with 'skb' argument

the new bpftrace function has following syntax:

skboutput(file, skb, caplen)

it will store packet from 'skb' object to tcpdump 'file' and limit the size by 'caplen', for example:

# bpftrace -e 'kfunc:napi_gro_receive { skboutput("krava.pcap", args->skb, 1500);  }'
Attaching 1 probe...
...

# tcpdump -n -r ./krava.pcap
reading from file ./krava.pcap, link-type RAW (Raw IP)
dropped privs to tcpdump
02:43:07.069621 unknown ip 0
02:43:07.069640 unknown ip 0
02:43:07.084865 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 3619840828, win 1867, options [nop,nop,TS val 862106286 ecr 226183539], length 0
02:43:07.097883 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 53, win 1867, options [nop,nop,TS val 862106298 ecr 226183554], length 0
02:43:07.110942 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 105, win 1867, options [nop,nop,TS val 862106311 ecr 226183567], length 0
02:43:07.125958 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 157, win 1867, options [nop,nop,TS val 862106326 ecr 226183581], length 0
02:43:07.126282 unknown ip 0
02:43:07.138008 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 209, win 1867, options [nop,nop,TS val 862106339 ecr 226183596], length 0
02:43:07.138969 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 261, win 1867, options [nop,nop,TS val 862106339 ecr 226183596], length 0
02:43:07.151006 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 313, win 1867, options [nop,nop,TS val 862106351 ecr 226183608], length 0
02:43:07.152040 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 365, win 1867, options [nop,nop,TS val 862106352 ecr 226183609], length 0
02:43:07.153369 unknown ip 0
02:43:07.164084 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 417, win 1867, options [nop,nop,TS val 862106364 ecr 226183621], length 0
02:43:07.165010 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 469, win 1867, options [nop,nop,TS val 862106365 ecr 226183622], length 0
02:43:07.165020 IP 10.40.192.62.40032 > 10.37.152.221.ssh: Flags [.], ack 521, win 1867, options [nop,nop,TS val 862106366 ecr 226183623], length 0
...

you can use skboutput function from multiple places/probes with different files

it seems useful for debuging network stuff, but what do I know..? ;-)
I haven't got too much positive feedback yet to continue on with this

thanks for any thoughts on this

[fbs]

I like it. It is a useful thing to have and easier than digging into a skb straight away.

Being able to do bpftrace -e '{ magic($skb) } | tcpdump -r - would be useful too.

[danobi]

Does libpcap handle changes to struct sk_buff? skb layout probably doesn't change that often but I still wonder.

[olsajiri]

Does libpcap handle changes to struct sk_buff? skb layout probably doesn't change that often but I still wonder.

so there's skboutput bpf kernel helper that gets you skb data and store it to perf map/buffer,
so it's quite easy code, similar in the process as printf/syscat/cat functions

the pcap writer on the user space side is quite basic at the moment, but I've got few pointers where to get some examples of how to make it better and more robust

[liuhangbin]

Hi Jiri,

Is there any progress on this RFC? I feel this feature is useful for networking debug. And it could be used on my bpftrace version of dropwath.

Thanks
Hangbin

[olsajiri]

Hi Jiri,

Is there any progress on this RFC? I feel this feature is useful for networking debug. And it could be used on my bpftrace version of dropwath.

heya,
@liuhangbin the code is rebased, but I think we need small kernel change to work more user friendly, because now it drops packets that are bigger than caplen, and I think we could add flag to skb_output not to drop them but cut them

also it'd be great if you paste in here some of your usage examples so I have some case when asking for the kernel change

thanks,
jirka

[liuhangbin]

@olsajiri , thanks for your works. I will clone the code and try it first. After that, I will paste my testing script here.

[liuhangbin]

Basically, I would use this feature for bpftrace version of dropwatch . The current version only implements the dropwatch feature. But for bpftrace version of dwdump, the skboutput function is needed.

And an example use case looks like
bpftrace -e 't:skb:kfree_skb { skboutput("kfree_skb.pcap", args->skbaddr, 1500); }'

[fbs]

we keep these in src/ast/async_event_types.h so theyre all defined in one place

[xh4n3]

@olsajiri Hi Jiri, are you still working on this feature? It seems very helpful for digging network issues.
I've rebased your commits to the latest branch on my machine, and it works. If you need some help on the remaining job, I can take a look at it.

If you use skboutput("krava.pcap", args->skb, args->skb->len - 4);, no packet will be dropped, only some truncated by less than 4 bytes. Looks like an alignment issue, I will dig into it.

[olsajiri]

@olsajiri Hi Jiri, are you still working on this feature? It seems very helpful for digging network issues. I've rebased your commits to the latest branch on my machine, and it works. If you need some help on the remaining job, I can take a look at it.

If you use skboutput("krava.pcap", args->skb, args->skb->len - 4);, no packet will be dropped, only some truncated by less than 4 bytes. Looks like an alignment issue, I will dig into it.

@xh4n3 great, feel free to rebase/fix/push it ;-) I dropped that because I thought there's no interest in this,
not sure when I could get to it, so it'll be faster if you can do that, thanks