Now we have more sophisticated (and compact) implementation in ipftrace2 repository. Please check it as well.
ipftrace is a simple function tracer for Linux networking code with flow based filtering. It is similar to the ftrace in some sense but, you can trace which flow have gone through which functions inside the kernel which is usually more important information for the network people than which functions are called information provided by ftrace.
Usage: ipftrace [OPTIONS] MANIFEST
Track the journey of the packets in Linux network stack
Options:
-iv, --ipversion [4|6] Specify IP version
-s, --saddr TEXT Specify IP source address
-d, --daddr TEXT Specify IP destination address
-p, --proto TEXT Specify protocol
-sp, --sport TEXT Specify source port number
-dp, --dport TEXT Specify destination port number
-m, --module TEXT Specify custom match module name
-r, --regex TEXT Filter the function names by regex
-l, --length INTEGER Specify the length of function trace
-ls, --list-func List available functions
--help Show this message and exit.
- Python3.7 or above
- BCC
$ git clone https://github.com/YutaroHayakawa/ipftrace
$ cd ipftrace
$ sudo ./scripts/ipftrace-docker -p ICMP examples/manifest/generic.yaml
ipftrace-docker is a wrapper script for docker run. Since the ipftrace container requires some boring things to make it work (like expose the Linux source to the container, make container privileged, uploading manifest file to the container and so on), this script does it for you.
$ git clone https://github.com/YutaroHayakawa/ipftrace
$ cd ipftrace
$ sudo pip3 install .
$ sudo ipftrace -p ICMP examples/manifest/generic.yaml
Trace the ping ICMP packets
# ipftrace -p ICMP examples/manifest/5.4.0-rc7-btf.yaml
<...>
ICMP 10.231.244.75 -> 10.128.218.64
Time Stamp Function
495406078818070 nf_hook_slow
495406078929728 ip_output
495406078949359 nf_hook_slow
495406078970951 dev_queue_xmit
495406078986098 netdev_core_pick_tx
495406079008937 sch_direct_xmit
495406079024885 validate_xmit_skb_list
495406079036226 netif_skb_features
495406079057803 passthru_features_check
495406079071811 skb_network_protocol
495406079091507 dev_queue_xmit_nit
495406079106655 skb_clone
495406079132195 consume_skb
495406079143957 skb_release_head_state
495406079157152 skb_clone_tx_timestamp
495406079179885 skb_to_sgvec
ICMP 10.128.218.64 -> 10.231.244.75
Time Stamp Function
495406112568847 inet_gro_receive
495406112662367 skb_defer_rx_timestamp
495406112712251 consume_skb
495406112732787 ip_rcv
495406112784692 nf_hook_slow
495406112811560 nf_ip_checksum
495406112837230 __skb_checksum_complete
495406112856814 __skb_checksum
495406112883108 ip_route_input_noref
495406112905093 ip_route_input_rcu
495406112928973 fib_validate_source
495406112952321 ip_local_deliver
495406112972174 nf_hook_slow
495406112999152 ip_protocol_deliver_rcu
495406113020418 raw_local_deliver
495406113047608 skb_clone
495406113074430 raw_rcv
495406113109356 ipv4_pktinfo_prepare
495406113132611 sock_queue_rcv_skb
495406113169237 sk_filter_trim_cap
495406113198364 security_sock_rcv_skb
495406113230150 __sock_queue_rcv_skb
495406113260665 icmp_rcv
495406113289802 ping_rcv
495406113368263 kfree_skb
Trace the GSO behavior with custom module
# ipftrace -p TCP -d 10.231.206.32 -dp 8000 -m examples/module/gso.py examples/5.4.0-rc7-btf.yaml
<...>
TCP 10.231.244.75:33696 -> 10.231.206.32:8000
Time Stamp Function Custom Data
490440712480226 nf_hook_slow (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712519497 ip_output (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712538213 nf_hook_slow (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712549567 skb_gso_validate_network_len (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712566096 neigh_resolve_output (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712578933 eth_header (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712594839 dev_queue_xmit (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712604856 netdev_core_pick_tx (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712614715 sch_direct_xmit (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712623331 validate_xmit_skb_list (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712631858 netif_skb_features (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712641404 passthru_features_check (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712653638 skb_network_protocol (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712661015 __skb_gso_segment (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712673966 skb_mac_gso_segment (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712683431 skb_network_protocol (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712688616 inet_gso_segment (len: 2101 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712705180 tcp_gso_segment (len: 2081 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712717131 skb_network_protocol (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712750240 consume_skb (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712759857 skb_release_head_state (len: 2115 gso_size: 1448 gso_segs: 2 gso_type: SKB_GSO_TCPV4)
490440712772181 dev_queue_xmit_nit (len: 1514 gso_size: 0 gso_segs: 0 gso_type: )
ipftrace needs the YAML file called manifest file. Which seems like below.
functions:
- name: ip_rcv
skb_pos: 1
- name: ip_local_out
skb_pos: 3
- name: ip_output
skb_pos: 3
- name: ip_local_deliver_finish
skb_pos: 3
egress: true
- name: ip_forward
skb_pos: 1
- name: ip_route_input_noref
skb_pos: 1
<...>It contains following informtations.
name: The symbol name of the function to traceskb_pos: The position of the struct sk_buff* in the function argumentsegress: (optional) Indicates the end of the function chain
As for the name, you can specify any kernel functions which takes struct sk_buff* as an argument (the symbol must be exported). In addition to the function name, we need skb_pos which indicates the position of the struct sk_buff* in the argument list of the function (the first argument is position 1).
egress indicates the end of the function tracing. The list of the functions for the flow will be terminated and displayed when the packet reaches to the function annotated with egress: true. You need to find appropriate egress function to get correct result from ipftrace. This is probably a most difficult part of the ipftrace.
As for today, you need to write the manifest file by your self. But the grows of the BTF (BPF Type Format) infrastructure will greatly reduce the time to write it (but you still need to find the egress function). Currently we have an example manifest in the examples. Feel free to use it.
We also have useful experimental script to generate the manifest from BTF information in vmlinux. Please checkout scripts/btf_to_manifest.py.
It uses eBPF + kprobe for attaching the tracing programs to the kernel, parse the packet in the kprobe, filter out the unneccesary packets and output some log through perf. That's it.
- ipftrace cannot trace the function with skb_pos > 4 due to the limitation of the eBPF.
- We recommend you to mark
kfree_skbwithegress: truethis will catch the case which netfilter drops the packet. - ipftrace depends on the skb->network_header and skb->protocol to determine the IP address of the packet. So, if these informations are invalid, it cannot trace the function correctly. Due to this, ipftrace usually cannot trace the functions belongs to the "higher" layer than IP for TX path and "lower" layer than IP for RX path.