TEA

TEA(TC eBPF for AV) is a network emulator and a set of tools for SRS and any media systems.

Usage

We use docker to build and run codes:

docker run --privileged --rm -it ossrs/tea tc qdisc ls

Note: Please specify --privileged for tc and eBPF.

For example, build tc_stun_drop_all:

mkdir -p ~/git && cd ~/git &&
git clone https://github.com/ossrs/tea.git &&
docker run --rm -it -v $(pwd):/git -w /git/tea/tc_stun_drop_all ossrs/tea:latest make

Or start a docker in background:

mkdir -p ~/git && cd ~/git &&
docker run --privileged -d --name tea -it -v $(pwd):/git -w /git/tea ossrs/tea:latest bash &&
docker exec -it -w /git/tea/libbpf_stun_drop_all tea make

Please follow bellow examples and tools.

LIBBPF: STUN Drop All

Using libbpf to load the eBPF object to TC as clsact.

First, start a docker in background:

mkdir -p ~/git && cd ~/git &&
docker run -d --privileged --name tea -it -v $(pwd):/git -w /git/tea \
    ossrs/tea:latest bash

Next, build the eBPF program:

docker exec -it -w /git/tea/libbpf_stun_drop_all tea make

And attach eBPF bytecode to TC by:

docker exec -it -w /git/tea/libbpf_stun_drop_all tea ./libbpf_stun_drop_all

Then, start tcpdump to show packets, and using nc to send packets:

# Capture all UDP packets.
docker exec -it tea tcpdump udp -i any -X

# Start a UDP server, listen at 8000
docker exec -it tea nc -l -u 8000

# Send STUN binding request.
docker exec -it -w /git/tea/libbpf_stun_drop_all tea bash -c \
    "echo -en \$(cat binding_request.txt |tr -d [:space:]) |nc -p 55293 -w 1 -u 127.0.0.1 8000"

# Send STUN binding response.
docker exec -it -w /git/tea/libbpf_stun_drop_all tea bash -c \
    "echo -en \$(cat binding_response.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

Note: You will see the packets printed by tcpdump and nc server, before installing the eBPF TC qdisc.

All STUN packets will be dropped:

Dropping all STUN packets...
              nc-12768   [004] d....  7280.186241: bpf_trace_printk: Drop STUN packet, type=0x100, len=20480, magic=0x42a41221

For detail about TC and eBPF, please read Links: TC and Links: LIBBPF section.

LIBBPF: STUN NETEM

Using libbpf and TC netem for STUN packets only.

First, start a docker in background:

mkdir -p ~/git && cd ~/git &&
docker run -d --privileged --name tea -it -v $(pwd):/git -w /git/tea \
    ossrs/tea:latest bash

Next, build the eBPF program:

docker exec -it -w /git/tea/libbpf_stun_netem tea make

Add 3s delay for STUN packet:

docker exec -it -w /git/tea/libbpf_stun_netem tea \
    tc qdisc add dev lo root handle 1:0 prio &&
docker exec -it -w /git/tea/libbpf_stun_netem tea \
    tc qdisc add dev lo parent 1:3 handle 3:0 netem delay 3000ms &&
docker exec -it -w /git/tea/libbpf_stun_netem tea \
    tc filter add dev lo parent 1:0 bpf obj tc_index_to_classid_kern.o sec cls da &&
echo "OK"

Note: We add a prio qdisc at 1:0, which default to deliver packets by 1:1 and 1:2. And we also create a netem qdisc 3:0 at 1:3 which set delay to 3s. So we will use libbpf_stun_netem to deliver all STUN packets to netem which is classid 1:3.

Note: Please note that libbpf_stun_netem is a clsact, which change the skb->tc_classid to 3 which is 1:3, but we need another bpf filter tc_index_to_classid_kern.o which apply to netem.

And attach eBPF bytecode to TC by:

docker exec -it -w /git/tea/libbpf_stun_netem tea ./libbpf_stun_netem

Then, start tcpdump to show packets, and using nc to send packets:

# Capture all UDP packets.
docker exec -it tea tcpdump udp -i any -X

# Start a UDP server, listen at 8000
docker exec -it tea nc -l -u 8000

# Send STUN binding request.
docker exec -it -w /git/tea/libbpf_stun_netem tea bash -c \
    "echo -en \$(cat binding_request.txt |tr -d [:space:]) |nc -p 55293 -w 1 -u 127.0.0.1 8000"

# Send STUN binding response.
docker exec -it -w /git/tea/libbpf_stun_netem tea bash -c \
    "echo -en \$(cat binding_response.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

Note: You will see the packets printed by tcpdump and nc server, before installing the eBPF TC qdisc.

If send STUN messages, you'll find the packet is arrived after 3s：

Apply netem to all STUN packets...
              nc-2752    [000] d....  1206.400129: bpf_trace_printk: Apply netem to STUN packet, type=0x100, len=20480, classid=3

You can also check by:

docker exec -it -w /git/tea/libbpf_stun_netem tea tc -s class ls dev lo
#class prio 1:3 parent 1: leaf 3: 
# Sent 142 bytes 1 pkt (dropped 0, overlimits 0 requeues 0) 
# backlog 0b 0p requeues 0

docker exec -it -w /git/tea/libbpf_stun_netem tea tc -s qdisc ls dev lo
#qdisc netem 3: parent 1:3 limit 1000 delay 3.0s
# Sent 142 bytes 1 pkt (dropped 0, overlimits 0 requeues 0) 
# backlog 0b 0p requeues 0

You can also add loss and other features from netem.

LIBBPF: WebRTC Packets Identify

For WebRTC UDP packets, to identify STUN/DTLS/RTP/RTCP packets.

First, start a docker in background:

mkdir -p ~/git && cd ~/git &&
docker run -d --privileged --name tea -it -v $(pwd):/git -w /git/tea \
    ossrs/tea:latest bash

Next, build the eBPF program:

docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea make

And attach eBPF bytecode to TC by:

docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea ./libbpf_rtc_packets_identify

Then, use nc to send packets:

# Send STUN binding request.
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat binding_request.txt |tr -d [:space:]) |nc -p 55293 -w 1 -u 127.0.0.1 8000"

# Send STUN binding response.
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat binding_response.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

# Send DTLS Server Hello message
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat dtls_server_hello.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

# Send RTP packet(Opus)
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat rtp_opus.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

# Send RTP packet(STAP, H.264 SPS/PPS)
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat rtp_h264_stap.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

# Send RTCP packet(SR, Sender Report)
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat rtcp_sr.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

# Send RTCP packet(PLI, Picture Loss Indication)
docker exec -it -w /git/tea/libbpf_rtc_packets_identify tea bash -c \
    "echo -en \$(cat rtcp_pli.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

Note: You will see the packets printed by tcpdump and nc server, before installing the eBPF TC qdisc.

All WebRTC packets are identified:

Identify WebRTC packets...
              nc-220091  [001] d.... 52468.651187: bpf_trace_printk: Got STUN type=0x00, 55293=>8000
              nc-221981  [002] d.... 53112.085847: bpf_trace_printk: Got PLI 55293=>8000, len=54

For detail about TC and eBPF, please read Links: TC and Links: LIBBPF section.

TC: STUN Drop All

Using tc to load the eBPF object, drop all STUN packets, including binding request and response packets.

First, start a docker in background:

mkdir -p ~/git && cd ~/git &&
docker run -d --privileged --name tea -it -v $(pwd):/git -w /git/tea \
    ossrs/tea:latest bash

Next, build the eBPF program:

docker exec -it -w /git/tea/tc_stun_drop_all tea make

And attach eBPF bytecode to TC by:

docker exec -it tea tc qdisc add dev lo clsact &&
docker exec -it -w /git/tea/tc_stun_drop_all tea tc filter add dev lo egress bpf obj \
    tc_stun_drop_all_kern.o sec cls da &&
docker exec -it -w /git/tea/tc_stun_drop_all tea tc filter add dev lo ingress bpf obj \
    tc_stun_drop_all_kern.o sec cls da &&
echo "OK"

Then, start tcpdump to show packets, and using nc to send packets:

# Capture all UDP packets.
docker exec -it tea tcpdump udp -i any -X

# Start a UDP server, listen at 8000
docker exec -it tea nc -l -u 8000

# Send STUN binding request.
docker exec -it -w /git/tea/tc_stun_drop_all tea bash -c \
    "echo -en \$(cat binding_request.txt |tr -d [:space:]) |nc -p 55293 -w 1 -u 127.0.0.1 8000"

# Send STUN binding response.
docker exec -it -w /git/tea/tc_stun_drop_all tea bash -c \
    "echo -en \$(cat binding_response.txt |tr -d [:space:]) |nc -p 55295 -w 1 -u 127.0.0.1 8000"

Note: You will see the packets printed by tcpdump and nc server, before installing the eBPF TC qdisc.

Now, nc server won't receive STUN packets, and we can check by:

docker exec -it tea \
    bpftool map dump pinned /sys/fs/bpf/tc/globals/tc_stun_drop_all

key: 0d 00 00 00 value: 03 00 00 00 There were 03 STUN packets dropped.

You can also check the last address by:

docker exec -it tea \
    bpftool map dump pinned /sys/fs/bpf/tc/globals/tc_stun_drop_all_ports

key: 0d 00 00 00 value: fd d7 40 1f The port is 0x1f40 (8000) and 0xd7fd (55293).

You can check the first 8 bytes of last packet payload by:

docker exec -it tea \
    bpftool map dump pinned /sys/fs/bpf/tc/globals/tc_stun_drop_all_bytes

key: 0d 00 00 00 value: 00 01 00 50 21 12 a4 42 Which is a binding request.

Because btf_printk is not available for TC loader, so we use map to show debugging information, and all maps are pinned to global namespace, please check by:

docker exec -it tea tree /sys/fs/bpf
# /sys/fs/bpf
# |-- ip -> /sys/fs/bpf/tc/
# |-- tc
# |   `-- globals
# |       |-- tc_stun_drop_all
# |       |-- tc_stun_drop_all_bytes
# |       `-- tc_stun_drop_all_ports
# `-- xdp -> /sys/fs/bpf/tc/

Reset the TC by removing the qdisc:

docker exec -it tea tc qdisc del dev lo clsact

Or by remove the filters:

docker exec -it tea tc filter del dev lo egress
docker exec -it tea tc filter del dev lo ingress

For detail about TC and eBPF, please read Links: TC section.

For SRS

Map ports for SRS:

mkdir -p ~/git && cd ~/git &&
docker run -d --privileged --name tea -it -v $(pwd):/git -w /git/tea \
    --env CANDIDATE="192.168.3.85" -p 1935:1935 -p 1985:1985 -p 8080:8080 -p 8000:8000/udp \
    ossrs/tea:latest bash

Note: Please see Getting Started for detail.

To attach to eth0 for SRS:

docker exec -it -w /git/tea/tc_stun_drop_all tea tc qdisc add dev eth0 clsact &&
docker exec -it -w /git/tea/tc_stun_drop_all tea tc filter add dev eth0 egress bpf obj \
    tc_stun_drop_all_kern.o sec cls da &&
docker exec -it -w /git/tea/tc_stun_drop_all tea tc filter add dev eth0 ingress bpf obj \
    tc_stun_drop_all_kern.o sec cls da &&
echo "OK"

If start a SRS or WebRTC server, all WebRTC clients will fail because STUN is disabled.

docker exec -it -w /git/srs/trunk tea ./objs/srs -c conf/console.conf

Publish a RTMP stream to SRS:

docker run --rm -it ossrs/srs:encoder ffmpeg -stream_loop -1 -re -i doc/source.flv \
  -c copy -f flv rtmp://host.docker.internal/live/livestream

The WebRTC player will be fail.

Reset the TC by removing qdisc:

docker exec -it -w /git/tea/tc_stun_drop_all tea tc qdisc del dev eth0 clsact

Note: The player should recover if SRS session is not timeout.

Packet Hex Escaped String

Capture the packet by wireshark or tcpdump, then open by Wireshark, select the packet, right click and choose Copy > ...as Escaped String.

Please see example at tc_stun_drop_all/binding_request.txt which is copied from files/h5-play-stun.pcapng.

About vmlinux.h

Generate the vmlinux.h if you want:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux-ubuntu-focal-5.15.0-52-generic.h

Note: There might be no BTF in docker, so you can run in an ubuntu server.

Note: For more information about vmlinux.h, please read BTFGen: One Step Closer to Truly Portable eBPF Programs, BTFHub and Running non CO-RE Tracee

About BTF

There are some BTF for old kernel or docker:

For Ubuntu 18: 5.4.0-84-generic.btf.tar.xz
For Ubuntu 20: 5.8.0-23-generic.btf.tar.xz

Or generate from latest ubuntu server which has /sys/kernel/btf/vmlinux:

# For example, uname -r is 5.15.0-52-generic
cp /sys/kernel/btf/vmlinux vmlinux-ubuntu-focal-$(uname -r).btf && 
tar Jcf vmlinux-ubuntu-focal-$(uname -r).btf.tar.xz vmlinux-ubuntu-focal-$(uname -r).btf

Now, we can generate the vmlinux.h, for example:

tar xf 5.4.0-84-generic.btf.tar.xz &&
bpftool btf dump file 5.4.0-84-generic.btf format c > vmlinux-ubuntu-bionic-5.4.0-84-generic.h

BTF is required for eBPF CO-RE, to compatible with different kernel versions without rebuild it.

Note: For more information about BTF and CO-RE, please read BTFGen: One Step Closer to Truly Portable eBPF Programs, BTFHub and Running non CO-RE Tracee

Ubuntu 18 (bionic)

To run eBPF on Ubuntu 18 bionic, should statically build and link in Ubuntu 20 focal, for example, STUN NETEM:

docker exec -it -w /git/tea/libbpf_stun_netem tea make clean static

Now, we start an Ubuntu 18 bionic container, or run in VM server:

mkdir -p ~/git && cd ~/git &&
cd ~/git/tea && docker build -t tea:bionic -f Dockerfile.ubuntu18.bionic . &&
cd ~/git && docker run -d --privileged --name bionic -it -v $(pwd):/git -w /git/tea \
    tea:bionic bash

Then, create a BTF file for Ubuntu 18 bionic:

mkdir ~/git/tea/tmp &&
docker exec -it -w /git/tea/tmp bionic make -f ../libbpf_stun_netem/Makefile vmlinux

Add 3s delay for STUN packet:

docker exec -it -w /git/tea/libbpf_stun_netem bionic \
    tc qdisc add dev lo root handle 1:0 prio &&
docker exec -it -w /git/tea/libbpf_stun_netem bionic \
    tc qdisc add dev lo parent 1:3 handle 3:0 netem delay 3000ms &&
docker exec -it -w /git/tea/libbpf_stun_netem bionic \
    tc filter add dev lo parent 1:0 bpf obj tc_index_to_classid_kern.o sec cls da && 
echo "OK"

And attach eBPF bytecode to TC by:

docker exec -it -w /git/tea/tmp bionic ../libbpf_stun_netem/libbpf_stun_netem

Then, start tcpdump to show packets, and using nc to send packets:

# Start a UDP server, listen at 8000
docker exec -it bionic nc -l -u -p 8000

# Send STUN binding request.
docker exec -it -w /git/tea/libbpf_stun_netem bionic bash -c \
    "echo -en \$(cat binding_request.txt |tr -d [:space:]) |nc -p 55293 -w 1 -u 127.0.0.1 8000"

All STUN packets is delayed.

Links: TC

Traffic Control HOWTO Martin A. Brown 2006.
Linux Advanced Routing & Traffic Control HOWTO Bert Hubert 2012.
netem: Network Emulation Linux iproute2.
[译] Facebook 流量路由最佳实践：从公网入口到内网业务的全路径 XDP/BPF 基础设施（LPC, 2021） Arthur Chiao 2020
[译] 深入理解 tc ebpf 的 direct-action (da) 模式（2020） Arthur Chiao 2021
[译] 流量控制（TC）五十年：从基于缓冲队列（Queue）到基于时间（EDT）的演进（Google, 2018） Arthur Chiao 2022

Links: LIBBPF

libbpf, contains an eBPF loader which takes over processing LLVM generated eBPF ELF files for loading into the kernel.
bpftool, allows inspection and simple manipulation of eBPF programs and maps.
Features of bpftool: the thread of tips and examples to work with eBPF objects Quentin Monnet 2021
BPF Portability and CO-RE Andrii Nakryiko 2020
BTFGen: One Step Closer to Truly Portable eBPF Programs Mauricio Vásquez Bernal 2022
BTFHub, provides BTF files for existing published kernels that don't support embedded BTF.
[译] BPF 可移植性和 CO-RE（一次编译，到处运行）（Facebook，2020） Arthur Chiao 2021

Winlin 2022.12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

TEA

Usage

LIBBPF: STUN Drop All

LIBBPF: STUN NETEM

LIBBPF: WebRTC Packets Identify

TC: STUN Drop All

For SRS

Packet Hex Escaped String

About vmlinux.h

About BTF

Ubuntu 18 (bionic)

Links: TC

Links: LIBBPF

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
.github/workflows		.github/workflows
btf		btf
files		files
libbpf_rtc_packets_identify		libbpf_rtc_packets_identify
libbpf_stun_drop_all		libbpf_stun_drop_all
libbpf_stun_netem		libbpf_stun_netem
tc_stun_drop_all		tc_stun_drop_all
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.ubuntu18.bionic		Dockerfile.ubuntu18.bionic
LICENSE		LICENSE
README.md		README.md

License

winlinvip/tea

Folders and files

Latest commit

History

Repository files navigation

TEA

Usage

LIBBPF: STUN Drop All

LIBBPF: STUN NETEM

LIBBPF: WebRTC Packets Identify

TC: STUN Drop All

For SRS

Packet Hex Escaped String

About vmlinux.h

About BTF

Ubuntu 18 (bionic)

Links: TC

Links: LIBBPF

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages