IMPORTANT!
- To run the proxy manually, run
sudo go run cmd/main.goorsudo make run. - To run a full e2e test, run
sudo make all
The proxy has been tested in the following OS, with the respective kernel and bpf tools versions.
Also, it's required to run it as root user.
The system must be able to run BPF programs.
-
Kernel
5.15.0-47-generic -
golang 1.18
-
BPF packages:
binutils-bpf/jammy 2.38-2ubuntu1+3 amd64
bpftrace/jammy 0.14.0-1 amd64
libbpf-dev/jammy 1:0.5.0-1 amd64
libbpf0/jammy,now 1:0.5.0-1 amd64 [installed,automatic]- Kernel
5.18.17-200.fc36.x86_64 - golang 1.18
- BPF packages:
libbpf-0.7.0-3.fc36.x86_64
libbpf-devel-0.7.0-3.fc36.x86_64
bpftrace-0.14.1-1.fc36.x86_64
bpftool-5.19.4-200.fc36.x86_64- Create a new vm (ubuntu 22.04), and scan it with nmap
$ nmap -sT -p 1-10000 192.168.122.172
Starting Nmap 7.92 ( https://nmap.org ) at 2022-09-16 00:51 CEST
Nmap scan report for 192.168.122.172
Host is up (0.00024s latency).
Not shown: 9999 closed tcp ports (conn-refused)
PORT STATE SERVICE
22/tcp open ssh
Nmap done: 1 IP address (1 host up) scanned in 0.57 seconds- This is an ubuntu 22.04
root@proxy-last:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"- Install required packages (commands extracted from Dockerfile, it includes more pkg than needed)
$ apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install --no-install-recommends -y ca-certificates clang curl git llvm libelf-dev make netcat openssh-server openssl golang && rm -rf /var/lib/apt/lists/*
$ apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install --no-install-recommends -y autoconf bison cmake dkms flex gawk gcc python3 rsync libiberty-dev libncurses-dev libpci-dev libssl-dev libudev-dev \
&& curl https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.13.tar.gz | tar -xz \
&& make -C /linux-5.13 headers_install INSTALL_HDR_PATH=/usr \
&& make -C /linux-5.13/tools/lib/bpf install INSTALL_HDR_PATH=/usr \
&& make -C /linux-5.13/tools/bpf/bpftool install \
&& apt-get remove -y \
autoconf bison cmake dkms flex gawk gcc python3 rsync \
libiberty-dev libncurses-dev libpci-dev libssl-dev libudev-dev \
&& apt autoremove -y \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /linux-5.13- Clone the repository
git clone https://github.com/fbac/proxy-tcp-roundrobin/
Cloning into 'proxy-tcp-roundrobin'...
remote: Enumerating objects: 78, done.- Run as a background process Notice the debug messages, these are for checking the backends ignored/added, and also the pinning of ebpf programs, maps and fd's
root@proxy-last:~/proxy-tcp-roundrobin# make run
-e # fly-proxy build started
go build -o bin/fly-proxy cmd/proxy/*.go
-e
# running fly-proxy
bin/fly-proxy &
-e
# wait until all listeners are ready
2022/09/15 22:58:35 started listener in :5001
2022/09/15 22:58:35 DEBUG Prog SkLookup(echo_dispatch)#17 is pinned: true
2022/09/15 22:58:35 DEBUG Map SockMap(echo_socket)#16 is pinned: true
2022/09/15 22:58:35 DEBUG Map Hash(echo_ports)#15 is pinned: true
2022/09/15 22:58:35 DEBUG: App: five-thousand listener FD: 14
2022/09/15 22:58:35 DEBUG: App: five-thousand adding port: 5200
2022/09/15 22:58:35 DEBUG: App: five-thousand adding port: 5300
2022/09/15 22:58:35 DEBUG: App: five-thousand adding port: 5400
2022/09/15 22:58:35 skipping unhealthy backend: bad.target.for.testing: lookup bad.target.for.testing: no such host
2022/09/15 22:58:35 started listener in :6001
2022/09/15 22:58:35 DEBUG Prog SkLookup(echo_dispatch)#26 is pinned: true
2022/09/15 22:58:35 DEBUG Map SockMap(echo_socket)#25 is pinned: true
2022/09/15 22:58:35 DEBUG Map Hash(echo_ports)#24 is pinned: true
2022/09/15 22:58:35 DEBUG: App: six-thousand listener FD: 23
2022/09/15 22:58:35 DEBUG: App: six-thousand adding port: 6200
2022/09/15 22:58:35 DEBUG: App: six-thousand adding port: 6300
2022/09/15 22:58:35 DEBUG: App: six-thousand adding port: 6400
2022/09/15 22:58:35 started listener in :7001
2022/09/15 22:58:35 DEBUG Prog SkLookup(echo_dispatch)#34 is pinned: true
2022/09/15 22:58:35 DEBUG Map SockMap(echo_socket)#33 is pinned: true
2022/09/15 22:58:35 DEBUG Map Hash(echo_ports)#32 is pinned: true
2022/09/15 22:58:35 DEBUG: App: seven-thousand listener FD: 31
2022/09/15 22:58:35 DEBUG: App: seven-thousand adding port: 7200
2022/09/15 22:58:35 DEBUG: App: seven-thousand adding port: 7300
2022/09/15 22:58:35 DEBUG: App: seven-thousand adding port: 7400
root@proxy-last:~/proxy-tcp-roundrobin#- Scan ports again
[aranda@hyperion :: ~ ] $ nmap -sT -p 1-10000 192.168.122.172
Starting Nmap 7.92 ( https://nmap.org ) at 2022-09-16 00:59 CEST
Nmap scan report for 192.168.122.172
Host is up (0.00023s latency).
Not shown: 9987 closed tcp ports (conn-refused)
PORT STATE SERVICE
22/tcp open ssh
5001/tcp open commplex-link
5200/tcp open targus-getdata
5300/tcp open hacl-hb
5400/tcp open pcduo-old
6001/tcp open X11:1
6200/tcp open lm-x
6300/tcp open bmc-grx
6400/tcp open crystalreports
7001/tcp open afs3-callback
7200/tcp open fodms
7300/tcp open swx
7400/tcp open rtps-discovery
Nmap done: 1 IP address (1 host up) scanned in 0.60 seconds- Also, on the proxy vm, some tcp connection debug messages will popup This is just for debugging purposes, here we can also check the load balancing is working properly in round robin fashion.
root@proxy-last:~/proxy-tcp-roundrobin#
2022/09/15 22:59:31 proxying data from :7001 to tcp-echo.fly.dev:7001
2022/09/15 22:59:31 proxying data from :6001 to tcp-echo.fly.dev:6001
2022/09/15 22:59:31 proxying data from :5001 to tcp-echo.fly.dev:5001
2022/09/15 22:59:31 proxying data from :7001 to tcp-echo.fly.dev:7002
2022/09/15 22:59:31 proxying data from :7001 to tcp-echo.fly.dev:7001
2022/09/15 22:59:31 proxying data from :5001 to tcp-echo.fly.dev:5002
2022/09/15 22:59:31 proxying data from :7001 to tcp-echo.fly.dev:7002
2022/09/15 22:59:31 proxying data from :6001 to tcp-echo.fly.dev:6002
2022/09/15 22:59:31 proxying data from :6001 to tcp-echo.fly.dev:6001
2022/09/15 22:59:31 proxying data from :6001 to tcp-echo.fly.dev:6002
2022/09/15 22:59:31 proxying data from :5001 to tcp-echo.fly.dev:5001
2022/09/15 22:59:31 proxying data from :5001 to tcp-echo.fly.dev:5002- Test manually with netcat from your localhost
[aranda@hyperion :: ~ ] $ echo "test" | nc -v -4 192.168.122.172 7200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.122.172:7200.
TEST- The debug message will popup as well
2022/09/15 23:03:14 proxying data from :7001 to tcp-echo.fly.dev:7001