engine: throttling-aware jafar replacement

[bassosimone]

We currently have jafar. It is a tool to simulate censorship. It uses the Linux kernel network filtering and routing functionality to make endpoints unaccessible, drop packets, ~transparently redirect to DNS/TLS/HTTP proxies implementing blocking.

During #1797, I started sketching up the prototype for jafar2. The code is currently not part of jafar, because I was doing research and needed to break/change things quickly.

This issue documents some insights on how we could improve jafar and merge good stuff from jafar2.

1. We should support network emulation inside jafar. This includes adding extra latency, dropping packets, traffic shaping and generally all the policies that are possible using Linux's netem.

2. We should investigate the possibility of using Linux's transparent proxy to intercept QUIC, DNS, TLS. We currently do not support intercepting/blocking QUIC, but for setting iptables rules. However, if we switch to the transparent proxy model, we can drop a bunch of iptables rules in favour, instead, of dropping directly in the proxies.

3. We should introduce configuration files describing the conditions in which we want to run a test. For example, we may want to run both ndt7 and websteps with 3G-like network conditions. It would be useful to put this information inside of a configuration file.

4. If we want to support running any command, it would be more seamless to switch to a getopt-like options parsing model where everything after the -- defines what child command we want to run.

5. An interesting idea is to tell jafa2 to automatically choose what to block, then run a full scan, let jafar2 write down what it has blocked, and then check whether this is mirrored into the measurement files.

There is no timeline/priority attached to this issue, for now. It will be fine to keep the changes out of tree into a separate branch and continuing to refine them as we need them for developing websteps.

Here are some suggestions from @FedericoCeratto re: how to better restructure the current prototype:

1. most kernels are now tickless and it's important to run tests on bare metal as opposed to docker (which basically boils down to manually creating a bridge, attaching a virtual NIC to it, and then running the program so that it uses the virtual NIC as opposed to the default NIC of the system);

2. netem now implements shaping, so we can avoid using TBF;

3. it would be nice to have reproducible jafar2 experiments to integrate into the CI;

4. it's possible to better the simulation of mobile networks by using netem's options for slotting and reordering.

I'll transform this issue into an Epic so we can create child issues.

[bassosimone]

Scripts to work with namespaces

Creating a namespace

#!/bin/bash
set -ex
ip netns add jafar
ip link add jafar0 type veth peer netns jafar name jafar1
ip link set dev jafar0 up
ip -n jafar link set dev jafar1 up
ip address add 10.17.14.1/24 dev jafar0
ip -n jafar address add 10.17.14.11/24 dev jafar1
ip -n jafar route add default via 10.17.14.1 dev jafar1
iptables -t nat -A POSTROUTING -s 10.17.14.0/24 -o wlp2s0 -j MASQUERADE
mkdir -p /etc/netns/jafar/
echo "nameserver 8.8.8.8" > /etc/netns/jafar/resolv.conf

Three notes: (1) it seems that Fedora's firewalld prevents this kind of namespaces from working because it blocks certain ICMP packets and therefore causes errors and (2) systemd's 127.0.0.53 resolver does not work inside the namespace and we need to create a namespace specific directory, which is then overlay-mounted when we enter the namespace and (3) the script is currently hardcoding the correct interface for me.

Using the namespace

#!/bin/bash
set -ex
ip netns exec jafar "$@"

Removing the namespace

#!/bin/bash
set -ex
ip netns del jafar
iptables -t nat -D POSTROUTING -s 10.17.14.0/24 -o wlp2s0 -j MASQUERADE
rm -rf /etc/netns/jafar

Useful links

• https://unix.stackexchange.com/questions/443898/separate-dns-configuration-in-each-network-namespace

• https://gist.github.com/dpino/6c0dca1742093346461e11aa8f608a99

• https://unix.stackexchange.com/questions/552857/why-are-my-network-connections-being-rejected

• https://itnext.io/create-your-own-network-namespace-90aaebc745d

• https://hintcafe.net/post/78293519027/running-a-process-inside-a-network-namespace

[bassosimone]

Alternative approach

See how google/martian and Shopify/toxyproxy work. It may be that we can implement most operations in userspace without any kernel support, which is potentially quite great to perform QA on all platforms.

Part of this approach will include some transparent proxying. Roughly speaking we need to transparently proxy the system resolver and DNS over UDP resolvers to a local proxy. For TCP connections, the most complete solution is probably to redirect to a SOCKS5 proxy. For QUIC connections, the ideal would be some sort of SOCKS5 using QUIC. If we take out the SOCKS5 part, then we need to use the SNI, which is missing for URLs such as http://1.1.1.1.1, which are URLs we want to test.

This project https://github.com/freedomio/fio-go may also be useful if we need to implement a "socks5 for QUIC" approach.

[bassosimone]

Another possible design

We will stop using netfilter for network filtering. We can do filtering and routing directly inside the implementation by adding hooks that can direct traffic elsewhere to netxlite. We previously had a package called selfcensor, which attempted to do this. The issue with such a package is that it contained both the mechanism and the policy to selfcensor. We're now going to do it a bit differently: the mechanism (allowing for transparent interception) will live inside netxlite, while the policy will be inside netxlite/filtering. The default mechanism will just use the stdlib. The policy will allow to direct traffic to services that are basically Jafar's services, except that we'll not be using any netfilter this time.