Skip to content

Commit

Permalink
chore(sim,linux): docs, ci updated with sudo
Browse files Browse the repository at this point in the history
  • Loading branch information
thedevbirb committed Feb 7, 2024
1 parent 4798eed commit c866e48
Show file tree
Hide file tree
Showing 2 changed files with 93 additions and 26 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ jobs:
with:
cache-on-failure: true
- name: cargo test
run: cargo test --all
run: sudo HOME=$HOME $(which cargo) test --all
- name: cargo test all features
run: cargo test --all --all-features
run: sudo HOME=$HOME $(which cargo) test --all --all-features

cargo-lint:
runs-on: ubuntu-latest
Expand Down
115 changes: 91 additions & 24 deletions msg-sim/README.md
Original file line number Diff line number Diff line change
@@ -1,65 +1,132 @@
# `msg-sim`

## Overview

This crate provides functionality to simulate real-world network conditions
locally to and from a specific endpoint for testing and benchmarking purposes.
It only works on MacOS and Linux.

## Implementation

### MacOS

On MacOS, we use a combination of the `pfctl` and `dnctl` tools.
[`pfctl`](https://man.freebsd.org/cgi/man.cgi?query=pfctl&apropos=0&sektion=8&manpath=FreeBSD+14.0-RELEASE+and+Ports&arch=default&format=html) is a tool to manage the packet filter device. [`dnctl`](https://man.freebsd.org/cgi/man.cgi?query=dnctl&sektion=8&format=html) can manage
the [dummynet](http://info.iet.unipi.it/~luigi/papers/20100304-ccr.pdf) traffic shaper.
[`pfctl`](https://man.freebsd.org/cgi/man.cgi?query=pfctl&apropos=0&sektion=8&manpath=FreeBSD+14.0-RELEASE+and+Ports&arch=default&format=html)
is a tool to manage the packet filter device.
[`dnctl`](https://man.freebsd.org/cgi/man.cgi?query=dnctl&sektion=8&format=html)
can manage the
[dummynet](http://info.iet.unipi.it/~luigi/papers/20100304-ccr.pdf) traffic
shaper.

The general flow is as follows:

* Create a dummynet pipe with `dnctl` and configure it with `bw`, `delay`, `plr`
- Create a dummynet pipe with `dnctl` and configure it with `bw`, `delay`,
`plr`

Example:
```bash
sudo dnctl pipe 1 config bw 10Kbit/s delay 50 plr 0.1
```

* Create a loopback alias with `ifconfig` to simulate a different endpoint and
set the MTU to the usual value (1500)
`bash sudo dnctl pipe 1 config bw 10Kbit/s delay 50 plr 0.1 `

- Create a loopback alias with `ifconfig` to simulate a different endpoint and
set the MTU to the usual value (1500)

Example:
```bash
sudo ifconfig lo0 alias 127.0.0.3 up
sudo ifconfig lo0 mtu 1500
```

* Use `pfctl` to create a rule to match traffic and send it through the pipe
`bash sudo ifconfig lo0 alias 127.0.0.3 up sudo ifconfig lo0 mtu 1500 `

- Use `pfctl` to create a rule to match traffic and send it through the pipe

Example:

```bash
# Create an anchor (a named container for rules, close to a namespace)
(cat /etc/pf.conf && echo "dummynet-anchor \"msg-sim\"" && \
echo "anchor \"msg-sim\"") | sudo pfctl -f -

# Create a rule to match traffic from any to the alias and send it through the pipe
echo 'dummynet in from any to 127.0.0.3 pipe 1' | sudo pfctl -a msg-sim -f -
(cat /etc/pf.conf && echo "dummynet-anchor \"msg-sim\"" && \ echo "anchor \"msg-sim\"") | sudo pfctl -f -

# Create a rule to match traffic from any to the alias and send it through the

pipe echo 'dummynet in from any to 127.0.0.3 pipe 1' | sudo pfctl -a msg-sim -f

# Enable the packet filter

sudo pfctl -E
```

* Remove the rules and the pipe
- Remove the rules and the pipe

```bash
# Apply the default configuration

sudo pfctl -f /etc/pf.conf

# Disable the packet filter

sudo pfctl -d

# Remove the alias & reset the MTU
sudo ifconfig lo0 -alias 127.0.0.3
sudo ifconfig lo0 mtu 16384
# Remove the dummynet pipes
sudo dnctl pipe delete 1

sudo ifconfig lo0 -alias 127.0.0.3 sudo ifconfig lo0 mtu 16384

# Remove the dummynet

pipes sudo dnctl pipe delete 1
```

### Questions
- Do we need to create 2 pipes to simulate a bidirectional link? MAN page seems to say so.

- Do we need to create 2 pipes to simulate a bidirectional link? MAN page seems
to say so.

### Linux
On Linux, we use dummy interfaces and `tc` with `netem` to simulate and shape traffic.

On Linux, we leverage network namespaces to simulate different networking
conditions, leveraging the `tc` and `netem` command to shape traffic.

On each namespace, we create a veth pair, with one end in the default namespace,
and then we configure the veth devices as needed.

The general flow is as follows:

- Create a network namespace with `ip netns add`
- Add a veth pair to the namespace with `ip link add`
- Set the veth pair up with `ip link set`
- Set the IP address of the veth pair in the namespace with `ip netns exec`
- Set the network emulation parameters with `tc qdisc add dev` both in the host and
the namespaced environment

Example:

```bash
# create namespace ns1
sudo ip netns add ns1
# create veth devices linked together
sudo ip link add veth-host type veth peer name veth-ns1
# move veth-ns1 device to ns1 namespace
sudo ip link set veth-ns1 netns ns1

# associate ip addr to veth-host device and spin it up
sudo ip addr add 192.168.1.2/24 dev veth-host
sudo ip link set veth-host up

# same but from ns1 namespace
sudo ip netns exec ns1 ip addr add 192.168.1.1/24 dev veth-ns1
sudo ip netns exec ns1 ip link set veth-ns1 up

# add latency etc to veth-ns1 from ns1 namespace
sudo ip netns exec ns1 tc qdisc add dev veth-ns1 root netem delay 3000ms loss 50%

# this should be slow
ping 192.168.1.1
```

#### How to run tests

Given that the tests require root privileges to modify the networking stack,
you can run them with the following command:

```bash
sudo HOME=$HOME $(which cargo) test # add your arguments here
```

We need to provide the `$HOME` environment variable to `sudo` to ensure that
it can find the Rust toolchain, and then we also need to provide the path of `cargo`.

0 comments on commit c866e48

Please sign in to comment.