The network performance is somehow disastrous.

[OrvilleQ]

Just created a server on my Hetzner server located at Germany. I self built the kernel to enable bbr and it's now on 14.0-RELEASE-p6.

cpu             ARM64
ident           HETZNER-CAX

include         "std.arm64"
include         "std.dev"

# TCP BBR
options         TCPHPTS
options         RATELIMIT
makeoptions     WITH_EXTRA_TCP_STACKS=1

# Include SoC specific configuration
include         "std.arm"
include         "std.virt"

I followed your document and set pf, virtual networks and dns up, then I create a jail with appjail quick packager virtualnet=":packager" (nat is enabled on the virtualnet), but when I download ports using gitup I notice extremely slow network speed.

Here's the result if I fetch a 10gb test file outside of jail:

root@fsn00:~ # fetch https://fsn1-speed.hetzner.com/10GB.bin
10GB.bin                                       19% of   10 GB  408 MBps    20s^C
fetch: transfer interrupted

And here's the result inside the jail:

root@packager:~ # fetch https://fsn1-speed.hetzner.com/10GB.bin
10GB.bin                                        0% of   10 GB  249 kBps 11h15m^C
fetch: transfer interrupted

It's 1500x times slower.

I'm new to FreeBSD, so I'm not sure where to start with debugging and digging useful information. If there are some specific details required, please inform me, and I will provide them as promptly as possible.

[DtxdF]

Hi @OrvilleQ

I am currently not at home which has 80 MB/s (rx) and 30 MB/s (tx), a big difference if we compare the ADSL link where I am.

Excerpt from net/speedtest-go:

# speedtest-go
...
✓ Latency: 42.665045ms Jitter: 117.528µs Min: 42.406088ms Max: 42.883802ms
✓ Download: 4.51Mbps (used: 5.38MB) (latency: 61ms jitter: 37ms min: 48ms max: 205ms)
✓ Upload: 0.76Mbps (used: 0.91MB) (latency: 73ms jitter: 21ms min: 47ms max: 139ms)

But if we compare the result of speedtest-go executed in a jail:

# appjail jail list -j speedtest-go
STATUS  NAME          TYPE  VERSION       PORTS  NETWORK_IP4
UP      speedtest-go  thin  13.2-RELEASE  -      10.0.0.5
# appjail cmd jexec speedtest-go speedtest-go
...
✓ Latency: 31.867667ms Jitter: 845.172µs Min: 31.07015ms Max: 33.496017ms
✓ Download: 4.87Mbps (used: 5.80MB) (latency: 87ms jitter: 111ms min: 35ms max: 451ms)
✓ Upload: 0.76Mbps (used: 0.91MB) (latency: 91ms jitter: 63ms min: 51ms max: 250ms)

It's even better, but this doesn't mean that the performance in jails is better, it's probably just better because speedtest-go used two different servers. Although there is an overhead when using virtual networks, it is actually so small that you should not notice any difference. Of course, in your case there is a problem since you notice a performance loss. Check the bridge's MTU to confirm if it matches the host's MTU currently used:

# appjail network list
NAME   NETWORK   CIDR  BROADCAST      GATEWAY   MINADDR   MAXADDR        ADDRESSES  DESCRIPTION      MTU
ajnet  10.0.0.0  10    10.63.255.255  10.0.0.1  10.0.0.1  10.63.255.254  4194302    AppJail network  576
# ifconfig jext | grep mtu
jext: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 576
# grep MTU /usr/local/etc/appjail/appjail.conf 
DEFAULT_VIRTUALNET_MTU=576
# appjail version
3.2.0+15f7db9c07e005d852119b2346b93b9900d4e023

Another recommendation is to try to minimize configuration overhead:

• Use GENERIC kernel instead of a custom one as it may or may not affect network performance in jails, but if this happens it is probably a bug. Note: I'm just trying to deduce the cause, not conclude that a custom kernel can affect its performance.
• If you have configured your firewall, try to minimize the configuration to a minimum:

# cat /etc/pf.conf
nat-anchor 'appjail-nat/jail/*'
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"

This is what AppJail requires to work when using Virtual Networks, no more, no less.

• Use another networking feature (for discard only) like VNET, Bridges or even Netgraph. If you have more than one interface or you don't have a problem with a jail stealing your interface, I recommend using VNET. VNET will steal the target interface from the host and you can make any allowed changes to that interface, so you have more control and no overhead. Bridges and Netgraph have a bit of overhead and under normal circumstances you shouldn't see any real difference as mentioned above.

Bridges and Virtual Networks use if_bridge(4) and if_epair(4) drivers, and most jail frameworks and users use the same drivers for jail networking, so if you notice a performance overhead, this will affect to many people unless it is just a problem in your system.

• AppJail is used more on AMD64 platforms, but I know some users who have deployed AppJail on ARM platforms and there are no problems. Test on AMD64 platforms (for discard only).

---

I hope the above can help you, tell me any results you find.

[OrvilleQ]

@DtxdF Thank you for your detailed instructions. I'll log the things I did down below in case I miss something.

---

appjail network list did not show MTU, maybe because I'm using the release version. However, the configuration file says MTU defaults to 1500, which is the same as my EX_IF interface vtnet0. ifconfig ajnet also shows the MTU is 1500, so I guess MTU is not the problem.

root@fsn00:~/Kernel # ifconfig vtnet0 | grep mtu
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500

root@fsn00:~ # ifconfig ajnet
ajnet: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500

The pf.conf file is as minimal as possible already.

root@fsn00:~ # cat /etc/pf.conf
nat-anchor "appjail-nat/jail/*"
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"

Test the network speed using speedtest-go inside and outside the jail. The upload performance inside the jail is 40% slower compared to outside the jail, but it is still fast enough.

Inside the jail:

✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 663.682µs Jitter: 177.098µs Min: 444.743µs Max: 1.066988ms
✓ Download: 19.49Mbps (used: 23.23MB) (latency: 0ms jitter: 1ms min: 0ms max: 5ms)
✓ Upload: 1659.30Mbps (used: 1978.04MB) (latency: 2ms jitter: 2ms min: 0ms max: 11ms)

Outide the jail:

✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 923.864µs Jitter: 1.263447ms Min: 439.323µs Max: 4.709956ms
✓ Download: 6298.97Mbps (used: 7508.96MB) (latency: 1ms jitter: 0ms min: 0ms max: 3ms)
✓ Upload: 2853.11Mbps (used: 3401.18MB) (latency: 1ms jitter: 1ms min: 0ms max: 8ms)

The inner connection inside ajnet seems to be good, get a reasonable result using iperf3.

root@packager:~ # iperf3 -c test                                              
Connecting to host test, port 5201                                            
[  5] local 10.0.0.2 port 56325 connected to 10.0.0.3 port 5201               
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd              
[  5]   0.00-1.00   sec   505 MBytes  4.24 Gbits/sec    0   1.61 MBytes       
[  5]   1.00-2.00   sec   511 MBytes  4.28 Gbits/sec    0   1.61 MBytes       
[  5]   2.00-3.05   sec   538 MBytes  4.30 Gbits/sec    1    864 KBytes       
[  5]   3.05-4.04   sec   508 MBytes  4.33 Gbits/sec    0    864 KBytes       
[  5]   4.04-5.00   sec   488 MBytes  4.24 Gbits/sec    0    864 KBytes       
[  5]   5.00-6.01   sec   499 MBytes  4.14 Gbits/sec    0   1.05 MBytes       
[  5]   6.01-7.00   sec   500 MBytes  4.23 Gbits/sec    0   1.05 MBytes       
[  5]   7.00-8.00   sec   520 MBytes  4.37 Gbits/sec    0   1.05 MBytes       
[  5]   8.00-9.00   sec   530 MBytes  4.44 Gbits/sec    0   1.05 MBytes       
[  5]   9.00-10.01  sec   528 MBytes  4.40 Gbits/sec    0   1.05 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -                             
[ ID] Interval           Transfer     Bitrate         Retr                    
[  5]   0.00-10.01  sec  5.01 GBytes  4.30 Gbits/sec    1             sender  
[  5]   0.00-10.01  sec  5.01 GBytes  4.30 Gbits/sec                  receiver

iperf Done.                                                                   

After compiling the Kernel to GENERIC with BBR enabled, nothing changed. I'll try disabling BBR next.

[OrvilleQ]

root@fsn00:~ # sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd

✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 696.362µs Jitter: 239.203µs Min: 398.222µs Max: 1.077307ms
✓ Download: 19.88Mbps (used: 23.70MB) (latency: 0ms jitter: 0ms min: 0ms max: 2ms)
✓ Upload: 1634.93Mbps (used: 1948.99MB) (latency: 2ms jitter: 1ms min: 0ms max: 8ms)

After BBR disabled still nothing changed.

---

Edit: I was using appjail quick packager virtualnet=":packager default" to create the jail.

[OrvilleQ]

On a fresh Arm64 FreeBSD environment update to the latest patch (14.0-RELEASE-p6) with only appjail installed, using ZFS, EXT_IF and ON_IF set to vtnet0. pf configured.

set up a test jail with appjail quick test virtualnet=":test default" nat overwrite start, the speedtest-go result is still extremely slow on download speed.

✓ Test Server: [41175] 60.00km Dresden (Germany) by CITUX GmbH
✓ Latency: 767.364µs Jitter: 442.745µs Min: 390.143µs Max: 2.030396ms
✓ Download: 19.89Mbps (used: 23.71MB) (latency: 1ms jitter: 1ms min: 0ms max: 4ms)
✓ Upload: 1310.75Mbps (used: 1562.54MB) (latency: 3ms jitter: 1ms min: 0ms max: 7ms)

[DtxdF]

Thanks for telling me about the driver you use, vtnet, this helped me dig deeper into your problem, at least to investigate further on Bugzilla:

• Bug 277718 - Really bad throughput of pf NAT over vtnet(4) with rx/txcsum enabled

The above is the most interesting, as it seems to be related to your problem, but look at the full search list:

• Full Search List

The inner connection inside ajnet seems to be good, get a reasonable result using iperf3.

Yes, this appears to be what was reported in the issue, since you are testing on the local system, packets don't travel outside of your interface.

0.3 Mbit/s download while upload seems to be unaffected

This is a quote from the bug, comment 0. As speedtest-go informs you, Download seems to be affected, while Upload is only affected a bit. Of course, see the full bug, as the developer says some things that are relevant to investigate further.

---

You haven't done anything wrong, this seems to be a bug between vtnet and the NAT implementation or something related. Of course this is not an AppJail issue, but I suggest the following:

• Add you to the CC list for bugs that seem related to your problem.
• Join the freebsd-net mailing list.
• Open a bug on Bugzilla or if there is an open bug that seems to be related to your issue, like the one above, let them know. At least if the bug affects a lot of people it should be fixed quickly.

[OrvilleQ]

You are right.

After read all the comments and found a post back in 2016 described similar issue, I set hw.vtnet.csum_disable="1" in /boot/loader.conf, and everything works as expected now.

inside the jail:

✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH                         
✓ Latency: 578.274µs Jitter: 410.754µs Min: 364.603µs Max: 1.790574ms                  
✓ Download: 3212.81Mbps (used: 3829.97MB) (latency: 2ms jitter: 1ms min: 0ms max: 4ms) 
✓ Upload: 2108.45Mbps (used: 2513.46MB) (latency: 1ms jitter: 0ms min: 0ms max: 3ms)   

outside the jail:

✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH                        
✓ Latency: 555.28µs Jitter: 78.684µs Min: 418.683µs Max: 690.805µs                    
✓ Download: 3489.01Mbps (used: 4159.22MB) (latency: 2ms jitter: 1ms min: 1ms max: 5ms)
✓ Upload: 1755.26Mbps (used: 2092.44MB) (latency: 2ms jitter: 1ms min: 1ms max: 6ms)  

But that means CPU now has to do all the checksum job and It could easily use around 50% CPU resource with maximum speed. Not a perfect solution but enough for now. I'll keep my eye on the bug fix in the future.

Once again, thank you sincerely for the help.

I've create a pull request to warn this issue in document. DtxdF/AppJail.docs#2

[DtxdF]

Thanks for the workaround and for the PR on the other repo! I'll merge it as soon as possible.

I'll keep this in mind when writing the appjail-network(1) man page.