tc and tsproxy shaping differ significantly

[worenga]

When comparing the load times of between tc and tsproxy i've noticed that they differ quite a lot

First some baseline measurement for https://google.de

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js https://google.de                                                         
[2017-02-14 11:47:05] Running chrome for url: https://google.de
[2017-02-14 11:47:05] Testing url https://google.de run 1
[2017-02-14 11:47:10] Testing url https://google.de run 2
[2017-02-14 11:47:14] Testing url https://google.de run 3
[2017-02-14 11:47:19] 21 requests, 752.09 kb, firstPaint: 670ms (±19.25ms), DOMContentLoaded: 725ms (±20.83ms), Load: 1.75s (±22.66ms), rumSpeedIndex: 762 (±9.68) (3 runs)
[2017-02-14 11:47:19] Wrote data to browsertime-results/google.de/2017-02-14T114704+0100

Now use tc shaping with cable and 3g profile

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tc --connectivity.profile cable https://google.de --connectivity.tc.device enp0s25
[2017-02-14 11:36:43] Running chrome for url: https://google.de
[2017-02-14 11:36:43] sudo tc qdisc add dev enp0s25 root netem delay 28ms loss 0% rate 5000kbps
[2017-02-14 11:36:43] Testing url https://google.de run 1
[2017-02-14 11:36:49] Testing url https://google.de run 2
[2017-02-14 11:36:55] Testing url https://google.de run 3
[2017-02-14 11:37:00] 21 requests, 752.13 kb, firstPaint: 898ms (±5.54ms), DOMContentLoaded: 959ms (±8.02ms), Load: 2.23s (±31.68ms), rumSpeedIndex: 1061 (±14.12) (3 runs)

[2017-02-14 11:37:00] Wrote data to browsertime-results/google.de/2017-02-14T113643+0100
[2017-02-14 11:37:00] sudo tc qdisc del dev enp0s25 root
➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tc --connectivity.profile 3g https://google.de --connectivity.tc.device enp0s25
[2017-02-14 11:37:18] Running chrome for url: https://google.de
[2017-02-14 11:37:18] sudo tc qdisc add dev enp0s25 root netem delay 300ms loss 0% rate 1600kbps
[2017-02-14 11:37:18] Testing url https://google.de run 1
[2017-02-14 11:37:29] Testing url https://google.de run 2
[2017-02-14 11:37:41] Testing url https://google.de run 3
[2017-02-14 11:37:52] 21 requests, 686.37 kb, firstPaint: 3.43s (±52.63ms), DOMContentLoaded: 3.84s (±44.82ms), Load: 8.38s (±51.03ms), rumSpeedIndex: 3849 (±90.67) (3 runs)
[2017-02-14 11:37:52] Wrote data to browsertime-results/google.de/2017-02-14T113717+0100
[2017-02-14 11:37:52] sudo tc qdisc del dev enp0s25 root

and now tsproxy:

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tsproxy --connectivity.profile cable https://google.de                           
[2017-02-14 11:39:17] Running chrome for url: https://google.de
[2017-02-14 11:39:17] Start tsproxy:/home/bewo/Projects/browsertime/vendor/tsproxy.py --rtt 28 --inkbps 5000 --outkbps 1000 -p 1080
[2017-02-14 11:39:17] Testing url https://google.de run 1
[2017-02-14 11:39:39] Testing url https://google.de run 2
[2017-02-14 11:40:01] Testing url https://google.de run 3
[2017-02-14 11:40:21] 21 requests, 686.31 kb, firstPaint: 3.25s (±170.98ms), DOMContentLoaded: 4.15s (±152.73ms), Load: 18.24s (±435.84ms), rumSpeedIndex: 4432 (±173.23) (3 runs)
[2017-02-14 11:40:21] Wrote data to browsertime-results/google.de/2017-02-14T113916+0100

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tsproxy --connectivity.profile 3g https://google.de  
[2017-02-14 11:40:33] Running chrome for url: https://google.de
[2017-02-14 11:40:33] Start tsproxy:/home/bewo/Projects/browsertime/vendor/tsproxy.py --rtt 300 --inkbps 1600 --outkbps 768 -p 1080
[2017-02-14 11:40:33] Testing url https://google.de run 1
[2017-02-14 11:42:13] Testing url https://google.de run 2
[2017-02-14 11:43:49] Testing url https://google.de run 3
[2017-02-14 11:44:54] 15 requests, 667.13 kb, firstPaint: 21.62s (±1.46s), DOMContentLoaded: 24.23s (±1.s), Load: 83.92s (±7.68s), rumSpeedIndex: 25016 (±1695.36) (3 runs)
[2017-02-14 11:44:54] Wrote data to browsertime-results/google.de/2017-02-14T114033+0100

So, recap: lets use DOMContentLoaded as an example metric here:

Baseline: 725ms (±20.83ms)
tc (cable): 959ms (±8.02ms)
tsproxy(cable): 4.15s (±152.73ms)
Difference: > 4x

tc(3g): 3.84s (±44.82ms)
tsproxy(cable): 24.23s (±1.s)
Difference: > 6x

Surely, this is only a spot check with little significance, but this results alone are alarming.
The following questions arise:
a) Which is the more realistic backend to use?
b) Why is tsproxy generally slower?

[soulgalore]

Hey @worenga this is tracked in #229, there's a problem running the TSProxy with Selenium. You should use TC (that is default if you use Docker) for now. The strange thing is that if I run Chrome/Firefox standalone with TSProxy it works fine (=the connectivity numbers are more or less realistic) but doing the same through Selenium makes the proxy super slow, so the numbers are not realistic at all.

Best
Peter

[worenga]

Thanks for the advice. I cannot seem to get Browsertime to connect to the proxy, when i start it outside manually, neither --proxy.http 127.0.0.1:1080 nor --proxy.https 127.0.0.1:1080 nor --chrome.args="--proxy-server='127.0.0.1:1080'" seems to work.

Furthermore, i was woundering regarding tc, how the uplink bandwith limitation is realized. It seems that it isnt.

[worenga]

Got it to work using
--chrome.args='--proxy-server=socks5://127.0.0.1:1080' --chrome.args='--host-resolver-rules="MAP * ~NOTFOUND, EXCLUDE localhost"'

Anyhow, regardless of whether i start the proxy inside browsertime or outside of browsertime the timings do still differ:

Baseline (since i changed locations since last time):

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js https://google.de
[2017-02-14 15:22:32] Running chrome for url: https://google.de
[2017-02-14 15:22:32] Testing url https://google.de run 1
[2017-02-14 15:22:35] Testing url https://google.de run 2
[2017-02-14 15:22:39] Testing url https://google.de run 3
[2017-02-14 15:22:43] 21 requests, 752.17 kb, firstPaint: 430ms (±22.85ms), DOMContentLoaded: 476ms (±18.97ms), Load: 964ms (±23.10ms), rumSpeedIndex: 443 (±14.61) (3 runs)

3g:
In another terminal i started:

➜  browsertime git:(261-detect-push) ✗ python vendor/tsproxy.py --rtt 300 --inkbps 1600 --outkbps 768

then

[2017-02-14 15:22:43] Wrote data to browsertime-results/google.de/2017-02-14T152231+0100
➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --chrome.args='--proxy-server=socks5://127.0.0.1:1080' --chrome.args='--host-resolver-rules="MAP * ~NOTFOUND, EXCLUDE localhost"' https://google.de 
[2017-02-14 15:22:48] Running chrome for url: https://google.de
[2017-02-14 15:22:48] Testing url https://google.de run 1
[2017-02-14 15:23:50] Testing url https://google.de run 2
[2017-02-14 15:24:47] Testing url https://google.de run 3
[2017-02-14 15:25:43] 17 requests, 668.19 kb, firstPaint: 14.52s (±394.37ms), DOMContentLoaded: 16.s (±424.95ms), Load: 55.44s (±1.19s), rumSpeedIndex: 17096 (±450.70) (3 runs)
[2017-02-14 15:25:43] Wrote data to browsertime-results/google.de/2017-02-14T152248+0100

with tc:

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tc --connectivity.profile 3g https://google.de --connectivity.tc.device enp0s25
[2017-02-14 15:26:20] Running chrome for url: https://google.de
[2017-02-14 15:26:20] sudo tc qdisc add dev enp0s25 root netem delay 300ms loss 0% rate 1600kbps
[2017-02-14 15:26:24] Testing url https://google.de run 1
[2017-02-14 15:26:34] Testing url https://google.de run 2
[2017-02-14 15:26:45] Testing url https://google.de run 3
[2017-02-14 15:26:55] 21 requests, 752.39 kb, firstPaint: 3.08s (±5.72ms), DOMContentLoaded: 3.44s (±0.71ms), Load: 7.58s (±20.65ms), rumSpeedIndex: 3392 (±2.16) (3 runs)
[2017-02-14 15:26:55] Wrote data to browsertime-results/google.de/2017-02-14T152619+0100
[2017-02-14 15:26:55] sudo tc qdisc del dev enp0s25 root

cable:
restarted proxy with

➜  browsertime git:(261-detect-push) ✗ python vendor/tsproxy.py --rtt 28 --inkbps 5000 --outkbps 1000  

then

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --chrome.args='--proxy-server=socks5://127.0.0.1:1080' --chrome.args='--host-resolver-rules="MAP * ~NOTFOUND, EXCLUDE localhost"' https://google.de 
[2017-02-14 15:27:33] Running chrome for url: https://google.de
[2017-02-14 15:27:33] Testing url https://google.de run 1
[2017-02-14 15:27:49] Testing url https://google.de run 2
[2017-02-14 15:28:05] Testing url https://google.de run 3
[2017-02-14 15:28:20] 21 requests, 686.16 kb, firstPaint: 2.33s (±82.53ms), DOMContentLoaded: 3.01s (±88.48ms), Load: 13.06s (±301.85ms), rumSpeedIndex: 3209 (±90.94) (3 runs)
[2017-02-14 15:28:20] Wrote data to browsertime-results/google.de/2017-02-14T152732+0100

and for tc:

➜  browsertime git:(261-detect-push) ✗ ./bin/browsertime.js --connectivity.engine tc --connectivity.profile cable https://google.de --connectivity.tc.device enp0s25                    
[2017-02-14 15:29:10] Running chrome for url: https://google.de
[2017-02-14 15:29:10] sudo tc qdisc add dev enp0s25 root netem delay 28ms loss 0% rate 5000kbps
[2017-02-14 15:29:11] Testing url https://google.de run 1
[2017-02-14 15:29:15] Testing url https://google.de run 2
[2017-02-14 15:29:19] Testing url https://google.de run 3
[2017-02-14 15:29:23] 21 requests, 751.95 kb, firstPaint: 630ms (±16.88ms), DOMContentLoaded: 683ms (±19.73ms), Load: 1.41s (±35.32ms), rumSpeedIndex: 701 (±18.30) (3 runs)
[2017-02-14 15:29:23] Wrote data to browsertime-results/google.de/2017-02-14T152910+0100
[2017-02-14 15:29:23] sudo tc qdisc del dev enp0s25 root

baseline DOMContentLoaded event: 476ms (±18.97ms)

3g DOMContentLoaded event (tsproxy): 16.s (±424.95ms)
3g DOmContentLoaded event (tc): 3.44s (±0.71ms)
Difference: 4,6x

cable DOMContentLoaded event (tsproxy): 3.01s (±88.48ms)
cable DOMContentLoaded event (tc): 683ms (±19.73ms)
Difference: ~5x

[soulgalore]

Yep, that's the same I got before, but if I start the browser standalone (not using Browsertime and Selenium) the TSProxy numbers are better and more inline with TC both for FF and Chrome. Check catchpoint/WebPageTest.tsproxy#10. If you test a site with more requests, the TSProxy number will be more off. My feeling is that something goes wrong running with Selenium but I wasn't able to understand what, if you have any input that would be great!

[worenga]

I see. I saw t hat we are setting the socks proxy manually through an argument/preference, i tried to use this interface here: https://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/proxy.html
However, i cant get it to use the socks proxy. It seems like it is just ignored.

[soulgalore]

I think it could be that NodeJS version of Selenium doesn't support SOCKS proxy yet but according to this it should: https://github.com/SeleniumHQ/selenium/blob/master/javascript/node/selenium-webdriver/CHANGES.md#v300-beta-3

When we set it up in Browsertime we use the CLI flags.

[tobli]

I think (from a quick read) that it should be fixed in the upcoming Firefox 52: mozilla/geckodriver#97

[worenga]

I also noticed that in the current version for tc the values are exchanged.
https://github.com/sitespeedio/browsertime/blob/master/lib/support/connectivity.js#L103
limiting the bandwidth for eth0 is limiting the upstream not the downstream.

Limiting the ingress traffic is a little more complicated, as one would need to setup an bridge interface using the ifb module as follows:

#modprobe ifb0
#ip link set ifb0 up
#tc qdisc add dev eth0 ingress
#tc filter add eth0 parent fff: protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0
#tc qdisc add dev eth1 root netem delay 25ms rate 16000kbit loss 0%

[soulgalore]

hmm yes that looks strange. There was a lot of issues with getting tc to work but you are right this seems wrong.

I think you need to run modprope outside the container so we need to check if there's another way 2do it. There's a lot of different versions if you check the history in https://github.com/sitespeedio/sltc/blob/master/lib/sltc.js

I think previous version worked in Docker on Mac but not Ubuntu or if it was the other way around.

[worenga]

Apart from modprobe, we could also use a virtual network namespace as in
here: https://github.com/phuedx/gee/blob/master/gee#L10
see also sitespeedio/sltc#3 again

[soulgalore]

Network namespaces sounds cool, not sure exactly how to integrate it, a PR would be super :)

[worenga]

Yes, the problem i see is that we somehow need to execute the SeleniumRunner inside a network namespace now (ip netns exec ...). I am not sure how to integrate this into the existing code. Ideally, we would need some kind of wrapper/decorator for SeleniumRunner.

[soulgalore]

Been thinking and I think the easiest way first to get better connectivity is to use modprobe ifb0 outside the container. It will not work on Mac but Linux if I remember correctly but we could catch the error and add a warning. We could rollback to https://github.com/sitespeedio/sltc/blob/d6ebbf5de5345057da63b213838a4111d516b93e/lib/sltc.js , clean it up and then when running docker on Linux make sure that mode´probe run before the container starts? ping @beenanner

[beenanner]

Yea not ideal, but the goal is to have something that works. ;-)

[soulgalore]

I tried it but https://github.com/sitespeedio/sltc/blob/d6ebbf5de5345057da63b213838a4111d516b93e/lib/sltc.js didn't work for me in the container. Modified according to sitespeedio/sltc#3 (comment) and then all commands goes through but speedtest doesn't work for me.

[soulgalore]

The only thing I could get to kind of work (in Docker) is https://gist.github.com/jterrace/1895081 but locally I see some diffs in the download speed, but it could be that I don't configure it right. @worenga do you have any input? I can push a branch today so it's easier to try out.

[soulgalore]

or hmm maybe it doesn't work, I need to test more tonight.

[worenga]

@soulgalore if you have a test branch i can look into it on the weekend.

[soulgalore]

@beenanner has tested a lot different solutions (and I've tried some too) and it seems like a bridge network for Docker can do the work but we still need more testing and find a solution to the problem of uploads if set the network to the full Docker container.

[beenanner]

Here is my steps with the bridge setup to test.

Pull latest master of sitespeedio/sltc from github and docker build it.

 docker build -t sitespeedio/sltc .

Create the 3g test network

docker network create --driver bridge --subnet=192.168.33.0/24 --gateway=192.168.33.10 --opt "com.docker.network.bridge.name"="docker1" 3gtest

tc qdisc del dev docker1 root
tc qdisc add dev docker1 root handle 1: htb default 12
tc class add dev docker1 parent 1:1 classid 1:12 htb rate 1mbit ceil 1mbit
tc qdisc add dev docker1 parent 1:12 netem delay 100ms

This creates a bridge where download is limited to 1 mbps and has a 100ms latency.

docker run -it --network 3gtest --entrypoint /bin/bash sitespeedio/sltc

speedtest-cli

exit

now test with the normal bridge to verify

docker run -it --entrypoint /bin/bash sitespeedio/sltc

speedtest-cli

exit

[soulgalore]

Keep track of the TS problems in #229 and I've updated the docs for how to setup Docker networks.