HLS BBA testbed
- NEVER never never run multiple instances of
vagrant upin parallel: machines will mix and everything will break.
- VirtualBox 4.3
- Vagrant 1.8.1
- Python 2.7.8: if this is not your default version, you can install pyenv and run
pyenv install 2.7.8.
- matplotlib 1.3.1 and scipy 0.14.0:
- if you're using system-wide Python in Ubuntu:
apt-get install python-matplotlib python-scipy
- if not:
pip install -r requirements.txt(Ubuntu packages you'll probably need: python-pip pkg-config python-dev libfreetype6-dev libpng12-dev)
- if you're using system-wide Python in Ubuntu:
- tcptrace 6.6.7, tshark 1.10.6: to plot
- GNU parallel (>20130522) Other versions could work.
- First, install dependencies.
- Clone this repository and our customized version of VLC media player:
git clone https://github.com/serl/hls-bba-testbed.git && cd hls-bba-testbed && git submodule update --init.
- The testbed comes without videos; in order to get the same version of Big Buck Bunny that we used (~800MB), run:
wget https://s3-eu-west-1.amazonaws.com/hls-bba-testbed/bbb8.tar.gz && tar -C code/http_server/static -xzf bbb8.tar.gz && rm bbb8.tar.gz. Clients will have access through the bottleneck to whatever is in the
- Create and configure the machines:
vagrant up && vagrant up client0 client1 client2. It will also compile VLC. This will take approximately 30 minutes and 5 GB of disk space. Machines are configured to get at most 2GB of RAM each (so 12GB in total).
Running the tests
- Populate the
testsdirectory by running
- Run one or more tests (in sequence) with
./run_schedule.sh [N] <session_dir> [session_dir...], where
Nis optional and represents the number of runs you want for each test (default
session_diris the directory containing a file named
tests/constant_1_bbb8_80ms_100%BDP_droptail/c01_1_bbb8_600kbit_bba3_keepalive_est). A shortcut to run all tests is
./run_schedule.sh tests/*/*/, but beware: it will take about 25 days.
- The results for each run of each test will be stored in
session_dir/N.tar.gz, for example
- Right after each run, a post_run routine will be launched, analyzing dropped packets and generating a plot about the test. In our example the image path will be
Analyzing the results
- A zoomable plot could be obtained by running
./plot_display.sh <rundir>, where
rundiris the path to a
.tar.gzfile: this will open the matplotlib graphical interface.
./check_tests.sh tests/will tell you if and which runs are missing.
./analyze_vlc_QoE.sh tests/will calculate the averages for some metrics and create some useful csv files:
tests/QoE_metrics/summary.csvby averaging on all the constant-bandwidth bottlenecks
tests/QoE_metrics_variable/v0(1|2)_summary.csvby averaging separately on the two variable-bandwidth bottleneck profiles (with fair shares of 4Mbit-1Mbit-4Mbit-600kbit-4Mbit and 4Mbit-2.8Mbit-1.5Mbit-1Mbit-600kbit-4Mbit).
Tips and tricks
- In order to have a comprehensive plot of multi-client runs:
python plot_vlc_compare_runs.py [export] <session_dir(s)>(and not
run_dir!). It will generate a plot with the bit rate requested and the bandwidth fair share, taking all the clients and all the runs. If more than one
session_diris asked for, images in png format will be exported in
session_dir, otherwise an interactive plot will open (the export behavior can be forced by giving the optional
- You can easily run
iperfby generating tests crafted for it:
python iperf_test.py(take a look at the Python script to see or modify the parameters); after each test, plots will be automatically generated; for interactive plots, you can run
./plot_display.sh <rundir>as usually.
- Update the testbed to the latest version:
./update.sh. It will resync to the repositories and, if wanted, recompile VLC.
- Receive an email at the end of the execution of the tests: create a file named
- Avoid halting all virtual machines after a batch of tests: create an empty file named
- Rerun post_run routine (calculating dropped packets and generating png plots):
- Recompile VLC:
vagrant up client0 && vagrant ssh client0 -c 'bash /vagrant/scripts/compile_vlc.sh update'.
- Run more than three clients and/or change the amount of RAM/CPUs: change the respective values in
Vagrantfile, then configure the new machines by running
vagrant up clientN clientN-1...or reconfigure old machines by halting them with
vagrant halt(the new values will be active at boot).
- You could exploit Python on a virtual machine (not recommended), use
./python.sh, and everything should (slowly) work out of the box.
- You can decompress/recompress individual or a group of runs by running
./(de)compress.sh <rundir(s)>. Un-compressed runs will work as well as compressed ones with (hopefully) all tools.
- Run tests with another video: make sure it is in Apple HLS format, and put it into
code/http_server/staticfolder. In order to make BBA-1+ work properly, you need to modify the HLS *sub-*playlists to include the segment sizes, by running
python hls-size-helper.py <path/to/subplaylist.m3u8> > <path/to/subplaylist_size.m3u8>; then modify (or copy) the master playlist to point to the new sub-playlists.
To be fixed
plot_vlc_scatters.shwas designed to generate (but does not work at the moment) scatterplots similar to the ones you can find in Akhshabi, Saamer, et al. "What happens when HTTP adaptive streaming players compete for bandwidth?" Proceedings of the 22nd international workshop on Network and Operating System Support for Digital Audio and Video. ACM, 2012.
compare_sessions.pywas designed to compare the bit rates obtained by different families with a bar graph; it is not maintained and probably not working anymore.