A Framework for testing web performance between different browser
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
agent remove hasal server dependency Aug 16, 2017
configs add auto clean disk space Dec 4, 2017
ejenti fix gist interface change exception Dec 5, 2017
flows changed flow filename Sep 6, 2016
jenkins remove unused keyword Jun 6, 2017
lib add flag to control gist max limit Dec 5, 2017
python Updating `.gitignore` and `test-tidy` for new local files Nov 7, 2017
scripts Adding temporary varibale `ret_code` to store the return code Jun 6, 2017
server Modify Treeherder related code for 4.0.0 Aug 30, 2017
tests Adding case for Intel Power Gadget - part 2 Nov 17, 2017
thirdParty Merge pull request #958 from ypwalter/sc_request Sep 6, 2017
tools Modify the old backfill-agent due to it will ddos service Nov 21, 2017
unit Addind Archive Information into Progress data for progress dashboard Nov 17, 2017
.gitignore Adding File Logger for Back Fill for debugging Nov 22, 2017
.pyup.yml update readme for pyup Aug 16, 2017
.travis.yml Adding unittest for Pulse related modules, and adding on Travis CI Nov 1, 2017
Makefile Adding unittest for Pulse related modules, and adding on Travis CI Nov 1, 2017
README.md Update README.md Aug 16, 2017
__init__.py Add Mach command for generating Python API documents Feb 14, 2017
appveyor.yml Merge branch 'master' into dev Sep 12, 2017
bootstrap-linux.sh Adding FirefoxNightly support in bootstrap Aug 21, 2017
bootstrap-mac.sh Modify bootstrap_mac.sh due to Homebrew upgrade opencv to opencv3 Aug 21, 2017
bootstrap.bat Modify bootstrap tools for `ejenti` requirements Aug 21, 2017
bootstrap.ps1 Merge branch 'master' into dev Sep 12, 2017
end.txt Create Firefox profile once at the begining of running suite Jan 11, 2017
mach First tidy Aug 30, 2016
requirements.txt Adding unittest for Pulse related modules, and adding on Travis CI Nov 1, 2017
requirements_devops.txt Modify the python packages requirements files Aug 17, 2017
requirements_mac.txt Modify the python packages requirements files Aug 17, 2017
runtest.py add status tag for sikuli type in runtest cmd Dec 6, 2017
setup.py Adding more supported output folder for query/remove Nov 21, 2017
suite.txt refactor facebook half ail cases Jun 8, 2017
suite_webdriver.txt First version of working webdriver for firefox Mar 9, 2017
svrConfig.json Modify server config Aug 5, 2016

README.md

Hasal Performance Test

Master

Master Linux Status Master Windows Status Updates Python 3

Dev

Dev Linux Status Dev Windows Status Updates Python 3

"Hasal" /ha'sɑlu/ this word is came from Indigenous Taiwanese "Bunun", and it means hail. Why we choose this word is because we expect our testing could like the hail, fast and weighty. And through our testing could bring more improvement on our performance or quality.

A Framework for testing web performance between different browser

Automatical Installation

If you do not wish to install Hasal automatically, please proceed to next section "Manual Installation". For those who wants to install Hasal, please be notified that a clean environment or

  • Ubuntu
.\bootstrap-linux.sh
  • Mac OS
.\bootstrap-mac.sh
  • Windows 7 Launch a command line console with administrator privilege.
bootstrap.bat
  • Windows 8 or above Launch a powershell console with administrator privilege.
Set-ExecutionPolicy Unrestricted
.\bootstrap.ps1

Please noted that Hasal in Windows systems can be run from command line only.

Manual Installation

  • Install Firefox
  • Install SikluliX (https://launchpad.net/sikuli/sikulix/1.1.0), place the runsikulix and other installation files in hasal/thirdparty. When sikulix setup diaglog come out, please select the Pack 1 and make sure scripting language: Python is checked.
  • Install video recording codes and libs (windows/mac:ffmpeg or ubuntu:avconv)
  • Install video recording main program (windows/mac:ffmpeg or ubuntu:avconv)
  • Install opencv
  • Download the client certificate from here (https://goo.gl/yfki48 -- note: needs a mozilla.com account ATM), place all certificates in your hasal working dir.
  • Run setup.py

For Ubuntu:

apt-get install virtualenv python-dev
virtualenv ~/.hasalenv            # or "make clean dev-env"
source ~/.hasalenv/bin/activate

pip install selenium

sudo apt-get install wget libav-tools ffmpeg libavc1394-0 libavformat-extra-53 libavfilter2 libavutil-extra-51 mencoder libavahi-common-data xsel xclip
wget https://github.com/bgirard/Gecko-Profiler-Addon/blob/master/geckoprofiler-signed.xpi?raw=true
wget https://github.com/Itseez/opencv/archive/3.0.0.zip
follow this link[http://www.pyimagesearch.com/2015/06/22/install-opencv-3-0-and-python-2-7-on-ubuntu/] to install opencv or:

sudo apt-get install build-essential cmake git pkg-config
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev
sudo apt-get install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libgtk2.0-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libatlas-base-dev gfortran
sudo pip install virtualenv virtualenvwrapper
sudo apt-get install pip
sudo apt-get install python-pip
sudo apt-get install python2.7-dev
pip install numpy
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.1.0
cd ..
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib/
git checkout 3.1.0
cd ../opencv

mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_C_EXAMPLES=OFF -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules -D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig
ln -s /usr/local/lib/python2.7/site-packages/cv2.so virtualenv_path_you_create/lib/python2.7/site-packages/cv2.so

cd PATH_TO_HASAL
python setup.py install

For Mac OS:

virtualenv PATH_TO_YOUR_VENV            # or "make clean dev-env"
source PATH_TO_YOUR_VENV/bin/activate

brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-frei0r --with-libass --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools
brew install libav
brew tap homebrew/science
brew install opencv
ln -s /usr/local//Cellar/opencv/2.4.13/lib/python2.7/site-packages/cv2.so PATH_TO_YOUR_VENV/lib/python2.7/site-packages/cv2.so

cd PATH_TO_HASAL
python setup.py install

OR manual build opencv2

virtualenv PATH_TO_YOUR_VENV           # or "make clean dev-env"
source PATH_TO_YOUR_VENV/bin/activate

download the opencv2 package, compile and install
download the opencv2 package here : https://github.com/Itseez/opencv/archive/2.4.13.zip
  unzip the package 
  cmake the folder unzipped 
  make
  make install
  ln -s /usr/local/lib/python2.7/site-packages/cv2.so PATH_TO_YOUR_VENV/lib/python2.7/site-packages/cv2.so

cd PATH_TO_HASAL
python setup.py install

Usage

Sample

  • Trigger the framework: python runtest.py

Usage

  runtest.py [--exec-config=<str>] [--firefox-config=<str>] [--index-config=<str>] [--online-config=<str>] [--global-config=<str>]
  runtest.py (-h | --help)

Options:

  -h --help | Show this screen.
  --exec-config=<str> | Specify the test execution config file; max-run, max-retry, advance, keep-browser etc. settings can be controlled in here. [default: configs/exec/default.json]
  --firefox-config=<str> | Specify the test Firefox config file; [default: configs/firefox/default.json]
  --index-config=<str> | Specify the index config file; you can specify which index you want to generate here. [default: configs/index/runtimeDctGenerator.json]
  --online-config=<str> | Specify the online config file; you can specify if you want to enable online data submission and other related settings here. [default: configs/online/default.json]
  --global-config=<str> | Specify the global config file; you can modify the output fn and status fn here. [default: configs/global/default.json]

Suite file template

  • regression test case format: test_script_path

    Example: tests.regression.gdoc.test_firefox_gdoc_read_basic_txt_1

  • pilot test case format: test_sikuli_script_path

    Example: tests/pilot/facebook/test_firefox_facebook_load_homepage.sikuli/

About generator

They are all inside configs/index/ directory. Generator is all about what data do we want to generate. We use runtimeDctGenerator.json as the default generator.

Generator What can we get?
runtimeDctGenerator The total running time and related statistics
speedIndexDctGenerator Get the SI/PSI for your test cases
inputLatencyDctGenerator The input latency time for specific action (2 images)
inputLatencyAnimationDctGenerator The input latency time for specific action (1 images)
frameThroughputDctGenerator Get the frame throughput time for specific action

Please be remind that most cases with ail in its naming are using inputLatencyAnimationDctGenerator and some of them are using inputLatencyDctGenerator. You may set it with --index-config= mentioned previously for runtest.py.

Basic Debugging

What if the Hasal framework doesn't work?

  1. Try to make the resolution of your pc to 1920*1080
  2. Try to set about:blank as your browser start-up page
  3. Remember to activate your virtualenv for all the things we set up for you
  4. Try to update your code and rerun bootstrap. For windows, you might want to recover the environment variable PATH from the backup file generated after you ran the bootstrap.bat

VM Template (out-of-date)

You can download the VM tempalte for Hasal framework environment from vagrant.

  • vagrant init shako/hasal
  • vagrant up --provider virtualbox
  • Default user name and password : hasal/hasal