Your browser, your page, your scripts!
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github new issue template Jun 30, 2018
bin changed logging to avoid saying which URL that is tested Dec 14, 2018
browserscripts Support for testing multiple pages (#685) Dec 13, 2018
browsersupport/firefox-profile use the certificate for WebPageReplay for Firefox Mar 2, 2018
docker remove setting adb port Jun 14, 2018
docs Fix text error about postScript Sep 7, 2017
lib generate logs on verbose for VM Dec 15, 2018
test Support for testing multiple pages (#685) Dec 13, 2018
vendor Log when we got problems splitting the video. (#689) Dec 14, 2018
.dockerignore Add a Dockerfile for running Browsertime. Aug 31, 2016
.eslintignore Use pre-compiled WPR to minimise the size of the container (#443) Feb 14, 2018
.eslintrc.json Strings instead of magic number for eslint config. Apr 4, 2018
.gitignore package locks are ok Nov 7, 2017
.travis.yml Support for testing multiple pages (#685) Dec 13, 2018
CHANGELOG.md changelog: simplify android Dec 14, 2018
CONTRIBUTING.md new path to issues [skip ci] Oct 6, 2013
Dockerfile FF 64 Dec 10, 2018
LICENSE first commit Aug 23, 2013
README.md stop measure is better name than collect Dec 13, 2018
appveyor.yml Install java on AppVeyor CI. Oct 14, 2015
browsertime.png hello world Jun 23, 2016
circle.yml Initial configuration for tests on CircleCI. Mar 31, 2016
index.js Make engine easier to understand. (#454) Feb 24, 2018
package-lock.json new chrome har Dec 13, 2018
package.json new chrome har Dec 13, 2018
release.sh disable latest/config/doc updates when we are on alpha/beta releases Nov 23, 2018

README.md

Browsertime - Your browser, your page, your scripts!

Build status Downloads Downloads total Stars

Documentation | Changelog

Browsertime

Access the Web Performance Timeline, from your browser, in your terminal!

Introduction

Browsertime lets you automate running JavaScript in your browser primary used to collect performance metrics. What exactly does that mean?

We think of a Browsertime as having four key capabilities:

  • It handles everything with the browser (Firefox/Chrome).
  • It executes a batch of default and configurable JavaScript when the URL has finished loading in the browser.
  • It records a video of the Browser screen used to calculate Visual Metrics.
  • It lets you run Selenium scripts before and after the browser access the URL (to login a user etc).

What is Browsertime good for?

It is usually used for two different things:

  • You run it as a standalone tool to collect performance timing metrics of your web site.
  • You integrate it in your tool as a JavaScript runner that collects whatever JavaScript metrics/information you want.

To understand how Browsertime does these things, let's talk about how it works. Here's an example of what happens when you give Browsertime a URL to test:

  1. You give your configuration to Browsertime.
  2. Browsertime uses the WebDriver (through Selenium) to start Firefox and Chrome (the implementations for the Webdriver is Chromedriver/Geckodriver).
  3. Browsertime starts FFMPEG to record a video of the browser screen
  4. The browser access the URL.
  5. When the page is finished loading (you can define yourself when that happens), Browsertime executes the default JavaScript timing metrics and collects:
  6. It also collects a HAR file that shows all requests/responses on the page.
  7. FFMpeg is stopped and the video is analysed. Browsertime collect Visual Metrics like Speed Index.

The result of the run is a JSON file with all the JavaScript metrics collected, a HAR file, a video recording of the screen and a screenshot.

A simple example

Use our Docker image (with Chrome, Firefox, XVFB and the dependencies needed to record a video):

$ docker run --shm-size=1g --rm -v "$(pwd)":/browsertime sitespeedio/browsertime https://www.sitespeed.io/

Or using node:

$ npm install browsertime -g
$ browsertime https://www.sitespeed.io/

Load https://www.sitespeed.io/ in Chrome three times. Results are stored in a JSON file (browsertime.json) with the timing data, and a HAR file (browsertime.har) in browsertime-results/www.sitespeed.io/$date/

I want more examples

Checkout the examples.

Browsers

Browsertime supports Firefox and Chrome on desktop. On Android we support Chrome.

But we want to support Opera (on Android) and when(?!) iOS Safari supports WebDriver we will add that too.

How does it work

Browsertime uses Selenium NodeJS to drive the browser. It starts the browser, load a URL, executes configurable Javascripts to collect metrics, collect a HAR file.

To get the HAR from Firefox we use the HAR Export Trigger and Chrome we use Chrome-HAR to parse the timeline log and generate the HAR file.

Speed Index and video

It's easiest to run our ready made Docker container to be able to record a video and calculate SpeedIndex because then you get all dependencies needed for free to run VisualMetrics.

The default video will include a timer and showing when the metrics happens, but you can turn that off using --video.addTimer false.

Test using Docker

You can build and test changes using Docker locally.

$ docker build -t sitespeedio/browsertime .
$ docker run --shm-size=1g --rm -v "$(pwd)":/browsertime sitespeedio/browsertime -n 1 https://www.sitespeed.io/

Connectivity

You can throttle the connection to make the connectivity slower to make it easier to catch regressions. The best way to do that is to setup a network bridge in Docker or use our connectivity engine Throttle.

Here's an full example to setup up Docker network bridges on a server that has tc installed:

#!/bin/bash
echo 'Starting Docker networks'
docker network create --driver bridge --subnet=192.168.33.0/24 --gateway=192.168.33.10 --opt "com.docker.network.bridge.name"="docker1" 3g
tc qdisc add dev docker1 root handle 1: htb default 12
tc class add dev docker1 parent 1:1 classid 1:12 htb rate 1.6mbit ceil 1.6mbit
tc qdisc add dev docker1 parent 1:12 netem delay 150ms

docker network create --driver bridge --subnet=192.168.34.0/24 --gateway=192.168.34.10 --opt "com.docker.network.bridge.name"="docker2" cable
tc qdisc add dev docker2 root handle 1: htb default 12
tc class add dev docker2 parent 1:1 classid 1:12 htb rate 5mbit ceil 5mbit
tc qdisc add dev docker2 parent 1:12 netem delay 14ms

docker network create --driver bridge --subnet=192.168.35.0/24 --gateway=192.168.35.10 --opt "com.docker.network.bridge.name"="docker3" 3gfast
tc qdisc add dev docker3 root handle 1: htb default 12
tc class add dev docker3 parent 1:1 classid 1:12 htb rate 1.6mbit ceil 1.6mbit
tc qdisc add dev docker3 parent 1:12 netem delay 75ms

docker network create --driver bridge --subnet=192.168.36.0/24 --gateway=192.168.36.10 --opt "com.docker.network.bridge.name"="docker4" 3gem
tc qdisc add dev docker4 root handle 1: htb default 12
tc class add dev docker4 parent 1:1 classid 1:12 htb rate 0.4mbit ceil 0.4mbit
tc qdisc add dev docker4 parent 1:12 netem delay 200ms

Then when you run your container you add the network with --network cable. You should also tell Browsertime that you set the connectivity external from BT. A full example running running with cable:

$ docker run --shm-size=1g --network=cable --rm sitespeedio/browsertime -c cable --connectivity.engine external https://www.sitespeed.io/

And using the 3g network:

$ docker run --shm-size=1g --network=3g --rm sitespeedio/browsertime -c 3g --connectivity.engine external https://www.sitespeed.io/

And if you want to remove the networks:

#!/bin/bash
echo 'Stopping Docker networks'
docker network rm 3g
docker network rm 3gfast
docker network rm 3gem
docker network rm cable

Throttle uses tc on Linux and pfctl on Mac to change the connectivity. Throttle will need sudo rights for the user running sitespeed.io to work.

To use throttle, use set the connectivity engine by --connectivity.engine throttle.

browsertime --connectivity.engine throttle -c cable https://www.sitespeed.io/

You can also use Throttle inside of Docker but then the host need to be the same OS as in Docker. In practice you can only use it on Linux. And then make sure to run sudo modprobe ifb numifbs=1 first and give the container the right privileges --cap-add=NET_ADMIN.

Script navigation [in coming 4.0 or later]

If you need a more complicated test scenario, you can define your own (Selenium)test script that will do the testing. Use your own test script when you want to test your page as a logged in user, the login page or if you want to add things to your cart.

You run your navigation script with --scriptNavigation.

The context object:

  • url - The URL that you want are under test
  • options - All the options sent from the CLI to Browsertime
  • log - an instance to the log system so you can log from your navigation script
  • index - the index of the runs, so you can keep track of which run that is running
  • storageManager - The Browsertime storage manager that can help you get read/store files to disk.
  • webdriver - The Selenium WebDriver object

The helper object got three methods that you can use:

  • navigate(URL) - Use this if you want to use the exact way as Browsertime navigates to a new URL (same settings with pageCompleteCheck etc). But that URL will not be measured automatically.
  • measure(URL) - Start measuring and navigate to a new page in one go and measure.
  • startMeasure() - Use this when you want to start to measure a page. This will start the video and prepare everything to collect metrics.
  • collect() - Collect metrics for a page.

The really simple version looks like this:

module.exports = {
  run(context, help) {
    return context.runWithDriver(async function() {
      context.log.info('Running script navigation');
      return help.measure('https://www.sitespeed.io/');
    });
  }
};

Testing a page after you have logged in:

module.exports = {
  run(context, help) {
     return context.runWithDriver(async function() {
        await help.navigate(
        'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page');
        // we fetch the selenium webdriver from context
        const webdriver = context.webdriver;
        // and get hold of some goodies we want to use
        const until = webdriver.until;
        const By = webdriver.By;
        // before you start, make your username and password
        const userName = 'USERNAME';
        const password = 'PASSWORD';
        driver.findElement(By.id('wpName1')).sendKeys(userName);
        driver.findElement(By.id('wpPassword1')).sendKeys(password);
        const loginButton = driver.findElement(webdriver.By.id('wpLoginAttempt'));
        loginButton.click();
        // we wait for something on the page that verifies that we are logged in
        await driver.wait(until.elementLocated(By.id('pt-userpage')), 6000);
        // You are now logged in, navigate to the page that we want to measure
        return help.measure(context.url);
    });
  }
};

And a example measuring the actual log in step:

module.exports = {
  run(context, help) {
     return context.runWithDriver(async function() {
        await help.navigate(
            'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page');
        // we fetch the selenium webdriver from context
        const webdriver = context.webdriver;
        // and get hold of some goodies we want to use
        const until = webdriver.until;
        const By = webdriver.By;
        // before you start, make your username and password
        const userName = 'USERNAME';
        const password = 'PASSWORD';
        driver.findElement(By.id('wpName1')).sendKeys(userName);
        driver.findElement(By.id('wpPassword1')).sendKeys(password);
        const loginButton = driver.findElement(webdriver.By.id('wpLoginAttempt'));
        // Before we click on the login button, start the measurement
        await help.startMeasure();
        // Login the user
        loginButton.click();
        // we wait for something on the page that verifies that we are logged in
        await driver.wait(until.elementLocated(By.id('pt-userpage')), 6000);
        // Make sure to remember to collect the metrics
        await helper.stopMeasure();
    });
  }
};

And test multiple pages:

module.exports = {
  run(context, help) {
    return context.runWithDriver(async function() {
      await help.measure('https://www.sitespeed.io');
      await help.measure('https://www.sitespeed.io/examples/');
      return help.measure('https://www.sitespeed.io/documentation/');
    });
  }
};

Test on your mobile device

Browsertime supports Chrome on Android: Collecting SpeedIndex, HAR and video! This is still really new, let us know if you find any bugs.

You need to install adb and prepare your phone before you start.

If you want to set connectivity you need to use something like Micro device lab or TSProxy.

$ browsertime --chrome.android.package com.android.chrome https://www.sitespeed.io --video --visualMetrics

If you are on Linux (we have tested Ubuntu 16) you can use our Docker container to drive your Android phone. A couple of things to remember:

  • You need to run in privileged mode --privileged
  • You need to share the USB ports -v /dev/bus/usb:/dev/bus/usb
  • Add -e START_ADB_SERVER=true to start the adb server
  • Turn of xvfb --xvfb false (we start that automatically)

If you use Docker you will automatically get support for video and SpeedIndex. You can get that without Docker but then need to install VisualMetrics dependencies yourself.

$ docker run --privileged -v /dev/bus/usb:/dev/bus/usb -e START_ADB_SERVER=true --shm-size=1g --rm -v "$(pwd)":/browsertime-results sitespeedio/browsertime -n 1 --chrome.android.package com.android.chrome --xvfb false --visualMetrics --video https://en.m.wikipedia.org/wiki/Barack_Obama

Configuration

Run $ bin/browsertime.js --help and you can see the configuration options.

Using WebPageReplay

Our Docker container now included WebPageReplay.

WebPageReplay will let you replay your page locally (getting rid of server latency etc) and makes it easier to find front end regressions.

It works like this:

  1. The start script starts WebPageReplay in record mode
  2. Then starts Browsertime accessing the URL you choose one time (so it is recorded)
  3. WebPageReplay is closed down
  4. WebPageReplay in replay mode is started
  5. Browsertime access the URL so many times you choose
  6. WebPageReplay in replay mode is closed down

You can change latency by setting a Docker environment variable. Use REPLAY to turn on the reply functionality.

Default browser is Chrome:

docker run --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime:3.0.0 https://en.wikipedia.org/wiki/Barack_Obama

Use Firefox:

docker run --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime:3.0.0 -b firefox -n 11 https://en.wikipedia.org/wiki/Barack_Obama

And Chrome on your Android phone. This will only work on Linux because you need to be able to mount the usb port in Docker:

docker run --privileged -v /dev/bus/usb:/dev/bus/usb -e START_ADB_SERVER=true --cap-add=NET_ADMIN --shm-size=1g --rm -v “$(pwd)“:/browsertime -e REPLAY=true -e LATENCY=100 sitespeedio/browsertime https://en.m.wikipedia.org/wiki/Barack_Obama --chrome.android.package com.android.chrome --xvfb false --chrome.args ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I= -n 11 --chrome.args user-data-dir=/data/tmp/chrome

Send metrics to Graphite

The easiest way to send metrics is to install jq and use it to pick the values you wanna track.

Here's an example on how you can pickup the median SpeedIndex from Browsertime and send it to your Graphite instance.

echo "browsertime.your.key.SpeedIndex.median" $(cat /tmp/browsertime/browsertime.json | jq .statistics.visualMetrics.SpeedIndex.median) "`date +%s`" | nc -q0 my.graphite.com 2003