This repository contains samples for demonstrating Sauce Performance features. The examples are written in Node.js and WebdriverIO but can be transformed into any language or any framework.
To be able to run the examples, you need to install some dependencies via:
$ npm installAll of this can be integrated into a CI pipeline to run performance test as part of your CI/CD cycle. Please have a look at our performance-CI-demo that is using Circle CI as continous integration platform.
Speedo is our Performance side kick that allows to use Sauce Performance without having to setup anything. You can install it on your machine using NPM:
$ npm i -g speedoor use our Docker image if used in a Docker based pipeline. More information can be found here.
Speedo can be used to test the performance of a specific URL with just one command line execution. It takes a care of everything necessary from creating the baseline up to asserting against it and downloading log artifacts at the end. You can Speedo by typing the following into the command line of your terminal:
$ speedo run https://www.saucedemo.com -u <USERNAME> -k <ACCESS_KEY>If you don't want to set username and access key everytime you call the command you can also just export these into your environment as follows:
export SAUCE_USERNAME=<USERNAME>
export SAUCE_ACCESS_KEY=<ACCESS_KEY>
$ speedo run https://www.saucedemo.comSpeedo provides a variety of parameters that tweak the environment you run your tests in. Have a look into our documentation for more details. The following example runs a Speedo test with the name "my Speedo test" in our EU datacenter and only checks for speedIndex and timeToInteractive metrics.
$ speedo run https://www.saucedemo.com --region eu --metric speedIndex --metric timeToInteractive --name "my Speedo test"Once the command is being executed Speedo shows an interactive interface that provides information on what it is currently doing and shows all metrics as well as a link to the Sauce Performance report page:
Note: By default Speedo runs its test in an emulated mobile environment that throttles the network to a "Good 3G" connection and the CPU by 4. This by design as it better helps us to detect performance problems between test runs. It can be disabled by setting throttleNetwork and throttleCpu parameters.
While Speedo is a great and simple tool to test the performance for a specific URL there are often scenarios that require more than just opening a web application. If you want to bring your application into a certain state (e.g. log-in or anykind of prior interaction) you can build your own custom automation script to test the performance of specific page transtion (hard and soft page transition). In flow.check.js you can find a simple flow where we log into our demo application, open a specific product, put that into our shopping card and open that shopping card at the end. For every page transtion performance metrics are being captured and tested. You can run the example by calling:
$ node ./flow.check.jsMake sure you have your Sauce credentials stored in the environment via:
export SAUCE_USERNAME=<USERNAME>
export SAUCE_ACCESS_KEY=<ACCESS_KEY>After every page load we use a custom Sauce command to assert the performance of the last page load to ensure that the captured metrics are withing the upper and lower boundary of our baseline:
let result = await browser.assertPerformance(JOB_NAME, ['load', 'speedIndex'])
assert.equal(result.result, 'pass', 'Performance test for opening main page did not pass')Since WebdriverIO supports Sauce Labs custom performance commands you can call the command like this directly. If you use other frameworks you have to the JSExecutor function. For more information on that, check out our wiki page.
You can also make strict assertion in the metrics. For example given your requirements are that your page load is not longer than 5 seconds total. For that you can use the log command to receive the captured metrics and assert them directly:
const metrics = await browser.execute('sauce:log', { type: 'sauce:performance' })
assert.ok(metrics.load < 5000)The log command returns an object with the following metrics:
{
estimatedInputLatency: 16,
timeToFirstByte: 7,
domContentLoaded: 5190,
firstVisualChange: 5277,
firstPaint: 5399,
firstContentfulPaint: 5399,
firstMeaningfulPaint: 5399,
lastVisualChange: 5605,
firstCPUIdle: 5399,
firstInteractive: 5399,
load: 5416,
speedIndex: 5404
}Note: our demo application has an obvious performance flaw in its inventory page. If you compare all captured Lighthouse Performance Scores you will see that the score dropped to 67% and the metrics have drastically decreased. In order to get an idea where the problem comes you can inspect the tracelog by clicking on the "View Trace" button on the performance details page. You can then drill into every function that has been executed on the page:
Sauce Labs capture performance metrics for soft as well as for hard page transitions. The example describe above shows how you can measure performance when the browser is loading a complete new page every time a page transition happens. However there are pages were the transition is soft meaning that there is no new page being loaded in the browser execution context. In the spa.check.js we are doing an order on Postmates.com. Here, besides of the initial page load, all page transitions are soft. For these we capture the same metrics as for hard page transitions with the exection of the load event duration:
{
estimatedInputLatency: 17,
timeToFirstByte: 12,
domContentLoaded: 508,
firstVisualChange: 0,
firstPaint: 62,
firstContentfulPaint: 62,
firstMeaningfulPaint: 4431,
lastVisualChange: 4980,
firstCPUIdle: 4471,
firstInteractive: 4471,
speedIndex: 2774
}Additionally some of the metric can be 0 (e.g. firstVisualChange) depending on how your application is initiating the transition.
Next to being able to test soft and hard page transitions Sauce Labs provides the ability to check the smoothness feel of a page. Whenever the user feels a stuttering or a juddering on a page it results in a bad performance experience as it seems that the browser is really busy computing the website. Sauce Labs has created a new custom command that allows to test against this. When being called, it scrolls for 5 seconds from the top to the bottom of the page and captures enough information to make an assumption on the jankiness. In jankiness.check.js you see an example of such an application. You can call the sample script by calling:
$ node ./jankiness.check.jsThe test opens a Google sample website where you can create a janky experience by adding Chromium images on the screen. As more images move on the screen as janky it will get. The command returns and object with the result of the test that you can use to assert:
const result = await browser.execute('sauce:jankinessCheck')
assert.ok(result.score > 0.9)As jankiness check captures the following metrics:
{
url: 'https://googlechrome.github.io/devtools-samples/jank/',
timestamp: 1568300525146,
loaderId: '4885adf0-d56e-11e9-aa53-e52a30c71d6a',
score: 0.36234456403004955,
value: {
metrics: {
averageFPS: 17.137733159928153,
scriptingTime: 2141,
renderingTime: 419,
otherTime: 1164,
idleTime: 1196,
forcedReflowWarningCounts: 21416,
scrollTime: 5148,
paintingTime: 228,
memoryUsageDiff: 9062620
},
diagnostics: {
layoutUpdateScore: 0.41371003066141043,
fpsScore: 0.28562888599880254,
idleDurationScore: 0.23232323232323232,
memoryUsageScore: 0.9818905494126537
}
}
}When observing the "jankiness" of a page we are looking into the following captured information:
- The average frames per seconds (FPS): like in a computer game or in videos the frames per second is a measurement of how many images (called frames) appear on a display in one second. As a general rule of thumb a good value of it are 60 FPS
- Idle duration of the browser: if the browser has a lot of things to do while scrolling it is a good indicator that this will influence the performance in a negative way, so as more the browser is not doing any rendering, scripting or painting work the better
- Forced reflows: if you request certain information of a DOM element in a specific way it forces the browser to render it on the screen even though the element has not changed at all. If this happens often, it also results in a bad performance experience
- Memory Usage: storing variables and data in memory is a common practice and in JavaScript usually cleaned by the garbage collector. However if you assign variables in a wrong way it can lead to the sitation that it will never be cleaned up and causes the browser to consume to much memory
For every area Sauce Labs creates a score. For example a average FPS rate of 60 results in a full score of 100%. All score (see in diagnostics) we generate a weighted mean to create one single jankiness performance score. This can used as a general measure for how well the smoothness of the page feels.

