Skip to content
Permalink
main
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time
3144 lines (2166 sloc) 184 KB

Browsertime changelog (we do semantic versioning)

16.11.1 - 2022-06-29

Fixed

  • Fixed --preWarmServer so it works with --headless, Docker and specific Android or iOS device #1815.

16.11.0 - 2022-06-28

Added

  • Firefox 102 and Edge 103 in the Docker container.
  • Use Edgedriver 103 #1812

Fixed

  • Upgraded to Selenium 4.3.0 #1813.

16.10.1 - 2022-06-26

Fixed

  • Verify that the CDP port is free before using it 1811.

16.10.0 - 2022-06-23

Added

  • Upgraded the Docker container to use Chrome 103.
  • Upgraded to Chromedriver 103.

16.9.1 - 2022-06-19

Fixed

  • Upgraded to Throttle 4 that internally uses ip route instead of route (one less dependency) #1767.

16.9.0 - 2022-06-15

Added

  • New wait.byCondition command. Thank you Icecold777 for PR #1803.
  • Collect number of CPU longtasks before largest contentful paint #1806.

Fixed

  • Instead of throwing errors and exit the tests if the page hasn't finished loading after 5 minutes, we now gracefully ends the test after 2 minutes (you can change that time with --maxLoadTime). That makes more sense than just throwing errors #1810.

16.8.1 - 2022-06-09

Fixed

16.8.0 - 2022-06-06

Added

  • Add --debug mode. Debug mode will run your tests and open devtools in Chrome/Edge/Firefox on desktop and will stop after each iteration so you can inspect the page. You can add your own breakpoint in your script with the breakpoint(name) command. To continue after your breakpoint, add the following code in the developer console in your browser: window.browsertime.pause=false; #1798.
  • Use Selenium WebDriver 4.2.0 #1801.
  • Updated to Firefox 101, Edge 102, Chrome 102 in the Docker container.

Tech

  • Added tests using NodeJS 18 #1772.

16.7.0 - 2022-05-20

Added

  • Include last CPU long task in the HAR file so we can show when it happens #1793.
  • Inlclude TTFB and INP in the Google Web Vital namespace #1792.

16.6.0 - 2022-05-20

Added

  • Implemented experimental Interaction to next paint that's useful if you test user journeys #1791.
  • Track when the last CPU long task happen as explained by Andy Davies of the webperf Slack channel #1789.

16.5.0 - 2022-05-11

Added

  • Make it possible to configure the max size in pixels for the filmstrip screenshots using --videoParams.thumbsize #1787.

Fixed

16.4.0 - 2022-05-11

Added

  • If we have the Largest Contentful Paint for the video, add that to the video text instead of DOMContentLoaded #1783.

Fixed

  • Only get LCP in Visual Metrics if the LCP API gives us a element #1786.

16.3.0 - 2022-05-07

Added

16.2.2 - 2022-05-06

Fixed

  • The Docker container uses Ubuntu 20 again (instead of 22) since there's been multiple problems (running on ARM and on some cloud services).

16.2.1 - 2022-05-04

Fixed

  • Add an acceptable error of 5 pixels to last frame matches for the visual metric portable script, thank you Gregory Mierzwinski for PR #1780.

16.2.0 - 2022-05-01

Added

  • Updated to Chrome and Chromedriver 101 #1773.
  • Updated to Edge and Edgedriver 101 #1778.
  • Use Geckodriver 0.31.0 #1775.
  • Added new alias for warm cache load. You can now use either --preURL or --warmLoad #1774.

Fixed

  • Updated to Selenium 4.1.2 #1779.

Tech

  • Updated dev dependencies #1776.

16.1.0 - 2022-04-20

Fixed

  • Handle negative x/y offsets when cropping images in the new portable visual metrocs script, thank you Gregory Mierzwinski for PR #1770.
  • Bumped Throttle dependency #1769.
  • Bumped chrome-har, chrome-remote-interface, dayjs and yargs dependencies #1771.

Added

  • Added blocking of Chrome and Edge phone home domains #1763.

16.0.1 - 2022-04-06

Fixed

  • If visual metrics fails and you also use --visualElements, make sure that file is also stored so we can reporduce the issue #1757.

16.0.0 - 2022-04-05

Changed

  • When I introduced flushing of the DNS per run I missed that it makes you need to run sudo to do it (Mac/Linux). That sucks so instead of being able to disable the flushing --disableDNSFlush you now enables it with --flushDNS #1752.

Added

  • Firefox 99 in the Docker container
  • Edge and Edgedriver 100

Fixed

  • Automatically save the original video if visual metrics fails #1755.

15.4.0 - 2022-03-30

Added

  • Updated to Chrome and Chromedriver 100 #1743.
  • There's a new version of the visual metrics script contributed by Gregory Mierzwinski that removed the dependency of ImageMagick and instead needs the Python dependencies OpenCV-Python and Numpy. The long term goal is to use this as default but for now you can test it with --visualMetricsPortable.

15.3.0 - 2022-03-07

Added

  • Chrome: If you set --user-data-dir you can clean the directory between each iteration using --chrome.cleanUserDataDir #1739

15.2.0 - 2022-03-04

Added

  • Updated to Chromedriver and Edgedriver 99.
  • Updated Docker container to use Chrome and Edge 99 #1738.

Fixed

  • Added more view ports for emulated mobile in Chrome #1736.

15.1.1 - 2022-02-24

Fixed

15.1.0 - 2022-02-24

Added

  • Collect timings from main document. The result includes a field named mainDocumentTimings and contains blocked, dns, connect, send, wait, receive, ssl as long as you get a HAR file from the browser #1735.

Tech

  • Cleaned the DNS flush code #1734.

15.0.1 - 2022-02-21

Fixed

  • Make sure connectivity is only set before the test begins #1733.

15.0.0 - 2022-02-20

Changed

  • Removed the built in support for TSProxy #1718. TSProxy only worked with Python 2.7 and we want to remove that dependency.
  • Removed support for using speedline and the CLI arg --chrome.visualMetricsUsingTrace #1727.

Fixed

  • Code cleanup: #1720
  • Moved test to Ava instead of mocha and moved most testing against a local HTTP server instead of internet #1725.

Added

  • New alias for getting the Chrome trace log: --chrome.trace #1728.
  • On OS X and Linux the DNS will be flushed between runs. You can disable that with --disableDNSFlush true #1732.

14.21.1 - 2022-02-09

Fixed

  • Always disable the new ChromeWhaxyNewUI splash screen in Chrome #1717.
  • Remove the Firefox persist log configuration that was needed for FF 94+ #1716.

14.21.0 - 2022-02-08

Added

  • Upgraded to Egdedriver 98
  • Upgraded to Edge 98 and Firefox 97 in the Docker container

14.20.3 - 2022-02-07

Fixed

  • There's a new setup how we build Docker releases and this release uses the new pipeline.

14.20.0 - 2022-02-07

Added

  • Docker container for both amd64 and arm64.

14.19.2 - 2022-02-05

Fixed

  • Upgraded to Geckodriver 0.30.0 #1710.
  • Upgraded to Chromedriver that always looks for Chromedriver in PATH for arm64.
  • Fixed getting Geckoprofile on Android when you do not specify Android package #1712.

14.19.1 - 2022-02-03

Fixed

  • Remove the old fix for setting CPUThrottling, that works now in Chrome without that fix #1708.
  • Do not set binary for Firefox when run on Android #1706.

14.19.0 - 2022-02-02

Added

  • Chrome and Chromedriver 98 #1704.

Fixed

  • Fix so we get the Firefox version on Android #1704.

14.18.1 - 2022-01-24

Fixed

  • If loading a URL failed and we retry and we logged that as an info message, but you as a user only need to know if it fails after X retries. The log message now logs at debug level #1701.
  • The summary log message [2022-01-24 16:12:38] INFO: https://www.sitespeed.io 27 requests, TTFB: 962ms (σ917.00ms), firstPaint: 2.92s (σ3.07s), firstVisualChange: 2.94s (σ3.07s), FCP: 2.92s (σ3.07s), DOMContentLoaded: 3.22s (σ2.98s), LCP: 2.92s (σ3.07s), CLS: 0.0389 (σ0.03), TBT: 769ms (σ53.00ms), Load: 3.77s (σ3.12s), speedIndex: 2.96s (σ3.07s), visualComplete85: 2.95s (σ3.06s), lastVisualChange: 4.54s (σ3.30s) (21 runs) that summaries all the runs used mean instead of median metric. That sucks when you do many runs and want to compare them. That is now fixed to show median number instead #1700.

14.18.0 - 2022-01-24

Added

  • Updated to Edge stable in the Docker container.

Fixed

  • A more safe way to get dockumentURI for Firefox. See PR #1699 and bug #1698.

14.17.0 - 2022-01-23

Added

  • New Select command #1696:

    • select.selectByIdValue(selectId, value)
    • select.selectByNameAndValue(selectName, value)
    • select.selectByIdAndIndex(selectId, index)
    • select.selectByNameAndIndex(selectName, index)
    • select.deselectById(selectId)
    • select.getValuesById(selectId)
    • select.getSelectedValueById(selectId)
  • New click by name command click.byName(name) #1697.

Fixed

  • Remove the top 10 rows of the image to handle progress bars on some mobile browser recordings. Thank you Gregory Mierzwinski for PR #1687.

14.16.0 - 2022-01-14

Added

  • Upgraded Geckodriver downloader that downloads a pre-built Geckodriver on Raspberry Pis.

14.15.0 - 2022-01-12

Added

  • Added so you can append a String to Firefox User Agent #1694, thank you Henrik Skupin for pointing me in the right direction. With #1695 we now have --appendToUserAgent that works in Chrome/Edge and Firefox.

14.14.1 - 2022-01-12

Fixed

  • Updated Chromedriver library that pickup already installed driver on Raspberry Pis. This makes it easier to run tests on an Android device from a Raspberry Pi.

14.14.0 - 2022-01-10

Added

  • Add support for Humble as connectivity engine for mobile phone testing. Make sure to setup Humble on a Raspberry Pi 4 and the choose engine with --connectivity.engine humble and set the URL to your instance --connectivity.humble.url http://raspberrypi.local:3000. Added in #1691.
  • Upgraded to Chrome 97, Edge 97 and Firefox 95 in the Docker container.
  • Upgraded to Chromedriver 97.

14.13.1 - 2022-01-01

Fixed

  • Added missing --chrome.appendToUserAgent in the CLI help.

14.13.0 - 2021-12-30

Added

  • Append text to Chrome/Edge user agent using --chrome.appendToUserAgent #1688.

Fixed

  • When you use Chrome and use a "emulated device" that will use the user agent that you provide using --userAgent#1689.

14.12.2 - 2021-12-09

Fixed

  • Added more info log to get a better feeling for what's going on #1686.
  • Fixed CPU throttling that was broken in Chrome 96 #1685

14.12.1 - 2021-12-01

Fixed

  • Changed log level to debug for Chrome/Edge when document request fails. It turns out that IRL that happens a lot and spam the log. Lets iterate over that functionality.

14.12.0 - 2021-11-30

Fixed

  • Adding error log message for Chrome/Edge when document request fails (faulty domain etc) #1682.
  • Added 'devtools.netmonitor.persistlog': true preference for Firefox to fix the HAR issue introduced in Firefox 94 #1684.

Added

  • Updated to Firefox 94 in the Docker image.

14.11.0 - 2021-11-23

Fixed

  • Use the viewport to determine if more cropping is needed in visual metrics. Thank you Gregory Mierzwinski for PR #1680.

Added

  • Updated to Selenium 4.1.0 #1679

14.10.2 - 2021-11-20

Fixed

  • Disabled the version check for Edge/Edgedriver in Edgedriver #1678.

14.10.1 - 2021-11-19

Fixed

  • Disabled the automatic Chrome/Chromedriver version check in Chromedriver #1676.
  • Loop until we find a frame with a good viewport or until we run out of retries for Visual Metrics. Thank you Gregory Mierzwinski for PR #1668.

14.10.0 - 2021-11-16

Added

  • Updated to Chromedriver 96 and Chrome 96 in the Docker container #1670.

Fixed

  • Added checks for Firefox HAR that it actually has a page.
  • Updated day js.

14.9.0 - 2021-11-07

Added

  • Updated to new Chome HAR PR #1666 that inlcudes chunk information.

Fixed

  • Updated to yargs 17.2.1 #1667.

14.8.0 - 2021-10-27

Added

  • New Edgedriver 95 and Edge 95 in the Docker container #1665.

Fixed

  • Fix for storing Firefox memory report and geckoprofile on Android 10+. Thank you Andrew Creskey for PR #1663.

14.7.0 - 2021-10-20

Added

  • Updated to Chromedriver 95. Added Chrome 95 in the Docker container.

14.6.1 - 2021-10-18

Fixed

  • If loading the URL fails (we get an exception, the browser crashes etc) make sure we stop the video correctly so it can be used #1657.

14.6.0 - 2021-10-13

Added

  • Updated to Selenium 4.0.0 #1655.
  • You can skip setting Browsertime default Firefox preferences with --firefox.noDefaultPrefs, thank you Gregory Mierzwinski for PR #1650.
  • You can skip setting Browsertime default Chrome options with --chrome.noDefaultOptions, fixed in #1651.
  • Updated to Firefox 93 in the Docker container.

Fixed

  • Updated eslint and prettier dependencies #1652.
  • Log Firefox preference and Chrome args on debug level #1654

14.5.0 - 2021-09-30

Added

  • Added CSS selector support for mouse commands #1645

Fixed

  • Internal refactoring of the mouse command #1646 and #1647.

14.4.0 - 2021-09-22

Added

  • Updated to Ch romedriver 94 and Chrome 94 in the Docker container #1643.

14.3.0 - 2021-09-16

Added

  • Update to Edgedriver 93. Update to Edge 93 in the Docker container #1641.
  • Updated to a new fresh WebPageReplay version in the Docker container #1635

Fixed

  • Enabling verbose logging (-vv and higher) with Chrome, broke tests. Fixed in #1640.

14.2.1 - 2021-09-06

Fixed

  • Updated Geckodriver dependency that uses a newer version of the tar-package.

14.2.0 - 2021-09-04

Added

  • Updated to Selenium 4.0.0-RC1 #1631.

Fixed

  • Somehow Throttle has been degraded, with this fix we are back to using 3.0.0.

14.1.0 - 2021-09-01

Added

  • Updated to Chromedriver 93 and added Chrome 93 in the Docker container #1630.

14.0.3 - 2021-08-31

Fixed

  • Catch if Chrome is missing FCP #1627.
  • Better error message if we have a Chrome/Chromedriver miss match #1628.
  • Make sure only one trace listener is created for Chrome when we get a trace #1629.

14.0.2 - 2021-08-20

Fixed

  • Removed that annoying extra line break in the log output #1626.
  • TTFB was reported as a string instead of a number in the JSON output #1625.

14.0.1 - 2021-08-12

Fixed

  • More tuning to exclude the top browser bar when calculating visual metrics that fixes too early first visual change when testing emulated mobile using Chrome in Linux #1623.

14.0.0 - 2021-08-12

Changed

  • Updated to Throttle 3.0 that do not set throttling on localhost by default on Mac OS. If you run test against a local server or use WebPageReplay on a Mac, you should add --connectivity.throttle.localhost to your test and it will work as before.

Fixed

  • Updated Throttle to 2.2.1 that fixes when two default interfaces are defined on Linux.

Added

13.1.4 - 2021-07-28

Fixed

  • Increased the default URL bar size for visual metrics from 18 pixel to 20 pixel #1619.

13.1.3 - UNRELEASED

  • This release was broken because of bad internet connectivity.

13.1.2 - 2021-07-23

Fixed

  • Increased the default URL bar size for visual metrics from 10 pixel to 18 pixel #1617.

13.1.1 - 2021-07-23

Fixed

  • Increased the extra wait time for orange to appear in the browser winrow when we record a video on desktop. Before we waited 400 ms, now we wait 800 ms #1616.

13.1.0 - 2021-07-22

Added

  • Chrome introduced a bug in how we measure first visual change (see #1614). Adding support to better configuration for notification/viewport with #1615.

13.0.0 - 2021-07-22

Breaking changes

  • With this release we drop support for NodeJs 10 (EOL ended some time ago).

Added

  • Updated the Docker container to use Chrome 92.
  • Updated to Chromedriver 92.
  • Upgrade to yargs 17 #1600.

Fixed

  • Fixed so that browsertime-result dir isn't created if you set your own result dir. Thank you Vicky Ledsom for PR #1611.

12.11.0 - 2021-07-15

Fixed

  • Fixed so we support JavaScript that ends with a ; when you select an element for for clicking #1610. See #1606 and thank you Vicky Ledsom for reporting.
  • Fixed recording videos on new Mac M1 for emulated mobile. We used to have a hack for getting the right DPI but it seems like that didn't work on Mac M1 (depending on what screen you are using) #1607.

Added

  • Added screen size for Moto G4 when you run Chrome as emulated mobile #1608.
  • Added Firefox 90 in the Docker container #1609.

12.10.0 - 2021-07-05

Fixed

  • Update container to use NodeJS 14.7.1.
  • Fix when LCP do not exist in Chrome #1602.

Added

  • Add Firefox memory support to Browsertime. This will measure how much memory that is used. Turn on with --firefox.memoryReport. Thank you dpalmeiro for PR #1601.

12.9.3 - 2021-06-24

Fixed

  • Reverted to yargs 16.2.0 since 17.X do not support NodeJS 10. Lets update to yargs 17 in next major.

12.9.2 - 2021-06-23

Fixed

  • Make sure video settings in activated on the iOS device before connecting with Safaridriver #1597.
  • Upgrade to chrome-remote-interface 0.30.1 #1599
  • Use yargs 17.0.1 #1595.

12.9.1 - 2021-06-09

Fixed

  • Finally a fix for driving Safari on iOS. Start/stop the Safaridriver manually, see #1596. Video on iOS is broken though, see if we can find a fix for that too.
  • Upgrade to use Selenium-4.0.0-beta.4 #1592 and updated throttle, dayjs, execa and chrome-remote-interface to run latest versions #1594.

12.9.0 - 2021-06-04

Added

  • Use the new definition of Cumulative Layout Shift #1591.
  • Include LCP info in the HAR so we later on can show LCP info in a har viewer #1590.
  • Pickup render blocking info from Chrome #1586, this will be useful in Chrome 92.

Fixed

  • Fix broken store the perf log for Chrome to disk #1585.

12.8.0 - 2021-06-02

Added

  • Updated the Docker container to use Firefox 89 and Edge 91.
  • Updated to Edgedriver 91.

12.7.0 - 2021-05-26

Added

  • Include the HTML element when you get timings for visual elements with --visualElements #1580. This makes it easier to debug.
  • Updated to Chrome and Chromedriver 91 #1583.
  • Updated to Edge 90 in the Docker container

Fixed

  • Fixed the broken CPU throttling in Chrome (when you do not run headless). See #1380 and #1582.

  • Fix broken getViewPort for Pixel 2 in mobile emulation, thank you Radu Micu for PR #1581.

12.6.1 - 2021-05-21

Fixed

  • Fixed adding cookies with equals sign in value. Thank you sebguilbaud for PR #1578.
  • Use data:text/html as orange screen for Safari as the rest of the browsers #1579.

12.6.0 - 2021-05-12

Added

  • Configure a value/score limit of when to paint shifting elements in the Layout Shift screenshot. Use --screenshotLSLimit. Defaulyt is 0.01 #1576.
  • Include the LCP HTML tag in result #1575.

12.5.0 - 2021-05-11

Added

  • Choose the color of the layout shift highlight in the screenshot by using --screenshotLSColor #1573.
  • Report the actual HTML for the Largest Contentful paint #1572.

Fixed

  • Better reporting of layout shift #1571. Add the domPath, the actual HTML of the shifted element and tuning of the coloring in the screenshot. Also make sure the time for the layout shift is reported.
  • Catch if iOS simulator start/stop fails #1569.

12.4.0 - 2021-05-02

Fixed

  • Safer checks to get LCP elements #1568

Added

  • Choose color how the LCP element is highlighted --screenshotLCPColor #1567.
  • First go at showing layout shifted elememnts in screenshot. Enable wth --screenshotLS #1565.

12.3.1 - 2021-04-30

Fixed

  • The script that get more info about largest contentful paint was not optimal and had bugs and log messages. Fixed in #1563.

12.3.0 - 2021-04-30

Added

  • Get a extra screenshot where the Largest Contentful Paint is highlighted --screenshotLCP and collect all elements in the LCP API #1561 #1562.

12.2.0 - 2021-04-27

Fixed

  • Fix broken "browser" category of script. Without the fix, it only worked in Firefox #1558. That meant no user agent string, and width/height of the browser window.
  • Removed getting the user agent string the pageinfo scripts, now when browser scripts work.

Added

  • Get the Safari version from the browser #1556.
  • Added simple browser cpu benchmark to run after metrics are collected #1559.

12.1.0 - 2021-04-23

Added

  • Updated to Firefox 88 in the Docker container.
  • New scroll to bottom command #1552.
  • Fixed so scroll to page command work in all browsers and not only Firefox #1551.

Fixed

  • Added missing --disable-dev-shm-usage cli option for Chrome, fixes sitespeed.io #3357.
  • Cleanup scrolling command #1553.

12.0.1 - 2021-04-16

  • With Chrome/Chromedriver 90 it seems to take longer time to close the browser than earlier versions, so when trying the next run, the previous browser is not closed #1548 fixed in #1549 by adding a 2 second sleep time before next try.

12.0.0 - 2021-04-15

Hi and welcome to the 12.0.0 release of Browsertime! Here's a list of what changed, what has been added and what has fixed. You can safely upgrade and please be aware that some metrics can change when you upgrade: We changed startup flags for Chrome and we made the "new" connectivity settings default (meaning 3g is faster than before and more realistic). Read more in each individual change.

A special thanks to Inderpartap Singh Cheema, Denis Palmeiro and Olaf Meeuwissen for their contributions to this release!

Changed

  • Display standard deviation instead of the home made median deviation in the cli output #1529.
  • Renamed layoutShift to the more correct cumulativeLayoutShift. This will is a breaking change if you use that metric. Updates to the new and coming layout shift changes announced by Google will be implemented the coming weeks.
  • Updated Chrome start flags on desktop following best practices and removing old flags #1507.
  • Updated Chrome start flags on Android following best practices and removing old flags #1506.
  • Finally the "new" connectivity settings are default. You can see the difference in https://github.com/sitespeedio/browsertime/blob/main/lib/connectivity/trafficShapeParser.js#L5-L104. Changed in #1540. If you wanna run with the legacy setting use --legacyConnectivityProfiles. See #1160 for the original change.
  • The default minimum wait time for waiting if a test is finished is now 8 seconds (instead of 5) #1542.

Fixed

  • Fix --chrome.blockDomainsExcept when you are using WebPageReplay #1532. Thank you Inderpartap Singh Cheema for the original fix!
  • Make sure gnirehtet is closed on the right device using device id #1527
  • Upgraded to Geckodriver 0.29.1.
  • Updated wpr_cert.pem to a new version for WebPageReplay #1316.
  • Include Google Web Vitals in the HAR file #1535.
  • Added 3 seconds wait time for geckoprofiler to start see #1538 and #1539
  • Added a ten seconds wait if getting the HAR from Firefox fails the first time #1546 fixes #3346.

Added

  • New metrics: Delta between TTFB and First Contentful Paint, Largest Contentful paint and First visual change #1528. You can use this if you have unstable TTFB and want to alert on front end metrics. Lets see when other also implement this :)
  • Made it easier for people to get Google Web Vitals. We copy that data under the googleWebVitals namespace in the result JSON #1521.
  • Added TTFB as a single metric #1522.
  • New stop watch command #1512. Measure time by:
       timer.start(); 
       // Do something
       // Stop the timer and add the result to the last tested URL
       timer.stopAndAdd();
    
  • Pre test/warm a URL with --preWarmServer. Do that to make sure your server has cached everything that is needed before your test #1515 and #1516.
  • Collect what HTML element change in cumulative layout shifts #1534
  • Added support for recording video on Safari iOS #1541.
  • New commands: scrolling by Pixels, Lines or Pages forward, back or refresh navigations, create new tabs or windows and switch to them and new mouse events such as context click, single click, double click, click and hold, release, and movement. Thank you Denis Palmeiro for PR #1533.
  • Improve proxy configuration support, thank you Olaf Meeuwissen for PR #1542.
  • Upgraded to Selenium 4.0.0-beta.3 #1544.
  • Updated to Chrome 90 in the Docker container and Chromedriver 90 as default #1543.

11.6.3 - 2021-03-17

Fixed

  • Last release broke getting Chromes netlog on desktop, fixed in #1511. Also added the ability to set the log level of the netlog.

11.6.2 - 2021-03-17

Fixed

  • Fixed getting the netlog from Chrome on Android #1508.
  • Combine the gecko perfstats from each process, thank you Andrew Creskey for the PR #1504.
  • Upgraded to Selenium 4.0.0-beta.2 (from beta.1) #1509.

11.6.1 - 2021-03-11

Fixed

  • Added Python2 to the Docker container for (Kubernetes)users that run TSProxy and made sure python2 is used for that proxy when you run in Docker #1503.

11.6.0 - 2021-03-08

Added

  • Make it possible to set number of threads used for FFMPEG using --videoParams.threads. Thank you Leo Blöcher for PR #1493.
  • Updated to Chrome 89 and Firefox 86 in the Docker container and updated to Chromedriver 89 #1499.
  • Get internal Firefox performance statistics by --firefox.perfStats. Thank you dpalmeiro for PR #1495.
  • Updated to Edgedriver 89 #1500.

Fixed

  • Better error log if we miss out on stats when adding fully loaded #1496.

11.5.0 - 2021-02-16

Added

  • Make it easy to run gnirehtet when you run your tests on your Android phone to reverese tethering the traffics. Start gnirehtet with --gnirehtet (gnirehtet needs to be in your PATH). Fixed in #1489 and #1490.
  • Updated to Selenium 4.0.0-beta.1 #1491.

Fixed

  • Follow the same pattern as sitespeed.io when replacing characters in file names #1488.

11.4.0 - 2021-02-02

Added

  • Added failure state in the result JSON. You get a new command: markAsFailure(message) where you can mark a test as a failure. If one run fails, all tests are marked as failed. The JSON from Browsertime (that is used in sitespeed.io) gets a failure status and an array of failure messages (potentially you can mark a run as failed multiple times). If we get an uncaught error from running the browser we also mark the test as failed #1482.s

Fixed

  • Updated dependencies to latest versions: jimp, yargs, find-up, dayjs and chrome-remote-interface #1485.

11.3.1 - 2021-02-02

Fixed

  • Fix so we start/close the iOS simulator before the first test and close it after the last one #1484.

11.3.0 - 2021-01-30

Added

  • Add better support for using the iOS simulator #1475 and #1480.
  • Record a video when using the iOS simulator #1476 and #1481.
  • Use throttle as default engine when you use the iOS simulator #1479.

11.2.0 - 2021-01-24

Added

  • Make it possible to listen to CDP events in scripting #1473. There's a new command in scripting (for browsers that supports it): commands.cdp.on. Here's an example to pickup all responses for a page :
    const responses = [];
    await commands.cdp.on('Network.responseReceived', params => {
     responses.push(params);
    });
    await commands.measure.start('https://www.sitespeed.io/search/');
    // Here you can check the array with all responses received

11.1.2 - 2021-01-20

Fixed

  • The last release didn't fix the problem when browsertime do not collect metrics.

11.1.1 - 2021-01-20

Fixed

  • Added a guard if we do not collect any metrics #1472.

11.1.0 - 2021-01-20

Added

  • Upgraded to Chromedriver 88 and Chrome 88 in the Docker container.

11.0.2 - 2021-01-19

Fixed

  • Make it configurable to press the power button when you start the tests with --androidPretestPowerPress#1462. Thank you Gregory Mierzwinski for the PR.

  • Catch if a URL change between runs. If you use a script, set an alias to a URL and then the next run the URL changed, that caused an error and Browsertime failed #1467. The alias for a URL wasn't propagated the right way, fixed in #1468.

  • This patch is for storing the gecko profile paths in the browsertime results. This makes it easier to figure out which gecko profiles go with which tests. #1466. Thank you Gregory Mierzwinski for the PR!

11.0.1 - 2021-01-05

Fixed

  • Setting path to Edge was broken, use --edge.binaryPath to point out the path to your Edge binary #1461.

11.0.0 - 2020-12-18

Changed

  • Record and keep the browser full screen (including URL bar) #1435. All metrics should stay the same with this change but the video and the code will be easier :) When we implemented video a long time ago we wanted to cut out the URL bar but it made it harder to keep the video to look ok on different OS.
  • Say goodbye to RUM Speed Index #1439.
  • Domain name on disk now uses underscore instead of dots in the name #1445.
  • Click the Android power button at the start of each test (instead of the home button #1447.

Fixed

  • Added a two minute timeout to get Geckoprofiler data #1440.
  • Made sure HAR stuff respect the skipHar flag #1438
  • Fix process ID fetch for Galaxy S5, thank you Michael Comella for PR #1449
  • If a web page timed out in Chrome, we missed to report that as an error, fixed in #1453.

Added

  • Automatically close "System not responding"-popup on Android if it exists #1444.
  • Add support using alias from CLI and use alias as folder name on disk #1443.
  • New option to store a more flat structure on disk converting the path part of the URL to one folder --storeURLsAsFlatPageOnDisk#1450
  • Updated to Selenium 4.0.0-alpha.8 #1451.
  • Updated to Firefox 84 in the Docker container.
  • Updated to Edgedriver 87
  • Added Edge in the Docker container #1458.

10.9.0 - 2020-11-18

Added

  • It's now easier to run Firefox Nightly/Beta on your Android phone. Use --firefox.nightly / --firefox.beta #1432.
  • Make it possible to add flags to Geckodriver using --firefox.geckodriverArgs #1433.

Fixed

  • The browsertime.json always included Firefox appConstants and the geckoprofiler JSON missed out on Visual Metrics, fixed in #1434.

10.8.0 - 2020-11-18

Added

  • Updated to Firefox 83 in the Docker container.
  • Updated Chrome and Chromedriver to 87.

10.7.0 - 2020-11-16

Added

  • Add page generator tag to the HAR file #1430

Fixed

  • Reverted to Geckodrover 0.27.0 since 0.28.0 cannot start Firefox on Android #1431.

10.6.5 - 2020-11-11

Fixed

  • Upgraded to Geckodriver 0.28.0.
  • Set firefox environment variables in process.env, PR by Kanishk thank you #1426.

10.6.4 - 2020-10-28

Fixed

  • Add the missing mobile phone id to the HAR file (with the other Android setup data) #1424.
  • Remove Firefox preferences that disables the speculative connection pool, thank you Andrew Creskey for the PR #1423.

10.6.3 - 2020-10-26

Fixed

  • Fix navigating to the same URL twice in the same script (that was broken in 10.0) #1421.
  • Make sure we log the Android phone id when the internet connection/USB connection fails on the phone #1420.

10.6.2 - 2020-10-23

Fixed

  • If the browser failed to navigate, the error caused the result.json to not be generated #1417.

10.6.1 - 2020-10-21

Fixed

10.6.0 - 2020-10-20

Added

  • Firefox 82 in the Docker container.

Fixed

  • Better log messages when the browser do not start start #1411.

10.5.0 - 2020-10-15

Added

  • Updated to Edgedriver 86.
  • Reboot your Android phone if the battery temperature don't meet your limit after X tries. Enable with --androidBatteryTemperatureReboot #1409.

10.4.1 - 2020-10-13

Fixed

  • Fix so that -vvv enables trace log level for Marionette when you use Firefox #1405.

10.4.0 - 2020-10-07

Added

  • Upgraded to Chromedriver 86 and Chrome 86 in the Docker container.

10.3.0 - 2020-10-03

Added

  • Add option to navigate with WebDriver instead of window.location with --webdriverPageload. Thank you Andrew Creskey for the PR #1396.
  • Add option for specifying logging format in visualmetrics.py. Thank you Gregory Mierzwinski for the PR throttl#1399.

Fixed

  • Fix bufferSize to proper 100MB default size for Geckoprofiler, thank you dpalmeiro for the PR #1394.

  • Max number of tries to check battery temperature on Android to make sure a test doesn't wait forever to run #1401.

10.2.1 - 2020-09-25

Fixed

  • Create the data dir for a URL direct after page complet fired so that collecting the MOZ log work #1388.

10.2.0 - 2020-09-23

Added

  • The Docker container now uses Firefox 81.

10.1.0 - 2020-09-22

Added

  • Extra love for running tests in Android: Press the home button at the start of a test, verify that the phone state is "device" before starting to test (no need to run tests on offline devices) and added possibility to verify the internet connection on the device through ping (enable with --androidVerifyNetwork) #1386.

Fixed

  • Reverted using buffered flag for Chrome Long Tasks (we missed Long Tasks with the new setup) #1383.

10.0.0 - 2020-09-20

The new 10.0 release mostly include technical changes that will make it easier for us in the future to make changes and keep Browsertime the number one performance engine :) However if yoy use Contentful and Perceptual Speed Index there are one breaking change.

Breaking changes

  • If you collect visual metrics, we do not calculate Contentful Speed Index and Perceptual Speed Index by default any more. Turn them on by using --visualMetricsPerceptual and --visualMetricsContentful. This will make your testing faster by default #1358.

Changed

  • Get CPU long tasks by default using Chrome: A couple of releases ago, Chrome started to support buffered long tasks, that means we don't need to inject any JS to make sure we catch all long tasks. The code is simpler and since (hopefully) the new buffered version doesn't give any performance penelty, we can start getting longtasks by default. #1341.

  • Stop recording the video direct after the PageCompleteCheck fired. This make the original video smaller, saves time converting the video to a vieable format and makes Visual Metrics a little faster #1357.

  • Use fast preset (instead of medium) when converting the video to a format that works in all video players #1359.

  • If you use scripting and use the ...AndWait methods, we now increased the extra wait time before we run the page complete check from 1000ms to 2000ms. With the other changes we done, this was needed for Firefox since it sometimes didn't have time to navigate before the complete check run #1375.

  • Make sure the screen is turned on independently of electric source (before it was only USB) on Android #1378

Tech

  • Making a better structure for each browser, so its more understandable when you can run browser specific code. The old browser delegate (now only browser name) has the newly named functions:
  • beforeBrowserStart
  • afterBrowserStart
  • beforeStartIteration
  • beforeEachURL
  • afterPageCompleteCheck
  • afterEachURL
  • failing
  • getHARs
  • beforeBrowserStop

This makes it easier to make sure when to collect metrics, stop trace logs and do whatever you need. Implemented in #1348 and #1367.

  • New structure for browser, trying to decrease line of code per file and making it easier to navigate the code and prepare for adding support for other browser drivers than Selenium #1354 #1355 #1356.

#1383* Moved page complete scripts to a new folder to make it clean #1361.

  • Restructure the video code #1364.

Added

  • By default the video is converted to a format that works in most video players. You can skip that convertion (to save time) by using --videoParams.convert false. Visual Metrics will still work, but the video may not work in your player #1360.

  • Make sure the video file is removed from the Android phone when its been copied to desktop #1377.

  • Making it easy to run Firefox on Android #1379.

  • Fix how MOZ_LOG collection works and allow custom MOZ_LOG settings #1382. Thank you Gregory Mierzwinski.

Fixed

  • Fixed broken CPU throttling in Chrome #1381.

9.4.2 - 2020-08-29

Fixed

  • Make sure First Paint is collected when First Contentful Paint exists in Firefox. Thank you Sean Feng for the PR #1347.

9.4.1 - 2020-08-28

Fixed

9.4.0 - 2020-08-26

Added

9.3.1 - 2020-08-24

Fixed

  • Use the correct settings to set the emulation for Chrome #1340.
  • Bump versions: dayjs, execs, speedline-core, yargs. jimp #1339.
  • Updated to Throttle 2.0.1 #1338.

9.3.0 - 2020-08-17

Added

  • Updated to Geckodriver 0.27.0 #1330.

Fixed

  • Updated Chrome-HAR to 0.11.11.
  • Fix to include visual metrics in the HAR, thank you Mason Malone for the PR #1335.

9.2.1 - 2020-07-31

Fixed

  • New chrome-har and updated day-js dependency.
  • Ignore a couple of more pixels at the bottom of the browser screen to ignore Chromiums loading info bar, thank you Pan Alexey for the PR #1327.

9.2.0 - 2020-07-28

Added

  • Updated to Firefox 79 in the Docker container.

9.1.0 - 2020-07-17

Added

  • Updated to Chromedriver and Edgedriver 84, Chrome 84 and Firefox 78 in the Docker container 1323.

9.0.2 - 2020-07-08

Fixed

  • Guard against missing FF options when running without CLI #1319.
  • Safer check for layoutShift #1318.

9.0.1 - 2020-07-07

Fixed

  • Fix support for running setup & teardown in a script. This was working fine from the CLI but not running it programmatically as reported in sitespeed.io #3068. #1317.

9.0.0 - 2020-06-26

Changed

  • Change how screenshot are stored to support multiple screenshots for one run. Screenshots per run is an array, store screenshots per folder per run screenshoots/1/ and name the default screenshot to afterPageCompleteCheck#1312.

Fixed

  • Updated to latest throttle, jimp, day-js.

Added

  • Add support to run privileged JavaScript in scripts and wait for page to load #1314.

8.14.0 - 2020-06-03

Added

  • Option to configure number of browser restart tries #1292. Use --browserRestartTries.
  • Use Firefox 77 in the Docker container.
  • Add support for Android power testing #1296. Thank you Gregory Mierzwinski for the PR! Also #1300 adds supports for power testing on Chrome.

Fixed

  • Running wiothout the CLI caused error logs removing Firefox appconstants #1294.
  • Fixed broken MS Edge support (since 80?) and upgraded to Edgedriver 83 #1298

8.13.1 - 2020-05-26

Fixed

  • The log was putting CLS in % even though it isn't #1288.
  • Updated dependencies: chrome-har, execa, chrome-remote-interface and dayjs #1290 and #1289

8.13.0 - 2020-05-19

Added

  • Upgraded to Chromedriver 83 Upgraded to Chrome 83 in the Docker container.
  • Include FCP and LCP in the timings in the HAR file #1285.

Fixed

  • Fixed decimal formating in statistics #1286

8.12.1 - 2020-05-19

Fixed

  • Fixed so metrics in the CLI always shows two decimals #1282.
  • Adopt the rename of layout-instability-api to CLS and remove report in percentage since it do not make sense #1283 and #1284.

8.12.0 - 2020-05-12

Added

  • Run tests with Safari Technology Preview using --safari.useTechnologyPreview #1280.

8.11.1 - 2020-05-09

Fixed

  • Metrics output was broken in 8.11.0 if you only do one run #1278.

8.11.0 - 2020-05-09

Added

  • Output TBT, CLS and TTFB in the CLI summary when availible #1276 and per run. Also unify how we output metrics from thee CLI