Skip to content
Permalink
main
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time
4154 lines (2888 sloc) 273 KB

CHANGELOG - sitespeed.io (we use semantic versioning)

25.3.2 - 2022-06-30

Fixed

  • Another go at fixing the preWarmServer issue #3683.

25.3.1 - 2022-06-29

Fixed

25.3.0 - 2022-06-28

Fixed

Added

25.2.1 - 2022-06-23

Fixed

  • Bug fix for adding custom CSS from plugins, than you Josh Duncan for PR #3679.

25.2.0 - 2022-06-22

Added

  • Updated the Docker container to use Chrome 103.

25.1.1 - 2022-06-20

Fixed

  • Upgraded to Browsertime 16.9.1 that upgraded to Throttle 4 that internally uses ip route instead of route (one less dependency).

25.1.0 - 2022-06-15

Added

  • Updated Browsertime to 16.9.0 with the following changes:
    • New wait.byCondition command. Thank you Icecold777 for PR #1803.
    • Collect number of CPU longtasks before largest contentful paint #1806.
    • Instead of throwing errors and exit the tests if the page hasn't finished loading after 5 minutes, we now gracefully ends the test after 2 minutes (you can change that time with --maxLoadTime). That makes more sense than just throwing errors #1810.

25.0.0 - 2022-06-10

Changed

--debug do not add verbose logging anymore, instead it uses Browsertimes debug mode. You can use breakpoints to debug your script. You can add breakpoints to your script that will be used when you run in --debug mode. At each breakpoint the browser will pause. You can continue by adding window.browsertime.pause=false; in your developer console.

Debug mode works in Chrome/Firefox/Edge when running on desktop. It do not work in Docker and on mobile. When you run in debug mode, devtools will be automatically open so you can debug your page.

In debug mode, the browser will pause after each iteration.

Read the documentation on how to debug sitespeed.io scripts.

Added

  • Added Firefox 101 and Edge 102 in the Docker container.

Fixed

  • Updated to Axe-core 4.2.2 #3673

24.9.0 - 2022-05-25

Added

  • Updated the Docker container to use Chrome 102 #3665.

24.8.1 - 2022-05-21

Fixed

  • Better fix for getting the correct browser name and version when storing latest run data #3661.

Added

  • Updated to Browsertime 16.7.0 from 16.4.0. With the latest versions we get:
  • Use --browsertime.videoParams.thumbsize to set the max size of the filmstrip thumbnails.
  • Interaction to next paint is collected for browsers that supports it (useful for your user journeys)
  • The time when the last CPU long task happens is a new metric, thank you Andy Davies for that idea!
  • You can now use groupAlias directly in a text file #3655. You can read how in the updated documentation.
  • Added an option for using the full Graphite namespace when latest screenshots/video is stored for a test #3660.

Fixed

  • If you set a user agent for Browsertime, also use it for the crawler #3652.
  • Fix missing browser info when storing latest run data #3658.

24.7.0 - 2022-05-11

Added

  • Updated to Browsertime 16.4.0 fixes a bug for LCP in the video and also output when LCP happens in the video.

Fixed

  • Make sure we got CLS from Browsertime when we try to render it #3650.

24.6.0 - 2022-05-10

Added

  • Updated to Browsertime 16.3.0 that collects Largest Contentful Paint from the browser (if the browser suppoorts the LCP API).

  • Show total download time per domain #3648.

24.5.1 - 2022-05-06

Fixed

  • Reverted to Ubuntu 20 in the Docker container. Ubuntu 22 gave Firefox problems on ARM and we also seen other problems with NodeJS in that container.
  • Updated PageXray/Coach Core that include SSL times in the total timings per domain (that was missed before).
  • Added more safe check when we miss Crux data.

24.5.0 - 2022-05-05

Added

  • Add INP and TTFB to the CRUX metrics #3645.
  • Updated the Chrome USer Experience dashboard to include the new metrics. You can see the dahsboard here and download the new one from here.

Fixed

  • Bumped Browsertime with a new fix release for the upcoming portable visual metrics script.

24.4.0 - 2022-05-04

Added

  • Updated to Edge 101 in the Docker container. Updated to latest Browsertime 16.2.0 with Edgedriver and Chromedriver 101.
  • Updated to Firefox 100 in the Docker container.

Fixed

  • If --firstParty do not match any URL, make sure we still show first vs third party #3643.
  • Updated to latest NodeJs in the slim container.

24.3.0 - 2022-04-27

Added

  • Updated to Chrome 101 in the Docker container #3641.

24.2.0 - 2022-04-26

Added

  • Updated base Docker image to use Ubuntu 22.04 #3636.

Fixed

  • Updated dependencies: aws-sdk, cli-color, fs-extra, influx, junit-report-builder and tape #3640

Tech

  • Use Ava for unit testing #3637
  • Use local HTTP server to speed up GitHub Action tests #3638.
  • Updated dev dependencies #3639.

24.1.0 - 2022-04-22

Added

  • Update Grafana auth settings to allow api token or basic auth, thank you Vladimir Stepanov for PR #3627.
  • Add TTFB to the metrics page #3630.
  • Use latest Coach core with updated PageXray and Third party web #3629.
  • Make it possible to disable annotations for Graphite. Set --graphite.sendAnnotation false to disable sending annotations #3625.
  • Show which run that is used in the metrics tab #3631.
  • Updated to Browsertime 16.1.0.

Updated

  • Update dependencies: google-cloud/storage, aws-sdk, dayjs, yargs #3635.

24.0.0 - 2022-04-06

Changed

  • Upgraded to Browsertime 16.0.0 that changed how the DNS is flushed. You need to add --flushDNS to your conifguration to flush the DNS between runs.

Added

  • Use Firefox 99 and Edge/Edgedriver 100 in the Dockker container.

Fixed

  • Better catch if Visual Metrics fails #3619
  • Fixed JUnit/alias bug #3620

23.7.0 - 2022-03-31

Added

23.6.1 - 2022-03-25

Fixed

  • Fixing text formatting in the JSON from latest run and make sure we display the time with UTC#3611.

23.6.0 - 2022-03-25

Added

  • Various fixes to add more content to the JSON stored from the latest run, making it easier to show more meta data for a run in Grafana #3607, #3609 and #3610.

23.5.2 - 2022-03-22

Fixed

  • Updated to latest NodeJS and Ubuntu updates in the Docker container with some security updates#3306. Also updated the slim container with latest NodeJS.
  • Make sure the shrinkwrap file is used when building the container #3604 and (hopefully) fix so that .dockerignore is not ignored when Github Actions build the containers.

23.5.0 - 2022-03-11

Added

  • The -slim Docker container now uses Firefox 98.
  • Updated Grafana to latest 8.4.3 in the Docker compose file.
  • Updated Graphite dashboards to use Timeseries graphs instead of old "Graph".
  • You can use -o or --open or --view to open the result page after you run sitespeed.io on Mac and Linux #3569.
  • Open the result with -o on Linux using xdg-open #3597.

23.4.0 - 2022-03-09

Added

  • Firefox 98 in the Docker container #3592.
  • Added link to each run in the side by side metric page, making it easier to go to the correct run #3593.

23.3.0 - 2022-03-07

Added

23.2.0 - 2022-03-05

Added

  • Updated to Chrome and Edge 99 in the Docker container. Updated to Chromedriver and Edgedriver 99 #3590.

Fixed

  • If you use alias and budget files, the outcome (result budget json and others) should use the alias of the URL instead of the URL #3582.
  • Ignore sustainable.setup messages when storing analysistorer #3578.

23.1.0 - 2022-02-24

Added

  • Updated Browsertime that collect timings from main document. The result includes a field named mainDocumentTimings and contains blocked, dns, connect, send, wait, receive, ssl as long as you get a HAR file from the browser.

  • Show Browsertime version in runtime settings #3575.

Fixed

  • Updated Chromedriver dependency that fixes installation on Mac M1 and some send keys issues.

23.0.1 - 2022-02-21

Fixed

  • Bumped to Browsertime 15.0.1 to make sure connectivity is set only once. In last release it was set in the beginning of the test and for eacch iteration. In practice that doesn't matter but it's very confusing if you check the logs.

23.0.0 - 2022-02-21

Breaking changes

  • In this release we updated to Browsertime 15.0.0 that drops built in support for TSProxy. The reason to drop TSProxy is that it only works in Python 2 and the sunset for Python 2 was January 1, 2020. If you still need TSProxy you can set it up yourself.

  • Browsertime also drop support for getting visual metrics from the trace log. It was added to evaluate if it could be a compliment to the video visual metrics but it was not. Removing it also decreased the number of dependencies in Browsertime.

  • Linux/Mac OS will flush the DNS between runs, that means that you can have some changes in DNS lookup time when you upgrade to 23.0.0.

Fixed

  • Fix broken JUnit output. Thank you rghetu for finding it #3569.
  • When comparing metrics side by side for different runs, mean/median and stddev was sometimes broken for some timing metrics. Fixed with #3573

Added

  • Use Browsertime 15.0.0.
  • Show alias name in the budget report page #3572.

22.1.2 - 2022-02-09

Fixed

  • Updated to Browsertime 14.21.1 that disables the new Chrome splash screen by default.

22.1.1 - 2022-02-08

Fixed

  • The slim container with only Firefox was broken in last release. It's now re-released with Firefox 96 and Firefox 97 coming soon.

22.1.0 - 2022-02-08

Added

  • Upgraded to latest Browsertime with Egdedriver 98.
  • Upgraded to Edge 98 and Firefox 97 in the (amd64) Docker containers.

22.0.0 - 2022-02-07

Breaking changes

  • If you use the Lighthouse plugin there's breaking changes:

In the new version we drop support for the following:

  • Running multiple runs with Lighthouse.
  • Using scripts to login the user (or whatever you need before you run your tests) It's a couple of reasons why I remove those features:
  • I been looking for a maintainer of the Lighthouse plugin for +1 year and I haven't found one. For me to be able to maintain it I want the plugin to be as simple as possible.
  • I deeply regret merging the PR for adding multiple runs for Lighthouse. That PR goes against everything I know about measuring performance. Lighthouse is not built for getting correct performance metrics, it's built to help (Chrome) developers to get insights how they make the page "faster". Lets stick to the basics and keep it possible to get those recommendations from Lighthouse.
  • Maybe someday Lighthouse will have support for user journeys, lets wait until that is officially supported and then I can check if it could be used in the plugin.

With the new release we also break how you configure Lighthouse. People has had problem with that since day 1. With the new version we support two new ways to configure Lighthouse:

  • By configuration JSON file. --lighthouse.config config.js
  • By Lightouse flags file. --lighthouse.flags flag.json

If you don't need to configure Lightouse you can use the default settings both for desktop and mobile. If you run without any settings, the plugin will use desktop settings. If you run with --mobile, --android or --ios the mobile settings will be used.

Added

  • Build Docker containers for both amd64 and arm64 to make containers work on Mac M1. The arm container contains Firefox and Chromium. Thank you whichfinder and Radu Micu for the help! Fixed in PR #3554.
  • When plugins is loaded, there's a new extra last step where we try to load the plugin as a globally installed npm module #3546.

Fixed

  • Fix so that we do not display the same 3rd party cookie multiple times #3545.
  • Updated Coach Core that includes the latest version of third party web and PageXray that find more fonts without mime type.

21.6.1 - 2022-01-24

Fixed

  • Updated to Browsertime 14.18.1 that makes the summary metric log message use median (instead of mean) and change a log message level to debug.

21.6.0 - 2022-01-24

Added

  • Updated to Edge stable release in the Docker container.
  • Remove Crux distribution table and use pie charts instead #3537
  • Add extra sleep time between Crux calls to make sure to not overload the API limit #3536.
  • Added extra Crux enable command line --crux.enable to enable Crux #3538. Its default value is true and you also need to supply the Crux key to run Crux. The reason for the new parameter is that you can now configure the key in your configuration JSON and set the enable to false and then you enable it with the CLI parameter when you actually need to run Crux.
  • Show Crux-metrics on the Summary page #3540.
  • Updated summary metrics tables with headings to make it easier to read #3541.
  • Added Browsertime 14.17.0 with new Select and click.byName commands. With that Browsertime version you also need to have ffprobe installed when you run Visual Metrics but that should already be installed.
  • Added Browsertime 14.18.0 with a fix for Firefox #1698

21.5.0 - 2022-01-14

Added

  • Upgraded to Browsertime 14.15.0 that adds support for --appendToUserAgent for Chrome/Edge/Firefox. And then Browsertime 14.16.0 that supports Geckodriver for Raspberry Pi.

21.4.0 - 2022-01-12

Added

  • Updated to a new build of WebPageReplay in the Docker container
  • Updated the Ubuntu base image to latest version and latest NodeJS in the Docketr container.
  • Upgraded Browsertime #3528:
    • Add support for Humble as connectivity engine for mobile phone testing. Make sure to setup Humble on a Raspberry Pi 4 and the choose engine with --connectivity.engine humble and set the URL to your instance --connectivity.humble.url http://raspberrypi.local:3000. Added in #1691.
  • Upgraded to Chrome 97 and Edge 97 in the Docker container.
  • Upgraded to Chromedriver 97.

Fixed

  • Updated Chromedriver library that automatically picks up the Chromedriver if it's installed on Raspberry Pi.

21.3.0 - 2022-01-01

Added

  • Updated to Browsertime 14.13.0 with the following fixes for the user agent:
    • Append text to Chrome/Edge user agent using --chrome.appendToUserAgent
    • When you use Chrome/Edge and use a "emulated device" that will use the user agent that you provide using --userAgent. Else it will use the user agent from your emulated device setting.
    • You can also use Edge to run emulated mobile with the same settings as Chrome.

21.2.2 - 2021-12-23

Fixed

  • Fix the error introduced in 21.2.0 for "Include page summary URL in the result JSON" #3525.

21.2.1 - 2021-12-22

Fixed

  • It turns out that Firefox 95 doesn't work with the HAR export trigger and the workaround that worked in Firefox 94 seems to not work in 95 see sitespeedio/browsertime#1671 (comment). That's why we are reverting to Firefox 94 in the Docker containers.

21.2.0 - 2021-12-22

Fixed

  • The catching of errors in the queue was broken and reported the error x times (x=number of plugins). Also when we had an error the result JSON was not stored. #3522.

Added

  • Updated to Firefox 95 and Edge 96 in the Docker container.
  • Include page summary URL in the result JSON #3523.

21.1.0 - 2021-12-06

Added

  • Added possibility choose name for storing a result JSON #3520. We use this for a feature that will be released soon.

21.0.1 - 2021-12-01

Fixed

  • Updated Browsertime that logs Chrome document request failures on debug log level instead so that the log is not cluttered.

21.0.0 - 2021-12-01

Changed

  • Updated to Coach Core 7.0.0. This will probably change your Coach score. The new Coach has the following changes:

    • Moved AMP advice to best practice instead of privacy #67.
    • Increased favicon max size advice from 5 to 10 kb #68
    • Renamed the fastRender advice to avoidRenderBlocking #73
    • Remove the third party async advice #74
    • Updated the layout shift advice to use cumulative layout shift #75
    • Changed id of the Google Tag Manager advice #79
    • Updated third-party-web to 0.12.6.
    • Use Chrome(ium) render blocking information to know if a request is render blocking or not #66.
    • Report offending JavaScript assets if the JavaScript max limits kicks in #70.
    • New largest contentful paint advice #76.
    • New first contentful paint advice #77.
    • Added TBT in the CPU longtask advice #80.
    • Report content and transfer size for offending URLs #81.
    • Report offending assets with transfer/content size for page size limit #82.
    • Fix cases when JQuery is undefined. Thank you shubham jajodia for PR #64.
    • A better way to find offending layout shifters. Thank you shubham jajodia for PR #65.
    • Removed mentions aboout server push #69
    • Added more information on how to debug CPU advice #71.
  • Updated to sustainable plugin core code to co2 0.8.0. This include a biug fix to the 1byte model that will lower your co2 score #3519.

Added

  • Updated to Browsertime 14.12.0 that fixes the FF 94 HAR issue.
  • Updated green domains from the Green Web Foundation #3513.
  • Updated PerfCascade that support chunks when you use Chrome(ium) #3514.
  • Updated to Firefox 94 in the Docker image.

Fixed

  • Added missing summary boxes on start page #3515.

20.6.2 - 2021-11-20

Fixed

20.6.1 - 2021-11-20

Fixed

20.6.0 - 2021-11-17

Note: Do your test fails with Firefox 94? See Browsertime #1671. Until that is fixed in Firefox, it seems most use cases is fixed by adding a settle time (the browser rest for a while before we start the test). Try with --browsertime.settleTime 10000 to add a 10 seconds wait time before the test starts.

Added

20.5.0 - 2021-11-09

Added

Fixed

  • Ignore Influxdb and Grafana setup messages when you use the analysisstorer plugin #3508.

20.4.2 - 2021-11-07

Fixed

  • Updated to AXE core 4.3.5 #3501.
  • Use latest npm in the Docker container #3502.
  • Fixed the bug that caused so that you couldn't use the analyisstorer plugin together with Graphite #3506.
  • Updated to yargs 17.2.1

20.4.1 - 2021-10-29

Fixed

  • Reverted the change with npm in the Docker container since it broke both webpagetest and the +1 container installation

20.4.0 - 2021-10-29

Added

  • Updated to Browsertime 14.8.0 with Edgeriver 95.
  • Updated to Edge 95 in the Docker container.
  • Update to use NodeJS 16 in the Docker container #3495.
  • On Mac OS: Use --open or -o to open the result in your default browser. Thank you Johanna Lindh for the idea! PR #3493.
  • The +1 container now had Lighthouse 8.6.0.

Fixed

  • Removed npm from the Docker container when everything has been installed to fix npm security issues + decrease the size of the container.

20.3.1 - 2021-10-21

Fixed

  • Fix to handle integer value Android device serial, thank you Saurav Kumar for PR #3490.

20.3.0 - 2021-10-20

Added

  • Updated Browsertime that uses Chromedriver 95.
  • Updated Chrome 95 in the Docker container.

20.2.0 - 2021-10-14

Added

Fixed

  • Allow crawler to use cookies, thank you dammg for the PR #3472 with small fix #3473.

20.1.0 - 2021-10-05

Added

  • Use Firefox 93 in the Docker and the Docker slim container #3471.

20.0.0 - 2021-09-30

Please read the 20.0 blog post!

Breaking changes

  • Before you update to the new version: If you save your metrics to Graphite, have you upgraded those metrics to the new format as introduced in sitespeed.io the 15/4-2021? If not, please follow the guide in the documentation. If you haven't done that and still upgrade to sitespeed.io 20 you need to make sure you add --graphite.addSlugToKey false to your test else the metrics will be reported under a new key structure. The change was done in #3434.
  • Set throttle as default connectivity engine if you use Mac or Linux #3433. This makes it much easier to enable throttling. Our Docker container is not affected by this change.
  • There's a new default mobile --mobile for Chrome. The new default is Moto G4 (instead of iPhone 6) #3467.
  • When you run your tests on Safari on iOS the Coach is disabled by default #3468.

Added

19.6.0 - 2021-09-23

Added

  • Updated to Chrome 94 in the Docker container.
  • Upgraded to Browsertime 14.4.0 that included Chromedriver 94.

19.5.0 - 2021-09-17

Added

  • Updated to Edge 93 in the Docker container.
  • Updated to Browsertime 14.3.0.
  • Updated WebPageReplay to new version #3642.

19.4.2 - 2021-09-08

Fixed

  • Updated the Docker container to use Firefox 92 (instead of beta 92)
  • Updated the base Docker container to use a newer updated version of Ubuntu 20.04. See #3456.

19.4.1 - 2021-09-06

Fixed

19.4.0 - 2021-09-05

Added

  • Add option to ignore robots.txt when crawling. Use --crawler.ignoreRobotsTxt true to ignore. Thank you dammg for PR #3454!
  • Updated to Browsertime 14.2.0.

Fixed

  • If generating a HTML file failed, all generation failed. This fixes that and continue with the next file #3453.

19.3.0 - 2021-09-01

Added

  • Upgraded to Browsertime 14.1.0 with Chromedriver 93.
  • Added Chrome 93 in the Docker container.

Fixed

19.2.0 - 2021-08-27

Added

  • Upgraded to AXE core 4.3.2 #3441.
  • Added stddev/median/mean to the metrics side by side page #3443.
  • Added a generic text that we miss out of many metrics for Safari at the moment #3442.
  • Add option to add friendly name to junit test cases. Use --budget.friendlyName to set that. Thank you Vishal for the request. Done in PR #3448.

Fixed

  • Upgrade PerfCascade that catches if an HAR entry is missing content type #3445.

19.1.0 - 2021-08-20

Added

  • You can now see curated metrics side by side for all runs #3439.
  • The WebPageTest plugin is using the latest (0.5.0) version of the WebPageTest API.

Fixed

19.0.0 - 2021-08-13

Changed

  • Updated to Browsertime 14.0.1. The new 14 version uses Throttle 3.0 that has change if you use it on Mac OS: Updated Throttle 3.0 that do not set throttling on localhost by default on Mac OS. If you run test against a local server or use WebPageReplay on a Mac, you should add --browsertime.connectivity.throttle.localhost to your test and it will work as before.

Added

  • Updated to Edge 92 and Edgedriver 92 in the Docker container.
  • Updated to Firefox 92 beta in the Docker container to fix the devtools slowness bug that was inroduced in Firefox 90.

18.0.1 - 2021-07-29

Fixed

  • Updated to Browsertime 13.1.4.
  • Added cli parameter for keeping the original video --videoParams.keepOriginalVideo. The functionalty already exists but it wasn't exposed in sitespeed.io #3430.
  • The GPSI-plugin has a fix to catch when first input delay data is missing.
  • Fix broken tags for InfluxDB when you get Crux data from the GPSI-plugin. #3429. Data from the plugin has a testType tag that can have the following values: googleWebVitals, score or crux. Crux data has two more tags: metric and experience.

18.0.0 - 2021-07-26

Breaking changes

  • Drop support for NodeJS 10.
  • If you use Chrome the --chrome.timeline is now true by default (you can remove that from your configuration).

Added

  • Updated the Docker container to use Chrome 92.
  • Updated to Browsertime 13.1.1 (that uses Chromedriver 92).
  • Updated to latest PerfCascade so that the waterfall highlights which request that is the largest contentful paint (if its an image) and show render blocking info (if you use Chrome) #3407. You can checkout the documentation about where to see the render blocking information.
  • Show render blocking info in the Page Xray section #3246.

Fixed

  • Updated to Coach core 6.4.3

17.10.0 - 2021-07-16

Added

17.9.0 - 2021-07-16

Added

  • Updated to Browsertime 12.11.0.
  • Updated to Firefox 90 in the browser container #3420.
  • Update to AXE-core 4.2.3 #3417.
  • Added support for Firefox memory report, turn it on with --firefox.memoryReport #3416
  • The +1 container has been updated to use Lightouse 8.1.0.

17.8.3 - 2021-07-06

Fixed

17.8.2 - 2021-06-27

Fixed

  • Keep selected tab open across runs. Thank you Tanishq for PR #3409.
  • Update Docker container to use NodeJS 14.7.1.
  • Upgraded to Browsertime 12.9.3.
  • Updated Coach-core: Use all headers for Wappalyzer (before only the main document was used)

17.8.1 - 2021-06-10

Fixed

  • Updated Browsertime to 12.9.1 that fixes the problem with running Safari on iOS. However there are still one bug/problem that needs to be fixed with Safari on iOS: recording a video do not work.

  • The docker-compose file now uses Grafana 8.0.0

17.8.0 - 2021-06-04

Added

Fixed

  • Guard if you try to run Safari simulator in Docker #3405

17.7.0 - 2021-06-03

Added

  • New Browsertime 12.8.0 with Edgedriver 91 and updated to Firefox 89 and Edge 91 in the Docker container #3404.
  • Updated to use Coach-core 6.4.0 that uses wappalyzer-core 6.6.0 and a bug fix that make sure the private assets and cache header advice only checks GET requests.
  • The Lighthouse-plugin was updated to 8.0.0 in the +1 container.

Fixed

  • Catch if visual elements do not produce data (special case for amazon.com) 3402.

  • Upgrade to AXE-core 4.2.1 #3396.

Tech

  • Switch to sass instead of node-sass #3396.

17.6.0 - 3

Added

  • Updated to Browsertime 12.7.0 that fixes the problem with the broken CPU throttling in Chrome.
  • Chrome and Chromedriver 91. Edge 90 (Eddge 91 coming later this week) in the Docker container

Fixed

  • Display more information about the visual element that you are measuring #3394. Checkout the updated documentation on how to measure when elements are displayed in the viewport.

17.5.0 - 2021-05-21

Added

  • Make it easier to set up budget for Google Web Vitals #3386
  • Updated dashboards to in the Docker setup.

Fixed

  • New Browsertime 12.6.1 that fixes a bug when you use a equals sign in the value field in a cookie.
  • Updated dependencies with install warnings #3387,

17.4.0 - 2021-05-14

Added

  • Updated Browsertime and more love for showing better info from LCP and LS #3381.
  • Updated to Lighthouse 7.4.0 in the plus-1 Docker container.

17.3.1 - 2021-05-10

Fixed

  • 17.3.0 introduced a bug copying the same screenshot for multiple pages when you use the lateststorer plugin#3376.

17.3.0 - 2021-05-06

Added

  • Updated to Axe-core 4.2.0.
  • Show browser window size in runtime setting when using Android/iPhone #3362.
  • Also copy LS and LCP screenshots when using lateststorer plugin #3371.
  • Added colors (red/yellow/green) to Google Web Vitals summary box, using the same limits as set by Crux #3370.
  • Updated to latest Browsertime and remake on how to show LCP and LS.

Fixed

17.2.0 - 2021-04-27

Added

  • Show runtime settings in the HTML #3359. This makes it easier to see what settings are used. We gonna iterate and add more settings later.

  • Upgraded to Browsertime 12.2.0.

17.1.1 - 2021-04-23

Fixed

  • Updated to latest Browsertime that fixes the missing dev-shm flag for Chrome making running Chrome in Docker crash for some sites see #3357.

17.1.0 - 2021-04-20

Added

  • Updated to Firefox 88 in the Docker containers.

17.0.1 - 2021-04-17

Fixed

  • Updated Browsertime to 12.0.1 that fixes the problem with Chrome/Chromedriver 90 that introduced longer time to close the browser than earlier versions, so when trying the next run, the previous browser is not closed, fixed by adding a 2 second sleep time when closing the browser.

17.0.0 - 2021-04-15

Woohoo we shipped 17.0.0! There are many changes and you should read through the full changelog and focus on the new best practise section and breaking changes.

New best practices

One of the new things in 17 is the support for one extra key in Graphite: the name of the test. Set a computer friendly name of your test by using --slug. Then use the slug in the graphite key by adding --graphite.addSlugToKey to your run. When you do that change, should also convert your graphite data and your dashboards. The plan is like this:

  • In April 2021 you can convert your data and use the slug. You need to add --graphite.addSlugToKey true else you will get a log warning that you miss the slug for your test. All default dashboards in sitespeed.io will use the slug, so to use them you should add that new key and convert your data.
  • In September 2021 --graphite.addSlugToKey true will be set to default, meaning if you haven't upgraded your Graphite data yet, you need to set --graphite.addSlugToKey false to be able to run as before.
  • In November 2021 the CLI functionality will disappear and you need upgrade your Graphite metrics when you upgrade sitespeed.io.

You can read how to upgrade in the documentation.

When you have updated you will add one extra parameter to your test: --copyLatestFilesToBase. When you have done all these step you are ready for ... using the new dashboards in Grafana with screenshot and video from last run. This is SUPER useful to be able to fast see what's going on. Checkout what it looks like here and here.

We ship all new dashboards with some extra focus on Google Web Vitals (for sitespeed.io, WebPageTest, Google Page Speed Insights, Lighthouse and CRUX). We understand that these metrics is important for some users so lets focus on them. We also ship a couple new example dahsboards for how to setup user journeys. Documentation for those will come soon.

There's a new version of the Coach with a new super important privacy advice: Make sure you disable Chrome's new FLoC for your site. Read more about FLoC.

Breaking changes

  • We have changed some of the connectivity profiles, you can see the changes here. This means that if you use 3g connectivity using Throttle, your tests will have a faster TTFB than before. If you wanna hold on to the old settings you can do that by adding --browsertime.legacyConnectivityProfiles true to your tests.
  • If you have a budget using layoutShift that metric has now been renamed to cumulativeLayoutShift.
  • Read the full list of other changes in Browsertime.

What to think about when upgrading

You need to do a plan on when you want to upgrade Graphite to be able to use those new dashboards. You can upgrade to 17.0.0 and continue as before but to be able to use the new things in the dashboards you need to upgrade your Graphite data).

Other notable changes

Many thanks to Inderpartap Singh Cheema that fixed so that you can use --chrome.blockDomainsExcept together with WebPageReplay in the Docker container, so you more easily can focus on the performance disregarding 3rd party marketing scripts.

We have a couple of new metrics that is the delta between TTFB and First Contentful Paint, Largest Contentful paint and First visual change #1528. You can use this if you have unstable TTFB and want to alert on front end metrics. The metrics are automatically sent to Graphite.

Lets continue with all the changes.

Added

  • Updated to Browsertime 12.0.0 with the following changes:
    • Display standard deviation instead of the home made median deviation in the cli output #1529.
    • Renamed layoutShift to the more correct cumulativeLayoutShift. This will is a breaking change if you use that metric. Updates to the new and coming layout shift changes announced by Google will be implemented the coming weeks.
    • Updated Chrome start flags on desktop following best practices and removing old flags #1507.
    • Updated Chrome start flags on Android following best practices and removing old flags #1506.
    • Finally the "new" connectivity settings are default. You can see the difference in https://github.com/sitespeedio/browsertime/blob/main/lib/connectivity/trafficShapeParser.js#L5-L104. Changed in #1540. If you wanna run with the legacy setting use --legacyConnectivityProfiles. See #1160 for the original change.
    • The default minimum wait time for waiting if a test is finished is now 8 seconds (instead of 5) #1542.
    • Fix --chrome.blockDomainsExcept when you are using WebPageReplay #1532. Thank you Inderpartap Singh Cheema for the original fix!
    • Make sure gnirehtet is closed on the right device using device id #1527
    • Upgraded to GeckoDriver 0.29.1.
    • Updated wpr_cert.pem to a new version for WebPageReplay #1316.
    • Include Google Web Vitals in the HAR file #1535.
    • Added 3 seconds wait time for geckoprofiler to start see #1538 and #1539
    • Added a ten seconds wait if getting the HAR from Firefox fails the first time #1546 fixes #3346.
    • New metrics: Delta between TTFB and First Contentful Paint, Largest Contentful paint and First visual change #1528. You can use this if you have unstable TTFB and want to alert on front end metrics. Lets see when other also implement this :)
    • Made it easier for people to get Google Web Vitals. We copy that data under the googleWebVitals namespace in the result JSON #1521.
    • Added TTFB as a single metric #1522.
    • New stop watch command #1512. Measure time by:
         timer.start(); 
        // Do something
        // Stop the timer and add the result to the last tested URL
        timer.stopAndAdd();
      
    • Pre test/warm a URL with --preWarmServer. Do that to make sure your server has cached everything that is needed before your test #1515 and #1516.
    • Collect what HTML element change in cumulative layout shifts #1534
    • Added support for recording video on Safari iOS #1541.
    • New commands: scrolling by Pixels, Lines or Pages forward, back or refresh navigations, create new tabs or windows and switch to them and new mouse events such as context click, single click, double click, click and hold, release, and movement. Thank you Denis Palmeiro for PR #1533.
    • Improve proxy configuration support, thank you Olaf Meeuwissen for PR #1542.
    • Upgraded to Selenium 4.0.0-beta.3 #1544.
    • Updated to Chrome 90 in the Docker container and Chromedriver 90 as default #1543.
  • Added --preWarmServer that makes a request to your web server to fill the server cache before you start your tests.
  • Send navigation timings metrics by default to Graphite/Influx #3316.
  • Updated the wpt-plugin to send TBT and CLS to data storage.
  • Updated the Lighthouse plugin to send Goggle Web Vitals to data storage.
  • Updated to Lighthouse 7.3.0 in the Lighthouse plugin.
  • Lighthouse and GPSI plugins published to npm
  • Updated the GPSI plugin to send Google Web Vitals to data storage (and show it in the HTML).
  • Renamed layouShift to cumulativeLayoutShift
  • Send TTFB and Google Web Vitals by default to data storage for Browsertime (making it easier to find them in Influx).

Fixed

  • Updated to latest aws-sdk , google cloud storage and axe-core 4.1.4 #3334
  • Updated to latest Coach and PageXray.

16.10.3 - 2021-03-17

Fixed

  • Updated to Browsertime 11.6.3 since last version broke getting the netlog on desktop for Chrome.

16.10.2 - 2021-03-17

Fixed

  • Added missing download link in the HTML for the Chrome netlog #3315

16.10.1 - 2021-03-17

Fixed

  • Updated to Browsertime 11.6.2 with new Selenium beta 4 and a fix for getting the netlog for Chrome on Android.

16.10.0 - 2021-03-11

Added

  • Updated to Browsertime 11.6.0 with Chromedriver 89, Edgedriver 89.
  • Updated Docker containers to use Chrome 89 and Firefox 86.

Fixed

  • The Lighthouse plugin was upgraded to Lighthouse 7.2.0 and fixed broken support for alias.
  • Updated to AXE-core 4.1.3 #3302.
  • More fixes for showing correct screenshots #3305 and #3306.
  • Updated to Browsertime 11.6.1 that makes sure TSProxy uses Python2 when you run in a Docker container.

16.9.2 - 2021-02-24

Fixed

  • Another screenshot fix, to make sure we support whatever image format when we copy the image from the last run #3290.

16.9.1 - 2021-02-24

Fixed

  • The screenshot fix in 16.9.0 introduced an error when you copy latest images to the parent dir (upcoming feature in 17.0). Fixed in #3288.

16.9.0 - 2021-02-24

Added

  • Upgraded to Coach core 6.1 that finds Google reCAPTCHAs #3284.
  • Upgraded to Browsertime 11.5.0 that makes it easier to use gnirehtet #3281.

Fixed

  • Upgraded to AXE-core 4.1.2 #3282
  • Screenshots was not shown in the screenshot tab as reported in #3286 fixed in #3278.

16.8.1 - 2021-02-12

Fixed

  • Avoid sending slug/domain annotation names that collide. Fixed in #3279 and reported in #3277.

16.8.0 - 2021-02-08

Added

  • Updated Browsertime with the ability to mark a run as a failure. We gonna add more docs and try this ourselves and push it in 17.0 #3272.
  • Updated to latest VideoJS and changed how the video is displayed #3268.

Fixed

  • The --addSlugToKey command introduced 16.3.0 was broken and inserted the slug at the wrong place. Fixed now and we gonna push it and the documentation on how to use it in 17.0 #3274.

16.7.1 - 2021-02-02

Fixed

  • Updated to Browsertime 11.3.1 that open/closes the iOS simulator at the start/end of the test.

16.7.0 - 2021-02-01

Added

  • Updated to Browsertime 11.3.0 with better iOS simulator support. There are still some work to do, but you can try it out with sitespeed.io https://www.wikipedia.org -b safari --safari.useSimulator --safari.deviceUDID YOUR_DEVICE_ID --video --visualMetrics. List your device ids with xcrun simctl list devices.

16.6.0 - 2021-01-25

Added

  • Upgraded to Browsertime 11.2.0 with support for listening on events for CDP in scripting.

16.5.0 - 2021-01-20

Added

  • Upgraded to Chrome 88 in the Docker container and updated Browsertime that uses Chromedriver 88. Also upgraded Browsertime to 11.1.2 that fixes a bug if Browsertime do not collect any metrics, that broke testing with WebPageReplay.

16.4.0 - 2021-01-20

Added

  • Added support for GCS to also upload latest screenshots/video #3258-
  • Automatically create a result base URL if you upload to GCS (and do not configure one) #3259.
  • Include index.html when we log the link to the S3/GCS result bucket to make it easier to copy/paste #3260.

Fixed

  • Updated to Browsertime 11.0.2 that hopefully fixes the problem when you use an alias in scripting and the URL change between runs.

16.3.2 - 2021-01-14

Fixed

  • Make sure the right connectivity name is picked up for the latest copied file. Before the fix using alias for your connectivity didn't work #3255.

16.3.1 - 2021-01-14

Fixed

  • Fixed broken URL in Slack message (if you used an alias) #3254
  • Make sure alias from the cli, from file or from script is handled the same way #3253.
  • Added missing slug in annotation #3251.
  • Add missing browser/connectivity name in the latest files #3249.

16.3.0 - 2021-01-13