SiteSpeed test exits with 1 with suppressExitCode in place

[niskhakova-gitlab]

Have you read the documentation?

• Yes, I've read the how to make a reproducable bug guide
• Yes, I've read the how to debug my script guide

URL

https://staging.gitlab.com/explore

What are you trying to accomplish

When using SiteSpeed v31.1.x version, exit code is 1 with budget.suppressExitCode in place. When running the same command using v31.0.x - exit code is 0.

Might be related to the recent change in v31.1 - #4044 but needs to be confirmed. Thanks in advance!

What browser did you use?

Chrome

How to reproduce

## sha256:5f212ca4ae23de9888f00ab089471277b0f93d41a572a95658d6b60d71abaac8
export SITESPEED_IMAGE=sitespeedio/sitespeed.io:latest

# budget.json https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/blob/main/budget.json?ref_type=heads, config.json https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/blob/main/config.json?ref_type=heads
docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io $SITESPEED_IMAGE --config config.json --budget.suppressExitCode --budget.configPath budget.json  https://staging.gitlab.com/explore -n 1

echo $?
1

# Using older image
export SITESPEED_IMAGE=sitespeedio/sitespeed.io:31.0.1

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io $SITESPEED_IMAGE --config config.json --budget.suppressExitCode --budget.configPath budget.json  https://staging.gitlab.com/explore -n 1

echo $?
0

Log output

https://staging.gitlab.com/explore -n 1
Google Chrome 120.0.6099.109
Mozilla Firefox 121.0
Microsoft Edge 120.0.2210.77
[2024-01-16 12:38:14] INFO: Versions OS: linux 6.2.0-1018-gcp nodejs: v20.9.0 sitespeed.io: 31.1.1 browsertime: 21.0.0 coach: 8.0.2
[2024-01-16 12:38:14] INFO: Running tests using Chrome - 1 iteration(s)
[2024-01-16 12:38:15] INFO: Use Chrome trace categories: -*,disabled-by-default-lighthouse,v8,v8.execute,blink.user_timing,devtools.timeline,disabled-by-default-devtools.timeline,disabled-by-default-devtools.timeline.stack
[2024-01-16 12:38:15] INFO: Testing url https://staging.gitlab.com/explore iteration 1
[2024-01-16 12:38:43] INFO: Take after page complete check screenshot
[2024-01-16 12:38:45] INFO: Take cumulative layout shift screenshot
[2024-01-16 12:38:46] INFO: Take largest contentful paint screenshot
[2024-01-16 12:38:50] INFO: Use the visual metrics portable script
[2024-01-16 12:38:50] INFO: Get visual metrics from the video
[2024-01-16 12:38:58] INFO: https://staging.gitlab.com/explore 29 requests, TTFB: 697ms, firstPaint: 1.30s, firstVisualChange: 1.27s, FCP: 1.30s, DOMContentLoaded: 1.83s, LCP: 1.30s, CLS: 0.1819, TBT: 449ms, CPUBenchmark: 64ms, Load: 1.90s, speedIndex: 1.30s, visualComplete85: 1.27s, lastVisualChange: 1.90s
[2024-01-16 12:38:59] INFO: Failing budget First Contentful Paint for https://staging.gitlab.com/explore with value 1.296 s max limit 1.000 s
[2024-01-16 12:38:59] INFO: Failing budget Total Blocking Time for https://staging.gitlab.com/explore with value 449 ms max limit 300 ms
[2024-01-16 12:38:59] INFO: Failing budget Cumulative Layout Shift for https://staging.gitlab.com/explore with value 0.182 max limit 0.1
[2024-01-16 12:38:59] INFO: Budget: 3 working, 3 failing tests and 0 errors
[2024-01-16 12:39:01] INFO: HTML stored in /sitespeed.io/sitespeed-result/staging.gitlab.com/2024-01-16-12-38-14
1 page analysed for https://staging.gitlab.com/exp... (1 run, Chrome/desktop/[object Object])
   Score / Metric                Median
   -------------                 ------
!  First Paint                   1.296 s
√  First Contentful Paint        1.296 s
   Fully Loaded                  2.305 s
   Page Load Time                1.902 s
√  Largest Contentful Paint      1.296 s
!  First Visual Change           1.266 s
   Last Visual Change            1.900 s
√  Speed Index                   1.305 s
!  Visual Readiness              634 ms
   TTFB                          697 ms
   Cumulative Layout Shift       0.182
✗  Total Duration of Long Tasks  664 ms
!  Number of Long Tasks          2
!  Total Blocking Time           449 ms
✗  Max Potential FID             499 ms
√  Coach Overall Score           92
√  Coach Best Practice Score     91
√  Coach Privacy Score           98
!  Coach Performance Score       89
√  Total Requests                29
   Image Requests                2
✗  Total Transfer Size           1.5 MB
   HTML Transfer Size            12.4 KB
   JavaScript Transfer Size      538.9 KB
✗  JavaScript Content Size       1.7 MB
   CSS Transfer Size             133.2 KB
   Image Transfer Size           6.1 KB

[soulgalore]

That looks like something I broke, sorry. I didn't remember we had that suppress functionality. Let me fix that for the next release.

[soulgalore]

I released 31.2.0 today, hopefully that will fix the problem, please let me know when you have tried it.

[niskhakova-gitlab]

@soulgalore many thanks for the quick turnaround on this! I tried the latest version and unfortunately exit code is 1 with --budget.suppressExitCode in place. Please let me know if something's needed from my side.

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:latest --config config.json --budget.suppressExitCode --budget.configPath budget.json  https://staging.gitlab.com/explore -n 1

Google Chrome 120.0.6099.109
Mozilla Firefox 121.0
Microsoft Edge 120.0.2210.77
[2024-01-18 10:22:13] INFO: Versions OS: linux 6.2.0-1018-gcp nodejs: v20.9.0 sitespeed.io: 31.2.0 browsertime: 21.0.0 coach: 8.0.2
[2024-01-18 10:22:13] INFO: Running tests using Chrome - 1 iteration(s)
[2024-01-18 10:22:14] INFO: Use Chrome trace categories: -*,disabled-by-default-lighthouse,v8,v8.execute,blink.user_timing,devtools.timeline,disabled-by-default-devtools.timeline,disabled-by-default-devtools.timeline.stack
[2024-01-18 10:22:14] INFO: Testing url https://staging.gitlab.com/explore iteration 1
[2024-01-18 10:22:42] INFO: Take after page complete check screenshot
[2024-01-18 10:22:44] INFO: Take cumulative layout shift screenshot
[2024-01-18 10:22:46] INFO: Take largest contentful paint screenshot
[2024-01-18 10:22:50] INFO: Use the visual metrics portable script
[2024-01-18 10:22:50] INFO: Get visual metrics from the video
[2024-01-18 10:23:00] INFO: https://staging.gitlab.com/explore 28 requests, TTFB: 769ms, firstPaint: 1.36s, firstVisualChange: 1.37s, FCP: 1.36s, DOMContentLoaded: 1.81s, LCP: 1.36s, CLS: 0.0063, TBT: 0ms, CPUBenchmark: 64ms, Load: 1.95s, speedIndex: 1.52s, visualComplete85: 2.03s, lastVisualChange: 2.03s
[2024-01-18 10:23:00] INFO: Failing budget First Contentful Paint for https://staging.gitlab.com/explore with value 1.363 s max limit 1.000 s
[2024-01-18 10:23:00] INFO: Budget: 4 working, 1 failing tests and 0 errors
[2024-01-18 10:23:03] INFO: HTML stored in /sitespeed.io/sitespeed-result/staging.gitlab.com/2024-01-18-10-22-13
1 page analysed for https://staging.gitlab.com/exp... (1 run, Chrome/desktop/[object Object])
   Score / Metric                Median
   -------------                 ------
!  First Paint                   1.363 s
√  First Contentful Paint        1.363 s
   Fully Loaded                  2.177 s
   Page Load Time                1.950 s
√  Largest Contentful Paint      1.363 s
!  First Visual Change           1.366 s
   Last Visual Change            2.033 s
√  Speed Index                   1.517 s
!  Visual Readiness              667 ms
   TTFB                          769 ms
   Cumulative Layout Shift       0.006
✗  Total Duration of Long Tasks  686 ms
✗  Number of Long Tasks          3
√  Total Blocking Time           0 ms
√  Max Potential FID             0 ms
√  Coach Overall Score           93
√  Coach Best Practice Score     99
√  Coach Privacy Score           98
!  Coach Performance Score       89
√  Total Requests                28
   JavaScript Requests           12
   CSS Requests                  5
✗  Total Transfer Size           1.5 MB
   HTML Transfer Size            12.4 KB
   JavaScript Transfer Size      539.3 KB
✗  JavaScript Content Size       1.7 MB
   CSS Transfer Size             133.5 KB
   Image Transfer Size           1.1 KB

 echo $?
1

[soulgalore]

Sorry new try with 31.2.1. This time I could verify that it works for me.

[niskhakova-gitlab]

@soulgalore thank you very much for the quick fix, truly appreciate the help on this! 🙇 Confirmed that it works 🎉