[CI] [v0.11] lighthouseOptions.throttlingMethod always set to provided

[mwskwong]

Describe the bug

Despite throttle is true by default, lighthouseOptions.throttlingMethod always has the value provided, effectively disabling throttle entirely. Explicitly setting lighthouseOptions.throttlingMethod in the config file doesn't work as well.

Command used:

unlighthouse-ci --site <url> --debug

Config file:

export default {
  cache: false,
  ci: {
    buildStatic: true,
    budget: {
      performance: 90,
      accessibility: 91,
      'best-practices': 100,
      seo: 92,
    },
  },
};

Debug output (some of the paths are masked):

[Unlighthouse 11:18:22 AM] ⚙ Starting Unlighthouse at root: / cwd: /
[Unlighthouse 11:18:22 AM] ⚙ Discovered config definition { config: { cache: false, ci: { buildStatic: true, budget: [Object] } },
  sources: [ '/unlighthouse.config.ts' ] }
⚙ Unable to locale page files, disabling route discovery.                           Unlighthouse 11:18:22 AM
[Unlighthouse 11:18:22 AM] ℹ Using system chrome located at: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome.
⚙ Post config resolution { routerPrefix: '/',                                       Unlighthouse 11:18:22 AM
  apiPrefix: '/api',
  cache: false,
  client:
   { groupRoutesKey: 'route.definition.name',
     columns:
      { overview: [Array],
        performance: [Array],
        accessibility: [Array],
        'best-practices': [Array],
        seo: [Array] } },
  scanner:
   { customSampling: {},
     ignoreI18nPages: true,
     maxRoutes: 200,
     skipJavascript: true,
     samples: 1,
     throttle: true,
     crawler: true,
     dynamicSampling: 8,
     sitemap: true,
     robotsTxt: true,
     device: 'mobile',
     exclude: [ '/cdn-cgi/*' ] },
  server: { port: 5678, showURL: false, open: true },
  discovery: false,
  root: '/',
  outputPath: '/.unlighthouse',
  debug: true,
  puppeteerOptions:
   { headless: true,
     ignoreHTTPSErrors: true,
     defaultViewport:
      { mobile: true, width: 412, height: 823, deviceScaleFactor: 1.75, disabled: false },
     executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' },
  puppeteerClusterOptions:
   { monitor: true,
     workerCreationDelay: 500,
     retryLimit: 3,
     timeout: 300000,
     maxConcurrency: 7,
     skipDuplicateUrls: false,
     retryDelay: 2000,
     concurrency: 3,
     puppeteer:
      PuppeteerNode {
        _isPuppeteerCore: true,
        _changedProduct: false,
        connect: [Function: bound bound connect],
        defaultBrowserRevision: '122.0.6261.69',
        configuration: [Object],
        launch: [Function: bound launch],
        executablePath: [Function: bound executablePath],
        defaultArgs: [Function: bound defaultArgs],
        trimCache: [Function: bound trimCache] AsyncFunction } },
  lighthouseOptions:
   { onlyCategories: [ 'performance', 'accessibility', 'best-practices', 'seo' ],
     throttlingMethod: 'provided',
     throttling:
      { rttMs: 300,
        throughputKbps: 700,
        requestLatencyMs: 1125,
        downloadThroughputKbps: 2625,
        uploadThroughputKbps: 2625,
        cpuSlowdownMultiplier: 4 },
     formFactor: 'mobile',
     screenEmulation:
      { mobile: true, width: 412, height: 823, deviceScaleFactor: 1.75, disabled: false },
     emulatedUserAgent:
      'Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36' },
  urls: [],
  site: '<url>',
  ci:
   { budget: { performance: 90, accessibility: 91, 'best-practices': 100, seo: 92 },
     buildStatic: true,
     reporter: 'jsonSimple',
     reporterConfig: { lhciHost: undefined, lhciBuildToken: undefined } },
  hooks: { 'resolved-config': [AsyncFunction: resolved-config] },
  chrome:
   { useSystem: true,
     useDownloadFallback: true,
     downloadFallbackVersion: 1095492,
     downloadFallbackCacheDir: '/.unlighthouse' } }

Reproduction

See above

System / Nuxt Info

System:
    OS: macOS 14.2.1
    CPU: (8) arm64 Apple M1 Pro
    Memory: 1.26 GB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.19.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.2.4 - /usr/local/bin/npm
    pnpm: 8.15.4 - ~/Library/pnpm/pnpm
  Browsers:
    Brave Browser: 110.1.48.171
    Chrome: 121.0.6167.184
    Edge: 122.0.2365.63
    Firefox Nightly: 122.0a1
    Safari: 17.2.1

Also reproducible in GitHub Actions ubuntu-latest

[harlan-zw]

Hi @mwskwong, thanks for the issue.

I've pushed up a fix for overriding the throttling.

v0.11 was meant to re-introduce throttling on by default but the configuration was actually set up incorrectly, I've gone with a custom throttling config to account for the CPU and network already getting trashed. Saying that, Unlighthouse has been and will never be for producing accurate performance metrics that are effected by variability.

Try out 0.11.4 and let me know if it's working better for you.

[mwskwong]

Hi @mwskwong, thanks for the issue.

I've pushed up a fix for overriding the throttling.

v0.11 was meant to re-introduce throttling on by default but the configuration was actually set up incorrectly, I've gone with a custom throttling config to account for the CPU and network already getting trashed. Saying that, Unlighthouse has been and will never be for producing accurate performance metrics that are effected by variability.

Try out 0.11.4 and let me know if it's working better for you.

For the original intent of this issue, it seems to load the correct config now in v0.11.4. However, would you mind elaborating on the "we need a custom throttling profile to account for the CPU/ network already getting blasted" part? i.e. the rationale of loosening the throttling.

[harlan-zw]

The issue to solve is that the performance scores reported by Unlighthouse are either always too low (when throttling is enabled) or too high (because throttling is disabled). Users expect to see performance scores matching PSI.

While we're running multi-threaded Lighthouse scans there isn't really a solution besides fine-tuning the throttling