Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task log output is too verbose, difficult to read #950

Closed
14 tasks done
dbjorge opened this issue Nov 2, 2021 · 1 comment · Fixed by #1075
Closed
14 tasks done

Task log output is too verbose, difficult to read #950

dbjorge opened this issue Nov 2, 2021 · 1 comment · Fixed by #1075
Labels
feature request status: resolved This issue has been merged into main and deployed to canary.

Comments

@dbjorge
Copy link
Contributor

dbjorge commented Nov 2, 2021

Summary

The task log output from both the ADO extension and GitHub action include a ton of noise that users shouldn't need to parse through; the actual actionable information from a run is buried in the middle of the log and not highlighted meaningfully.

Pipelines and Actions both support specific syntax for stdout messages to enable different types of grouping/highlighting/etc; while designing a logger to support the suggestions in this issue, be sure to consider how to handle both systems' different syntax.

Actual vs expected

Actual output from an example run with a required baseline update (case 4 of ado-extension-staging-validation) - 169 lines
2021-11-02T19:50:08.9814274Z ##[section]Starting: [should fail] case 4: baseline missing failures
2021-11-02T19:50:08.9821025Z ==============================================================================
2021-11-02T19:50:08.9821509Z Task         : Accessibility Insights Azure DevOps Task
2021-11-02T19:50:08.9821883Z Description  : Scan accessibility issues in an Azure DevOps pipeline
2021-11-02T19:50:08.9822179Z Version      : 1.1.0
2021-11-02T19:50:08.9822455Z Author       : Accessibility Insights
2021-11-02T19:50:08.9822703Z Help         : 
2021-11-02T19:50:08.9822998Z ==============================================================================
2021-11-02T19:50:09.1991936Z installing runtime dependencies...
2021-11-02T19:50:09.3385233Z yarn install v1.22.17
2021-11-02T19:50:09.4468120Z [1/4] Resolving packages...
2021-11-02T19:50:09.4777135Z warning Lockfile has incorrect entry for "apify-shared@^0.5.0". Ignoring it.
2021-11-02T19:50:09.5120446Z warning Lockfile has incorrect entry for "socket.io@^2.3.0". Ignoring it.
2021-11-02T19:50:09.5122034Z warning Lockfile has incorrect entry for "underscore@1.12.1". Ignoring it.
2021-11-02T19:50:09.5385119Z warning Lockfile has incorrect entry for "apify-shared@^0.6.0". Ignoring it.
2021-11-02T19:50:09.5403086Z warning Lockfile has incorrect entry for "apify-shared@^0.1.45". Ignoring it.
2021-11-02T19:50:09.5796664Z warning Lockfile has incorrect entry for "http-signature@~1.2.0". Ignoring it.
2021-11-02T19:50:09.5822215Z warning Lockfile has incorrect entry for "css-what@2.1". Ignoring it.
2021-11-02T19:50:09.5824908Z warning Lockfile has incorrect entry for "nth-check@~1.0.1". Ignoring it.
2021-11-02T19:50:09.7566166Z warning accessibility-insights-scan > apify > socket.io > debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:09.7651204Z warning accessibility-insights-scan > apify > socket.io > socket.io-parser > debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:09.7692334Z warning accessibility-insights-scan > apify > socket.io > engine.io > debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:10.1510415Z warning Lockfile has incorrect entry for "axios@^0.21.1". Ignoring it.
2021-11-02T19:50:10.1511332Z warning Lockfile has incorrect entry for "marked@^2.0.0". Ignoring it.
2021-11-02T19:50:10.6201531Z [2/4] Fetching packages...
2021-11-02T19:50:10.8566642Z [3/4] Linking dependencies...
2021-11-02T19:50:10.8575211Z warning "accessibility-insights-report > office-ui-fabric-react@7.98.1" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8576645Z warning "accessibility-insights-report > office-ui-fabric-react@7.98.1" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8586296Z warning "accessibility-insights-report > office-ui-fabric-react > @fluentui/react-focus@7.18.1" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8587662Z warning "accessibility-insights-report > office-ui-fabric-react > @fluentui/react-focus@7.18.1" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8590382Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/foundation@7.10.1" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8591727Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/foundation@7.10.1" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8594265Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks@7.14.0" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8595907Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks@7.14.0" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8598361Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/utilities@7.33.5" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8599922Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/utilities@7.33.5" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8603536Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks > @fluentui/react-window-provider@1.0.2" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8605002Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks > @fluentui/react-window-provider@1.0.2" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8606870Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/styling > @fluentui/theme@1.7.4" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8609021Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/styling > @fluentui/theme@1.7.4" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:12.4438052Z [4/4] Building fresh packages...
2021-11-02T19:50:12.4640699Z Done in 3.13s.
2021-11-02T19:50:14.0667002Z [Trace][info] === Starting accessibility scanning of URL http://localhost:5858
2021-11-02T19:50:14.0668807Z [Trace][info] === Chrome app executable: /usr/bin/google-chrome
2021-11-02T19:50:14.1113410Z INFO  PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to 1738 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
2021-11-02T19:50:14.1138556Z INFO  PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
2021-11-02T19:50:14.1247887Z INFO  Launching Puppeteer {"args":["--disable-dev-shm-usage","--no-sandbox","--disable-setuid-sandbox","--js-flags=--max-old-space-size=8192","--no-sandbox","--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"],"defaultViewport":{"width":1920,"height":1080,"deviceScaleFactor":1},"useChrome":true,"timeout":15000,"headless":true,"executablePath":"/usr/bin/google-chrome"}
2021-11-02T19:50:15.2863699Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.4885365Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.6899873Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.8998253Z Processing page http://localhost:5858/
2021-11-02T19:50:15.9069421Z Discovered 2 links on page http://localhost:5858/
2021-11-02T19:50:16.1213249Z Found 3 accessibility issues on page http://localhost:5858/
2021-11-02T19:50:17.1786109Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.3801515Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.5817595Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.7840499Z Processing page http://localhost:5858/linked1/
2021-11-02T19:50:17.7893439Z Discovered 2 links on page http://localhost:5858/linked1/
2021-11-02T19:50:18.9931340Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.1954982Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.3970696Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.5997987Z Processing page http://localhost:5858/linked2/
2021-11-02T19:50:19.6044738Z Discovered 1 links on page http://localhost:5858/linked2/
2021-11-02T19:50:20.7927508Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:20.9945812Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:21.1960170Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:21.3989854Z Processing page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.4034314Z Discovered 1 links on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5728322Z Found 2 accessibility issues on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5803110Z INFO  PuppeteerCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
2021-11-02T19:50:21.5824184Z INFO  PuppeteerCrawler: Final request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1864,"requestsFinishedPerMinute":32,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7454,"requestsTotal":4,"crawlerRuntimeMillis":7492,"requestsFinished":4,"requestsFailed":0,"retryHistogram":[4]}
2021-11-02T19:50:21.6436274Z Warning: Each child in a list should have a unique "key" prop.
2021-11-02T19:50:21.6437698Z 
2021-11-02T19:50:21.6439176Z Check the top-level render call using <ul>. See https://fb.me/react-warning-keys for more information.
2021-11-02T19:50:21.6442305Z     in li
2021-11-02T19:50:21.6442623Z     in SummaryReportDetailsSection
2021-11-02T19:50:21.6442929Z     in div
2021-11-02T19:50:21.6443185Z     in main
2021-11-02T19:50:21.6443472Z     in ContentSection
2021-11-02T19:50:21.6443757Z     in body
2021-11-02T19:50:21.6444016Z     in BodySection
2021-11-02T19:50:21.6444303Z     in ReportBody
2021-11-02T19:50:21.6780477Z [Trace][info] === Scan report saved successfully as /home/vsts/work/1/s/_accessibility-reports-case-4/index.html
2021-11-02T19:50:21.6785266Z Found 2 new violations compared to the baseline.
2021-11-02T19:50:21.6787446Z Found 0 cases where a previously baselined violation was fixed.
2021-11-02T19:50:21.6805177Z Saved new baseline file at /home/vsts/work/1/s/_accessibility-reports-case-4/missing-failures-5858.baseline
2021-11-02T19:50:21.6808175Z To update the baseline with these changes, either rerun with --updateBaseline or copy the updated baseline file to /home/vsts/work/1/s/dev/website-baselines/missing-failures-5858.baseline
2021-11-02T19:50:21.6849306Z [Exception] === { VError: An error occurred while scanning website page http://localhost:5858: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6851053Z     at Logger.trackExceptionAny (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:132995:29)
2021-11-02T19:50:21.6852551Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133836:29)
2021-11-02T19:50:21.6853365Z     at Generator.throw (<anonymous>)
2021-11-02T19:50:21.6854334Z     at rejected (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133765:65)
2021-11-02T19:50:21.6855077Z   jse_shortmsg:
2021-11-02T19:50:21.6855923Z    'An error occurred while scanning website page http://localhost:5858',
2021-11-02T19:50:21.6856605Z   jse_cause:
2021-11-02T19:50:21.6857254Z    Error: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6858444Z        at WorkflowEnforcer.failIfBaselineNeedsUpdating (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129208:19)
2021-11-02T19:50:21.6859691Z        at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129192:18)
2021-11-02T19:50:21.6860497Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6861449Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6862197Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6863352Z        at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6864662Z        at WorkflowEnforcer.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129190:16)
2021-11-02T19:50:21.6865847Z        at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:45)
2021-11-02T19:50:21.6867034Z        at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6867841Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6868774Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6869520Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6870670Z        at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6871973Z        at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6875440Z        at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:28)
2021-11-02T19:50:21.6876060Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6876788Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6877492Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6878432Z        at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6879810Z        at AllProgressReporter.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133399:16)
2021-11-02T19:50:21.6880811Z        at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133833:48)
2021-11-02T19:50:21.6881387Z        at Generator.next (<anonymous>),
2021-11-02T19:50:21.6881761Z   jse_info: {},
2021-11-02T19:50:21.6882092Z   message:
2021-11-02T19:50:21.6882975Z    'An error occurred while scanning website page http://localhost:5858: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.' }
2021-11-02T19:50:21.6890015Z [Trace][info] === Accessibility scanning of URL http://localhost:5858 completed
2021-11-02T19:50:21.6897695Z Exception thrown in extension:  Error: Error: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6898997Z     at WorkflowEnforcer.failIfBaselineNeedsUpdating (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129208:19)
2021-11-02T19:50:21.6900055Z     at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129192:18)
2021-11-02T19:50:21.6900645Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6901364Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6901906Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6902835Z     at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6903915Z     at WorkflowEnforcer.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129190:16)
2021-11-02T19:50:21.6904842Z     at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:45)
2021-11-02T19:50:21.6905808Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6906390Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6907121Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6907627Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6908559Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6909642Z     at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6910636Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:28)
2021-11-02T19:50:21.6911216Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6911925Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6912449Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6913366Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6914634Z     at AllProgressReporter.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133399:16)
2021-11-02T19:50:21.6915591Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133833:48)
2021-11-02T19:50:21.6916155Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6916948Z     at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129198:19)
2021-11-02T19:50:21.6917529Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6918233Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6918753Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6919810Z     at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6921026Z     at WorkflowEnforcer.failRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129197:16)
2021-11-02T19:50:21.6921961Z     at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133410:41)
2021-11-02T19:50:21.6922903Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6923480Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6924217Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6924744Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6925646Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6926730Z     at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6927711Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133410:24)
2021-11-02T19:50:21.6928289Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6929013Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6929520Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6930433Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6931512Z     at AllProgressReporter.failRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133409:16)
2021-11-02T19:50:21.6932485Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133837:48)
2021-11-02T19:50:21.6933035Z     at Generator.throw (<anonymous>)
2021-11-02T19:50:21.6933812Z     at rejected (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133765:65)
2021-11-02T19:50:21.7300172Z ##[error]Exit code 1 returned from process: file name '/home/vsts/agents/2.194.0/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js"'.
2021-11-02T19:50:21.7316103Z ##[section]Finishing: [should fail] case 4: baseline missing failures
An example of what I think this should look like - 46 lines, with grouping and highlighting
2021-11-02T19:50:08.9814274Z ##[section]Starting: [should fail] case 4: baseline missing failures
2021-11-02T19:50:08.9821025Z ==============================================================================
2021-11-02T19:50:08.9821509Z Task         : Accessibility Insights Azure DevOps Task
2021-11-02T19:50:08.9821883Z Description  : Scan accessibility issues in an Azure DevOps pipeline
2021-11-02T19:50:08.9822179Z Version      : 1.1.0
2021-11-02T19:50:08.9822455Z Author       : Accessibility Insights
2021-11-02T19:50:08.9822703Z Help         : 
2021-11-02T19:50:08.9822998Z ==============================================================================
2021-11-02T19:50:09.1991936Z ##[group]Installing runtime dependencies
2021-11-02T19:50:09.3385233Z yarn install v1.22.17
2021-11-02T19:50:09.4468120Z [1/4] Resolving packages...
2021-11-02T19:50:10.6201531Z [2/4] Fetching packages...
2021-11-02T19:50:10.8566642Z [3/4] Linking dependencies...
2021-11-02T19:50:12.4438052Z [4/4] Building fresh packages...
2021-11-02T19:50:12.4640699Z Done in 3.13s.
2021-11-02T19:50:12.4640699Z ##[endgroup]
2021-11-02T19:50:12.4640699Z ##[group]Scanning URL http://localhost:5858 for accessibility issues
2021-11-02T19:50:14.0667002Z ##[debug]Starting accessibility scanning of URL http://localhost:5858
2021-11-02T19:50:14.0668807Z ##[debug]Chrome app executable: /usr/bin/google-chrome
2021-11-02T19:50:14.1113410Z ##[debug]PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to 1738 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
2021-11-02T19:50:14.1138556Z ##[debug]PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
2021-11-02T19:50:14.1247887Z ##[debug]Launching Puppeteer {"args":["--disable-dev-shm-usage","--no-sandbox","--disable-setuid-sandbox","--js-flags=--max-old-space-size=8192","--no-sandbox","--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"],"defaultViewport":{"width":1920,"height":1080,"deviceScaleFactor":1},"useChrome":true,"timeout":15000,"headless":true,"executablePath":"/usr/bin/google-chrome"}
2021-11-02T19:50:15.8998253Z Processing page http://localhost:5858/
2021-11-02T19:50:15.9069421Z Discovered 2 links on page http://localhost:5858/
2021-11-02T19:50:16.1213249Z Found 3 accessibility issues on page http://localhost:5858/
2021-11-02T19:50:17.7840499Z Processing page http://localhost:5858/linked1/
2021-11-02T19:50:17.7893439Z Discovered 2 links on page http://localhost:5858/linked1/
2021-11-02T19:50:19.5997987Z Processing page http://localhost:5858/linked2/
2021-11-02T19:50:19.6044738Z Discovered 1 links on page http://localhost:5858/linked2/
2021-11-02T19:50:21.3989854Z Processing page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.4034314Z Discovered 1 links on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5728322Z Found 2 accessibility issues on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5803110Z ##[debug]PuppeteerCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
2021-11-02T19:50:21.5824184Z ##[debug]PuppeteerCrawler: Final request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1864,"requestsFinishedPerMinute":32,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7454,"requestsTotal":4,"crawlerRuntimeMillis":7492,"requestsFinished":4,"requestsFailed":0,"retryHistogram":[4]}
2021-11-02T19:50:21.5824184Z ##[endgroup]
2021-11-02T19:50:21.6780477Z Scan report saved successfully as /home/vsts/work/1/s/_accessibility-reports-case-4/index.html
2021-11-02T19:50:21.6897695Z ##vso[task.logissue type=error;sourcepath=dev/website-baselines/missing-failures-5858.baseline]The baseline file does not match scan results.
2021-11-02T19:50:21.6897695Z ##[error]The baseline file does not match scan results.
2021-11-02T19:50:21.6785266Z Found 2 new violations compared to the baseline.
2021-11-02T19:50:21.6787446Z Found 0 cases where a previously baselined violation was fixed.
2021-11-02T19:50:21.6805177Z Saved new baseline file at /home/vsts/work/1/s/_accessibility-reports-case-4/missing-failures-5858.baseline
2021-11-02T19:50:21.6808175Z To update the baseline with these changes:
2021-11-02T19:50:21.6808175Z     1. Ensure you've published the output directory as an "accessibility reports" artifact, and
2021-11-02T19:50:21.6808175Z     2. Copy the updated baseline file from your artifact to dev/website-baselines/missing-failures-5858.baseline
2021-11-02T19:50:21.7300172Z ##[error]Exit code 1 returned from process: file name '/home/vsts/agents/2.194.0/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js"'.
2021-11-02T19:50:21.7316103Z ##[section]Finishing: [should fail] case 4: baseline missing failures

Specific suggested improvements

I think each of these would work fine as independent improvements, no need to wait until all of them are done to make incremental progress.

These are all written from the perspective of the ADO extension, but ideally any ## commands would want to be able to use GitHub Actions equivalents there.

2021-11-02T19:50:21.6897695Z ##vso[task.logissue type=error;sourcepath=dev/website-baselines/missing-failures-5858.baseline]The baseline file does not match scan results.

(Fixed in #1075 )

  • Improve the action callout for the "baseline update required" case by replacing "If this is a PR, check the PR comments." with something that is actionable in the non-PR case, eg:
2021-11-02T19:50:21.6808175Z To update the baseline with these changes:
2021-11-02T19:50:21.6808175Z     1. Ensure you've published the output directory as an "accessibility reports" artifact, and
2021-11-02T19:50:21.6808175Z     2. Copy the updated baseline file from your artifact to dev/website-baselines/missing-failures-5858.baseline

(Fixed in #909)

@ghost ghost added the status: new This issue is new and requires triage by DRI. label Nov 2, 2021
@ghost ghost assigned karanbirsingh Nov 2, 2021
@dbjorge dbjorge changed the title ADO extension: task log output is too verbose, difficult to read Task log output is too verbose, difficult to read Nov 2, 2021
@karanbirsingh karanbirsingh added the status: ready for triage This issue is ready to be triaged by the Accessibility Insights team. label Nov 3, 2021
@ghost ghost assigned asksep Nov 3, 2021
@ghost
Copy link

ghost commented Nov 3, 2021

This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!

@ghost ghost removed the status: new This issue is new and requires triage by DRI. label Nov 3, 2021
@karanbirsingh karanbirsingh removed their assignment Nov 3, 2021
@asksep asksep added status: ready for work This issue is ready to be worked on. and removed status: ready for triage This issue is ready to be triaged by the Accessibility Insights team. labels Nov 9, 2021
@DaveTryon DaveTryon added the status: active This issue is currently being worked on by someone. label Feb 8, 2022
@ghost ghost removed the status: ready for work This issue is ready to be worked on. label Feb 8, 2022
@DaveTryon DaveTryon self-assigned this Feb 8, 2022
@brocktaylor7 brocktaylor7 added status: resolved This issue has been merged into main and deployed to canary. and removed status: active This issue is currently being worked on by someone. labels Feb 25, 2022
@DaveTryon DaveTryon removed their assignment Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request status: resolved This issue has been merged into main and deployed to canary.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants