Run unit tests inside browsers #90063

jrieken · 2020-02-05T10:34:37Z

With #90010 we have the ability to run unit tests inside browsers, using playwright. To run them use this command

> node test/browser/index.js

Use the --run <glob>-option to narrow select tests by file path and and use the --browser <webkit|chromium|firefox>-option to select the target runtime (note that firefox currently doesn't work, microsoft/playwright#842).

The plan is to make this part of our build but first we need to fix some test failures that seems to only fail in (some) browser types:

These test fail running inside browsers (chromium or webkit, node test/browser/index.js --browser chromium) @isidorn

  1) Debug - ANSI Handling Expected single sequence operation - /CHROMIUM/:
  2) Debug - ANSI Handling Expected single 8-bit color sequence operation - /CHROMIUM/:
  3) Debug - ANSI Handling Expected multiple sequence operation - /CHROMIUM/:

These test fail only when running node test/browser/index.js --browser webkit @alexdima

  1) MinimapCharRenderer letter d @ 2x - /WEBKIT/:
  2) Decoration Render Options css properties - /WEBKIT/:
  3) Decoration Render Options css properties, gutterIconPaths - /WEBKIT/:
  4) chunk based search #45770. FindInNode should not cross node boundary. - /WEBKIT/:
  5) FindController query options persistence wholeWord - /WEBKIT/:
  6) FindModel incremental find from beginning of file - /WEBKIT/:
  7) FindModel find model next - /WEBKIT/:
  8) FindModel find model next stays in scope - /WEBKIT/:
  9) FindModel find model prev - /WEBKIT/:
  10) FindModel find model prev stays in scope - /WEBKIT/:
  11) FindModel find model next/prev respects cursor position - /WEBKIT/:
  12) FindModel replace hello - /WEBKIT/:
  13) FindModel replaceAll hello - /WEBKIT/:
  14) FindModel listens to model content changes - /WEBKIT/:
  15) FindModel selectAllMatches - /WEBKIT/:
  16) FindModel issue #14143 selectAllMatches should maintain primary cursor if feasible - /WEBKIT/:
  17) FindModel issue #18711 replaceAll with empty string - /WEBKIT/:
  18) FindModel issue #27083. search scope works even if it is a single line - /WEBKIT/:
  19) Multicursor selection AddSelectionToNextFindMatchAction starting with single collapsed selection - /WEBKIT/:
  20) Multicursor selection Find state disassociation enters mode - /WEBKIT/:
  21) Multicursor selection Find state disassociation leaves mode when selection changes - /WEBKIT/:
  22) Multicursor selection Find state disassociation Select Highlights respects mode  - /WEBKIT/:

The text was updated successfully, but these errors were encountered:

isidorn · 2020-02-05T10:54:55Z

Those ansii tests were written as part of a PR. I do not have a good idea why they fail inside Chromium and work when run in VS Code. I tried debugging them with --debug but failed.
To unblock you I can move the tests under electron-browser.

jrieken · 2020-02-05T11:01:04Z

@isidorn this is all new but with --debug --browser chromium you should see a browser in which you should be able to use devtools

isidorn · 2020-02-05T11:08:57Z

@jrieken yes I tried that and I do see chromium, but none of the tests get rendered and the window closes before I can do anything (put a breakpoint).

jrieken · 2020-02-05T11:12:45Z

Yes. In the renderer the tests run without mocha UX. What you can try is this: start with --debug, open dev tools, select "pause on script exec", wait for the pause and set breakpoints that you want/need

#90063

isidorn · 2020-02-05T11:25:09Z

Yeah that works, just not very convenient as you have to be fast. Thanks!

jrieken · 2020-02-05T14:47:50Z

FYI - you can now debug tests by simply opening the test/browser/renderer.html page in a browser. Use a uri like this, notice the m-query arguments (this is AMD module ids):

file:///Users/jrieken/Code/vscode/test/browser/renderer.html?m=vs/editor/test/browser/services/decorationRenderOptions.test&m=vs/editor/test/browser/services/decorationRenderOptions.test

In the presence of test failures, the runner will also make an attempt to print the right link for you. 👇

…s afterwards on Safari (#90063)

alexdima · 2020-02-05T15:42:42Z

Actually it appears that there are still failures on Windows:

 1) Paths (Node Implementation) dirname - /CHROMIUM/:
     ReferenceError: __filename is not defined
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/path.test.js:146:60)

  2) Paths (Node Implementation) extname - /CHROMIUM/:
     ReferenceError: __filename is not defined
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/path.test.js:219:18)

  3) Paths (Node Implementation) resolve - /CHROMIUM/:

      AssertionError:
path.win32.resolve(".")
  expect="/"
  actual="\\"
      + expected - actual

      -1
      +0

      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/path.test.js:369:20)

  4) Paths (Node Implementation) basename - /CHROMIUM/:
     ReferenceError: __filename is not defined
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/path.test.js:382:46)

  5) Resources relativePath - /CHROMIUM/:

      AssertionError: from file://server/share/some/ to file://server/share2/some/path
      + expected - actual

      -/share2/some/path
      +../../share2/some/path

      at assertRelativePath (file:///C:/Alex/src/vscode/out/vs/base/test/common/resources.test.js:210:20)
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/resources.test.js:245:17)

  6) Debug - ANSI Handling Expected single sequence operation - /CHROMIUM/:

      AssertionError: Custom foreground class not found on element after foreground ANSI code #30.
      + expected - actual

      -false
      +true

      at file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:135:21
      at assertSingleSequenceElement (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:83:13)
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:134:17)

  7) Debug - ANSI Handling Expected single 8-bit color sequence operation - /CHROMIUM/:

      AssertionError: Custom color class not found after foreground 8-bit color code 38;5;0
      + expected - actual

      -false
      +true

      at file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:193:21
      at assertSingleSequenceElement (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:83:13)
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:192:17)

  8) Debug - ANSI Handling Expected multiple sequence operation - /CHROMIUM/:

      AssertionError: Foreground color class not found after multiple different ANSI codes.
      + expected - actual

      -false
      +true

      at file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:288:17
      at assertSingleSequenceElement (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:83:13)
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/workbench/contrib/debug/test/browser/debugANSIHandling.test.js:284:13)

jrieken · 2020-02-05T15:46:11Z

The tests for path have been moved to the node-layer. @alexdima I believe that you are missing a fresh compile

alexdima · 2020-02-05T15:51:37Z

@jrieken You're right. But after restarting gulp watch I see one test failure left:

  1) Resources relativePath - /CHROMIUM/:

      AssertionError: from file://server/share/some/ to file://server/share2/some/path
      + expected - actual

      -/share2/some/path
      +../../share2/some/path

      at assertRelativePath (file:///C:/Alex/src/vscode/out/vs/base/test/common/resources.test.js:210:20)
      at Context.<anonymous> (file:///C:/Alex/src/vscode/out/vs/base/test/common/resources.test.js:245:17)

jrieken · 2020-02-06T11:28:40Z

@aeschli can you take a look at those ☝️ test failures

aeschli · 2020-02-06T12:48:40Z

Running node test/browser/index.js --browser webkit on Ubuntu with Chromium I only get the following test failure:
(HEAD ea8d1b7)

1) MinimapCharRenderer letter d @ 2x - /WEBKIT/:

      [46,46,46,255,173,173,173,255,198,198,198,255,202,202,202,255,193,193,193,255,204,204,204,255,0,0,0,255,0,0,0,255] deepEqual [46,46,46,255,173,173,173,255,198,198,198,255,200,200,200,255,193,193,193,255,204,204,204,255,0,0,0,255,0,0,0,255]
      + expected - actual

         198
         198
         198
         255
      -  202
      -  202
      -  202
      +  200
      +  200
      +  200
         255
         193
         193
         193

jrieken · 2020-02-06T12:51:25Z

@aeschli yes - this is Chromium and windows

aeschli · 2020-02-06T16:07:57Z

I fixed the relativePath test failure

jrieken · 2020-02-07T10:54:45Z

This PR adds browser-unit-tests to our continuous build: #90153. There are still these failures

looks like linux fails to run webkit tests (see https://dev.azure.com/vscode/VSCode/_build/results?buildId=36640&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=db522a89-84be-55c1-d087-ef4dde315b47)
@alexdima windows with webkit seems to have decorations-test failure, https://dev.azure.com/vscode/VSCode/_build/results?buildId=36640&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=4f94dfce-af9e-52fd-5eb8-ca3006505634
@alexdima Mac with webkit seems that have minimap-test failure, https://dev.azure.com/vscode/VSCode/_build/results?buildId=36640&view=logs&j=a5e52b91-c83f-5429-4a68-c246fc63a4f7&t=855ed636-0983-5316-9b28-7efa8f10a65b

bpasero · 2020-02-07T11:16:33Z

@jrieken maybe something to consider: we support yarn smoketest --web --browser <type> for smoke tests. We could add a script like that for unit tests as well?

jrieken · 2020-02-07T11:23:50Z

With the PR above there is yarn test-browser

alexdima · 2020-02-07T11:45:19Z

@jrieken Could we make this one compatible with the other one, i.e.

node test\unit\browser\index.js --browser webkit --run src/vs/editor/test/browser/services/decorationRenderOptions.test.ts

does not work while it works in the other one, e.g.

scripts\test.bat --run src/vs/editor/test/browser/services/decorationRenderOptions.test.ts

This is superhelpful for me because I have a copy relative path plugin that I use on files and then I just paste the path

jrieken · 2020-02-07T13:58:34Z

The --run argument is now supported as in other tests, there is now --run for workspace relative paths (with TS → JS trickery) and there is --glob for patterns

jrieken · 2020-02-07T15:18:25Z

All tests green, PR merged, done

jrieken assigned rebornix, isidorn, alexdima and jrieken Feb 5, 2020

jrieken added the engineering VS Code - Build / issue tracking / etc. label Feb 5, 2020

isidorn added a commit that referenced this issue Feb 5, 2020

debug: move ansi handling to electron/browser

51bccba

#90063

alexdima added a commit that referenced this issue Feb 5, 2020

Cannot use RegExp.lastIndex to -1 because it will not find any matche…

ed6fae9

…s afterwards on Safari (#90063)

jrieken added this to the February 2020 milestone Feb 5, 2020

alexdima closed this as completed in 6cf823c Feb 5, 2020

alexdima reopened this Feb 5, 2020

jrieken assigned aeschli Feb 6, 2020

jrieken unassigned isidorn and alexdima Feb 6, 2020

aeschli added a commit that referenced this issue Feb 6, 2020

resource.relativePath fix for windows (for #90063)

a94e5d9

jrieken assigned alexdima and unassigned aeschli Feb 7, 2020

alexdima added a commit that referenced this issue Feb 7, 2020

Fix test on webkit + windows (#90063)

cde8f2a

alexdima added a commit that referenced this issue Feb 7, 2020

Fix tests on webkit by using custom clamping to uint8 (#90063)

5133f11

jrieken added a commit that referenced this issue Feb 7, 2020

support --run argument as before, #90063

03fb0b1

jrieken unassigned rebornix and alexdima Feb 7, 2020

jrieken closed this as completed Feb 7, 2020

vscodebot bot locked and limited conversation to collaborators Mar 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run unit tests inside browsers #90063

Run unit tests inside browsers #90063

jrieken commented Feb 5, 2020 •

edited

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020 •

edited

alexdima commented Feb 5, 2020

jrieken commented Feb 5, 2020

alexdima commented Feb 5, 2020

jrieken commented Feb 6, 2020

aeschli commented Feb 6, 2020

jrieken commented Feb 6, 2020

aeschli commented Feb 6, 2020

jrieken commented Feb 7, 2020

bpasero commented Feb 7, 2020

jrieken commented Feb 7, 2020

alexdima commented Feb 7, 2020

jrieken commented Feb 7, 2020 •

edited

jrieken commented Feb 7, 2020

Run unit tests inside browsers #90063

Run unit tests inside browsers #90063

Comments

jrieken commented Feb 5, 2020 • edited

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020

isidorn commented Feb 5, 2020

jrieken commented Feb 5, 2020 • edited

alexdima commented Feb 5, 2020

jrieken commented Feb 5, 2020

alexdima commented Feb 5, 2020

jrieken commented Feb 6, 2020

aeschli commented Feb 6, 2020

jrieken commented Feb 6, 2020

aeschli commented Feb 6, 2020

jrieken commented Feb 7, 2020

bpasero commented Feb 7, 2020

jrieken commented Feb 7, 2020

alexdima commented Feb 7, 2020

jrieken commented Feb 7, 2020 • edited

jrieken commented Feb 7, 2020

jrieken commented Feb 5, 2020 •

edited

jrieken commented Feb 5, 2020 •

edited

jrieken commented Feb 7, 2020 •

edited