Support running multiple tests suites at the same time

[DanTup]

Integration tests for extensions can be kinda slow so it'd be great if we could spawn separate instances of Code to run different tests at the same time on CI services. Currently this happens:

Running extension tests from the command line is currently only supported if no other instance of Code is running.

The word currently in that message suggests that this may be a temporary limitation, though I couldn't find an existing issue about relaxing that restriction.

Is it possible?

[joaomoreno]

I believe this limitation is lifted as soon as you use separate user data dirs, but @bpasero should know more.

[bpasero]

Yeah, different user data dirs are a possible workaround.

[DanTup]

Aha! That seems to work, though the vscode/bin/test doesn't allow you to pass it through. I'll see if I can make a PR to handle that. Thanks!

(If that's going to be the official way to support it, it may be worth tweaking this text to say that instead :-))

[DanTup]

I've opened microsoft/vscode-extension-vscode#144 which seems to do the job (in my local testing).

[DanTup]

I tried spawning multiple instances on Travis, and I see failures like this:

Error: Cannot find module 'vscode'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/Users/travis/build/Dart-Code/Dart-Code/out/test/helpers.js:17:12)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)

I'm positive there are no changes that would affect the restoring of the modules (the full diff is here). I tried adding a delay between each test spawn, but no change. I'll try and do more digging, but I thought it worth posting in case it seems familiar to anyone?

[kentonbmax]

I get @danup error above running jasmine. I get issue #70984 when running npm test out of the box.

[DanTup]

I gave up on this and just deal with the run times running them individually. Running VS Code on CI has become pretty unstable even without them overlapping (mostly "Render process crashed!" and "Failed to show someFile in editor undefined") so this just seems like something extra to go wrong. If I can get my setup to run faster, I might just use more shards on Travis/AppVeyor to try and speed things up.

[aaomidi]

Let me know if you've figured out the reason for the Render process crashed error.

[DanTup]

Unfortunately I did not. I gave up on trying to run them concurrently and instead run more entirely separate instances in the CI script (which means merging coverage, etc. but it seems most reliable). I still do see VS Code crash pretty frequently (running only one instance) though.

More recently, macOS runs just seem to hang at the end (see #82676) which means almost every test run is red now anyway 😭

[aaomidi]

Is it only happening to you on windows? (The process crashing)

[DanTup]

I don't remember to be honest - it was some time ago. I already spent more time on it than I'd like to repeat to find out again 😁