Merged
Conversation
This migrates all no-workspace VSCode integration tests to Jest by running `npx jest-codemods`, followed by manual fixes (mostly for Sinon and Proxyquire).
This results in parallel downloads of VSCode, which is not supported.
The timeouts need to be set either on a per-file basis, or per test by using the parameter in `it`. Since we have both Mocha and Jest types, we need to declare in the test file which one we're using.
Instead of calling `fail` and catching an error, this will use the `rejects.toThrow` assertion to check that a Promise rejects with a specific error.
Migrate no-workspace integration tests to Jest
This is a first pass on converting the cli-integration tests to Jest. It uses a custom Jest runner (based on the jest-runner-vscode) to install required extensions. It will also necessary to move some code for installating the CLI to ensure it was possible to install the CLI from outside VSCode. Most of the conversion has been done by `npx jest-codemods`, but all migration of Sinon has been done manually.
Instead of calling `fail`, we can just let the error be caught by Jest, which will automatically fail the tests. For other instances where we're calling `fail` in case an error was not thrown, we will instead use `.rejects.toThrow`.
Jest does not support skipping tests when the test has already started (which could also be in a before hook), so we need to manually return from the tests when the CLI version does not support a tested feature.
Mocha supported `.to.have.length` for objects, but Jest only allows `toHaveLength` on objects which have a length property (such as arrays).
For `to.contain`, `jest-codemods` seems to have converted these to be `expect.arrayContaining`, even for strings. This will make the correct change for strings.
There were some things that were breaking due to version checks. Since we aren't testing on these CLI versions (2.7.2 and 2.7.4 or older) anymore, we can remove these checks and simplify the tests.
This will ensure all mocks are restored after every test. This required a significant amount of changes in the tests since `jest.spyOn` now needs to be called in `beforeEach`, rather than in the `describe` block.
Apparently, we're not importing the same `config` file as is used by the actual extension, so mocking methods in this file does not do anything. However, we can mock the `vscode` module, so we can use this for returning different values in the configuration. We also need to mock the authentication session since we don't have one.
…' into jest-migration/cli-integration
Migrate CLI integration tests to Jest
This removes the Mocha, Sinon and Chai-related packages and removes unused code from the test suite.
This will apply the same change as the PR on `jest-runner-vscode` in the local `node_modules` on every install, ensuring that we have Windows support.
Clean up Mocha, Sinon and Chai
Since we are launching a completely different process for the extension tests than the process that is launched by VSCode, we need to add some special handling for the debugging. This will let the extension host/VSCode expose a debugging port, which VSCode will then connect to. This is "less desirable than letting the bootloader do its thing", but a packaged VSCode application does not allow using the bootloader (`NODE_OPTIONS`=`--require=...`). Therefore, we have to fallback to this option. See: https://github.com/microsoft/vscode-js-debug/blob/47c60558ec31902f42c255abb9b460078df02f9d/src/configuration.ts#L405-L411
Add debugging support for Jest integration tests
Instead of running the integration tests from the `out` directory, this will run the integration tests from the `src` directory using `ts-jest`. Unfortunately, we are not able to use TypeScript files for the `jest-runner-vscode` configuration since `cosmiconfig` (the package that handles the configuration loading for `jest-runner-vscode`) doesn't support loading TypeScript files by default.
Use ts-jest for running integration tests
This adds debugging support to jest-runner for the integration tests when they are run from the `out` directory. Unfortunately, this removes the ability to debug the non-integration tests, such as the pure tests.
Use `patch-package` to add Windows support
charisk
approved these changes
Nov 25, 2022
Add documentation for running a single test
This will patch `jest-runner-vscode` to retry tests. This is a temporary test to see if this will help with the flakiness of the CLI integration tests. The biggest problem with this is that it will launch multiple VSCode instances on every retry: - First try (not a retry): 1 instance - Second try: 2 instances - Third try: 3 instances - etc. I'm not sure why this is happening and can't really narrow it down to a specific cause. Even if I change the `runVSCode` call for the retry by a simple `cp.spawn` call, it still launches multiple instances.
Multiple VSCode instances were being launched when a second instance of VSCode was being spawned with the same user data directory. This is probably because VSCode restores the windows from the previous session, even when `-n`/`--new-window` is passed. This fixes it by patching `jest-runner-vscode` to always create a new temporary user data directory, rather than re-using the same one for all test suites.
This will update the `jest-runner-vscode` patch to retry tests that fail due to no test result being returned from the test runner. This will also add some retries to the `minimal-workspace` and `no-workspace` tests to help with flakiness.
Add retries of CLI integration tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a bundled PR of all work that has been done to migrate the integration tests to Jest. All individual commits should already have been reviewed.
Checklist
ready-for-doc-reviewlabel there.