Rerun on save not covering other test files

[murolem]

Describe the bug

In case of 2 or more test files, vitest in watch mode (command: vitest) doesn't rerun for all test files, only for the one file which was saved. Because of this, I can't see if any other tests are failing or not.

• On the first run, all the files are tested.
• On rerun triggered by saving in one of the files, only that file will be tested.

Reproduction

Minimal repo: https://github.com/murolem/vitest-issue-not-rerunning-for-all-files

System Info

System:
    OS: Windows 11 10.0.22621
    CPU: (16) x64 AMD Ryzen 7 5800X 8-Core Processor
    Memory: 8.55 GB / 31.93 GB
  Binaries:
    Node: 21.1.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.2.2 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Chromium (120.0.2210.133)
  npmPackages:
    vite: ^5.0.11 => 5.0.11 
    vitest: ^1.2.1 => 1.2.1

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[murolem]

Same goes for checking types using --typecheck option.

[sheremet-va]

I don't understand you issue. Vitest reruns only affected tests on file change, not every test.

[murolem]

I don't understand you issue. Vitest reruns only affected tests on file change, not every test.

Vitest shows that there's only 1 test file on a rerun, which isn't true. If I want to test just that file, I'll specify it when running the command.

[sheremet-va]

From what I can see in the reproduction, this issue is not about running affected tests, but running affected tests and tests that failed before. If I remember correctly, Vitest was running all failing tests plus affected tests before at some point, so this looks like a regression.

What Vitest will not do is run all tests when one test is changed because it doesn't make any sense - the status of all your other tests will not change because test files are isolated.

[murolem]

From what I can see in the reproduction, this issue is not about running affected tests, but running affected tests and tests that failed before. If I remember correctly, Vitest was running all failing tests plus affected tests before at some point, so this looks like a regression.

Yes, thanks, this is kinda what I was trying to convey. Was there a purpose to rerunning non-affected failed tests?

What Vitest will not do is run all tests when one test is changed because it doesn't make any sense - the status of all your other tests will not change because test files are isolated.

Makes sense — the results of the other tests can be reused regardless of their outcome.

[hi-ogawa]

Maybe related to this PR?

• fix(core): test change should trigger failed test rerun #3773

It looks like this PR is introducing "file change -> rerun file change and failed ones"

[ghost]

Is there a flag or is vitest supposed to to always rerun all tests when any test file changes? That's what I've been searching for...

[hi-ogawa]

is vitest supposed to to always rerun all tests when any test file changes?

@waynebloss Vitest is not supposed to do this as explained in #4997 (comment)

This is expected to provide a better experience for most of the users, but if this behavior is not preferable for some reason, then it would help if you can share more details about your use case.

As an exception, for example config files change, Vitest should re-run everything and we have forceRerunTriggers for that purpose https://vitest.dev/config/#forcereruntriggers. Maybe this is what you're looking for?

[ghost]

@hi-ogawa Thank you. I am a new user giving vitest a chance after enjoying the use of the excellent vite.

Mainly, I just want to see all the nice check marks (and totals) all of the time. So I am having to use nodemon to work around this currently.

The problem IMO is that the default console output is not stable after a rerun, so you have to scroll back up to history to see results from previous runs. It's also not stable depending on if there is more than one test file or not. (With a single test file, you get sub-check marks for each test in that file but with multiple test files you get no sub-check marks.)

I know this is not the place for this issue and I haven't completely explored all the reporting options yet so perhaps I'll look at those and also give the UI a try.

[ghost]

So, I solved my own related stability issue by just using --reporter=verbose. I guess what I really want is a verbose-caching reporter so I'll be continuing the search...

[ghost]

@hi-ogawa Here is my final report in response to your query:

• I have tried the UI and it is indeed a stable UI. I don't like that you can't see a whole outline of all test files and their tests at once and just scroll up and down to see details. However, more importantly, I really don't want to open yet another window to work with a given project, especially one that also takes up the terminal pane used to start the UI window. As such, I will continue using nodemon + vitest via vitest run --reporter=verbose for now.
• With regards to using forceRerunTriggers, I suppose that I could try to set it to be the same paths as the include files but when I tried that in my vitest.workspace.ts in my repo root I found out forceRerunTriggers is one of the configs that can't be in the workspace file. Even if it worked, it would be smelly IMO. See [0]
  • Also, what if a file that I change modifies some database e.g. a Redis store or some other external thing that isn't a file? I love the simplicity of just re-running all tests, especially at the beginning of a project when the tests all run very quickly.
  • At later stages of a project, I might turn that flag off when I want to focus on a single test file at a time.

So I think a simple flag to allow always rerunning would be useful.

[0]

[hi-ogawa]

@waynebloss Thanks for providing the details. It looks like you're feeling some UI/UX issues of builtin reporters, so if you want to expand on that, please feel free to open new issue or use new discussion if it's just a quick question https://github.com/vitest-dev/vitest/discussions

As for forceRerunTriggers, I think your "smelly" one indeed doesn't work since this config has to be defined for "root" config. Also I think having forceRerunTriggers same as include is not enough since I suppose you want to rerun everything when you change any source files (not necessary test files).

In your case, probably forceRerunTriggers: ["**/*"] is what you want. I made an example project here. Could you take a look?

https://stackblitz.com/edit/vitest-dev-vitest-xysf8w?file=vitest.config.ts

[ghost]

@hi-ogawa That works very well, thanks again!

I was also able to avoid creating a separate vite.config.ts file in each of my packages by using the extended workspace definition shown in my previous comment, but without the error shown. Originally to that end, I had tried to put the vite-tsconfig-paths plugins definition code into the root vite.config.ts but that did not work so knowing where a config can go remains somewhat confusing to me.

However, I can get rid of nodemon now and we'll see how that goes. Thank you very much for your help.

[sheremet-va]

As a team, we decided that it would be nice to show previously failed test files in the terminal, but we won't show the errors in those files to not clutter the output. The tests themselves will not be running, unless the change will affect the file.