Bug some of the tests in the report are not being executed.

[dingn1]

we have "spec/requests/api/artists_spec.rb": 5.043759346008301, in the report but its not actually executed in our CI. I was wondering is there a way to check what's the files being executed?(is it only possible to look through the output in the CI pipeline) And we do find that the problem is fixed after we regenerate the report again so not sure what's wrong(the line in the report was updated). By the way we are using minitest.

[dingn1]

There is 400 lines in our report and the test being executed is only 329(405 executed, 329 actual all others seems to be duplicate as from the output). So I was wondering if there is a way to figure out which one are skipped automatically.

[dingn1]

As I can see from the code is it because of the sorting of how the reported test are read from the report? maybe we need a sort there also?

[ArturT]

As I can see from the code is it because of the sorting of how the reported test are read from the report? maybe we need a sort there also?

Could you provide link to the part of the code you mean?

I was wondering is there a way to check what's the files being executed?(is it only possible to look through the output in the CI pipeline)

You can run bundle exec rake knapsack:minitest and see knapsack logger output with list of tests that are going to be executed. Leftover tests are those not in knapsack report yet.
https://github.com/ArturT/knapsack/blob/master/lib/knapsack/runners/minitest_runner.rb#L8

There is 400 lines in our report and the test being executed is only 329(405 executed, 329 actual all others seems to be duplicate as from the output). So I was wondering if there is a way to figure out which one are skipped automatically.

If you removed or renamed the file spec/requests/api/artists_spec.rb from your repository then knapsack won't run it because it checks whether the test file found in the knapsack report actually exists.

If test file exists in test directory but not in knapsack report then the test file will be run (it's called leftover test).

If you want to track newly added or renamed test files just try https://knapsackpro.com for a few week or so and compare results with what you have now.

[dingn1]

The file is there and its in the report but its not executed(from the output on CI, no pipeline have it) and from the output I get from the CI, i found that there are some files that are running multiple times across different nodes. The line I talked about is https://github.com/ArturT/knapsack/blob/516aedd5ecf3bd0c900ca00e9da22d139e0f7d1d/lib/knapsack/distributors/leftover_distributor.rb#L5

From what I get the total number of test that got ran is correct but as there is so many duplicate a lot are actually skipped(for the output of file name that get executed 76 duplicates are there).

[ArturT]

1. Do you use custom pattern KNAPSACK_TEST_FILE_PATTERN?
2. Do you see duplicated files in knapsack report json?
3. You could fork gem and try add uniq here report.keys.uniq
   https://github.com/ArturT/knapsack/blob/516aedd5ecf3bd0c900ca00e9da22d139e0f7d1d/lib/knapsack/distributors/leftover_distributor.rb#L5 and see how it works for you.

[dingn1]

We have KNAPSACK_TEST_FILE_PATTERN="spec/**{,/*/**}/*_spec.rb" no duplicate in the report.json file

[dingn1]

Thanks for your help got it fixed.

[ArturT]

Adding report.keys.uniq solved the problem? https://github.com/Vydia/knapsack/commit/566ef35123f8f8839067db27f36ce5435088e22d

Could you provide more context about the investigation you did? Is there a real bug in the gem then I'd like to fix that. If not then what was the root of your problem then?

[dingn1]

some of the configuration for running the test was wrong so however one of the pipeline is not executed on the CI. Its not a real bug.

[ArturT]

OK. Thank you!