Add KNAPSACK_PRO_SLOW_TEST_FILE_THRESHOLD to improve the RSpec split by examples feature with many skipped tests

[ArturT]

Story

TODO: link to the internal story

Related

• fix(RSpec split by examples): Properly determine slow test files when test example execution times and full test file execution time are both known #281

Description

Relate to the RSpec split by examples feature:
https://docs.knapsackpro.com/ruby/split-by-test-examples/

If a user skips some test files, for example, using the RSpec --tag option, then RSpec might still load a file but not run it or only run some of the test examples from it.

If a lot of test files are skipped, then the actual average CI node execution time would be much smaller than the estimated time (Knapsack Pro does not know how many test files are filtered out by --tag). This can lead to not determining enough test files as slow, which examples should split. As a result, CI build is unbalanced.

To solve that problem, we added a new environment variable (KNAPSACK_PRO_SLOW_TEST_FILE_THRESHOLD). You define a threshold (a number of seconds) for slow test files. Any test file above the threshold would be split by test examples.
For example, you expect your CI node to run no longer than 3 minutes, then set KNAPSACK_PRO_SLOW_TEST_FILE_THRESHOLD=180 (180 seconds == 3 minutes)

changes

• Add the KNAPSACK_PRO_SLOW_TEST_FILE_THRESHOLD option
• do not split test files by test cases if you run tests on a single CI node to run Knapsack Pro faster

Checklist reminder

• You added the changes to the UNRELEASED section of the CHANGELOG.md, including the needed bump (ie, patch, minor, major)
• You follow the architecture outlined below for RSpec in Queue Mode, which is a work in progress (feel free to propose changes):
  • Pure: lib/knapsack_pro/pure/queue/rspec_pure.rb contains pure functions that are unit tested.
  • Extension: lib/knapsack_pro/extensions/rspec_extension.rb encapsulates calls to RSpec internals and is integration and e2e tested.
  • Runner: lib/knapsack_pro/runners/queue/rspec_runner.rb invokes the pure code and the extension to produce side effects, which are integration and e2e tested.

[shadre]

Added suggestions