Improve test time execution tracking for RSpec
Do not allow to use the RSpec tag option together with the RSpec split by test examples feature
Track time spend in RSpec context hook
Update FAQ links in
knapsack_prowarnings and remove FAQ from readme
Use 0 seconds as a default test file time execution instead of 0.1s because Knapsack Pro API already accepts 0 seconds value
Verify test runner adapter bind method is called to track test files time execution and ensure
tmp/knapsack_prodirectory is not removed accidentally
Fix RSpec split by test examples feature broken by lazy generating of JSON report with test example ids
Add support for an attempt to connect to existing Queue on API side to reduce slow requests number
Process.clock_gettimeto measure track execution time
More actionable error message when RSpec split by examples is not working due to RSpec dry-run failure
Add support for env var
KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILEto allow accepting file containing test files to run
Improve logger to show failed requests URL and when retry will happen
Add production branch to non encryptable branches names
Update list of non encryptable branches
When you use Regular Mode then try 6 attempts to connect to the API instead of 3 attempts
KNAPSACK_PRO_MAX_REQUEST_RETRIESenvironment variable to let user define their own number of request retries to the API. It is useful to set it to
0for forked repos when you want to rely on Fallback Mode.
Improve detection of test file path in test-unit runner for test files with shared examples
Allow defining Queue Mode hooks multiple times (
Explicitly call root test runner class to avoid a confusing error when test runner gem is not loaded
KNAPSACK_PRO_RSPEC_TEST_EXAMPLE_DETECTOR_PREFIXto customize prefix for generating test examples report when using RSpec split by test examples
Add support for CI build ID for Github Actions
Migration path for Github Actions users - required
If you use Github Actions and Knapsack Pro Queue Mode then you must set in Github Actions environment variable:
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true. Thanks to that when you retry CI build then tests will run based on previously recorded tests. This solves problem mentioned in the PR.
Migration path for other users - just update
knapsack_progem. Nothing to change in your code :)
Support for non-delimited formatting params of RSpec like
Log when next retry request to Knapsack Pro API happens before starting Fallback Mode
Fix for an auto split of slow RSpec test files by test examples when using
parallel_testsgem. Save the JSON reports with unique file names with the CI node index in the name to avoid accidentally overriding the files on the same disk.
Increase request retry timebox from 4s to 8s to not flood Knapsack Pro API with too many requests in a short period of time and to give time for API server to autoscale and add additional machines to serve traffic
When Fallback Mode is disabled with env
KNAPSACK_PRO_FALLBACK_MODE_ENABLED=falsethen retry the request to Knapsack Pro API for 6 times instead of only 3 times.
Here is related info why some users want to disable Fallback Mode.
Automatically detect slow test files for RSpec and split them by test examples when
Add slow test file pattern
KNAPSACK_PRO_SLOW_TEST_FILE_PATTERNto define RSpec slow test files that should be split by test examples
Start sending API token in header
KNAPSACK-PRO-TEST-SUITE-TOKENinstead of a key
test_suite_tokenin JSON payload.
Raise an error when running Cucumber in Queue Mode and Cucumber system process doesn't finish execution correctly (for instance Cucumber process was killed by CI server due to lack of memory)
Fix bug in RSpec split by test examples in < RSpec 3.6.0 (related to custom json formatter)
Add support for tests split by test examples to RSpec older than 3.6.0
RSpec split test files by test examples (by individual
Note: See PR for more details. This is an experimental feature and it may not work for very large test suite.
How to use it: In order to split RSpec test files by test examples across parallel CI nodes you need to set flag:
Thanks to that your CI build speed can be faster. We recommend using this feature with Queue Mode to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.
Doing tests split by test examples can generate a lot of logs by
knapsack_progem in Queue Mode. We recommend to set log level to:
KnapsackPro::Hooks::Queue.before_queuehook has block of code that raises an exception then ensure the hook was called only once.
Pass non zero exit status from Cucumber as exit status for Cucumber executed in Queue Mode
IMPORTANT Do not allow Fallback Mode when the CI node was retried to avoid running the wrong set of tests
Please read the PR description if you are using retry failed CI node feature on your CI (for instance you use Buildkite).
Increase delay between request retry to Knapsack Pro API from 2s to 4s for 2nd request and from 4s to 8s for 3rd request
KNAPSACK_PRO_CUCUMBER_QUEUE_PREFIXto allow run Cucumber with spring gem in Queue Mode
Allow to use Queue Mode for old RSpec versions that don't have
Add test runner name to
KNAPSACK-PRO-CLIENT-NAMEheader send to Knapsack Pro API
Add support for Codefresh.io CI provider
Add support for GitHub Actions
Add support for job index and job count for parallelism in Semaphore 2.0
CI_PIPELINE_IDas build ID for GitLab CI because it is unique across parallel jobs
Load GitLab CI first to avoid edge case with order of loading envs for
Add Queue Mode for Cucumber
Add support for
KNAPSACK_PRO_TEST_FILE_LISTenvironment variable to run explicitly listed tests
Fix log info when measured time of tests was lost
Logs error on lost info about recorded timing for test files due to missing json files in Queue Mode
Fix bug: default test file time should not be added to measured time
Reduce data transfer and speed up usage of Knapsack Pro API for Queue Mode
Run Fallback Mode when
OpenSSL::SSL::SSLErrorcertificate verify failed for API
KNAPSACK_PRO_LOG_DIRto set directory where to write logs
Retry request 3 times when API returns 5xx HTTP status
Add support for Semaphore CI 2.0
Use .test domain for development mode
- Add metadata to the gemspec
Run Fallback Mode for exception
Add support for GitLab CI env vars CI_NODE_TOTAL and CI_NODE_INDEX.
Add test file exclude pattern.
Track time execution of all tests assigned to CI node in Queue Mode even when they did not run due syntax error or being pending/empty in test run.
Fix bug with not being able to set log level via logger wrapper.
Release 1.0.0 is backward compatible with all previous releases.
Run tests in Fallback Mode when API response is 5xx
Add support for Solano CI and AppVeyor
Add support for Cirrus CI
Remove recursion in Queue Mode
Switch to fallback mode when failed to open TCP connection to API
Fix to record proper time for around(:each) in RSpec
Add Queue Mode for Minitest
Add support for Heroku CI environment variables.
Add support for Cucumber 3.
Add support for test-unit test runner.
Update RSpec timing adapter to be more resilient.
Add support for Codeship environment variables.
Show short warning for not executed test files on CI node. Show explanation in debug logs.
Fallback mode for Queue Mode when Knapsack Pro API doesn't work.
Add in Queue Mode the RSpec summary with info about examples, failures and pending tests.
Fix not working message
Global time execution for testsat end of each subset of tests from work queue.
Autoload knapsack_pro rake tasks with Rails Railties.
Add before and after queue hooks
Add ability to set test_dir using an environment variable.
Extract correct test directory from test file pattern that has multiple patterns.
Clear RSpec examples without shared examples in similar way as in RSpec 3.6.0
Add after subset queue hook and example how to use JUnit formatter in Queue Mode.
Replace rake task installer
knapsack_pro:installwith online installation guide. Remove
Remove timecop gem from required dependencies list.
Add support for Gitlab CI.
More info about Buildkite in installer.
More info about CircleCI in installer.
Add another explanation why test files could not be executed on CI node in Queue Mode.
Show better explanation what to do when there is missing test suite token environment variable.
Show messages about not executed test files as warnings in logs.
Handle case when start timer was not called (rspec-retry issue).
RSpecQueueProfileFormatterExtensionto show profile summary only once at the very end of RSpec Queue Mode output.
- Fix command visible at the end of RSpec Queue Mode output to be able retry test files with spaces in name.
- Fix command visible at the end of RSpec Queue Mode output to be able retry test files without RSpecQueueSummaryFormatter which is dedicated only for Queue Mode.
Add RSpec Queue Formatter to hide duplicated pending and failed tests in Queue Mode
You can keep duplicated pending/failed summary with flag
KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS. More can be found in read me.
Add encryption for branch names
Add supported for log levels
Move all messages related to requests to Knapsack Pro API in log
debuglevel and keep
infolevel only for important messages like how to retry tests in development or info why something works this way or the other (for instance why tests were not executed on the CI node).
- Update license to MIT.
- Add info about Jenkins to installer.
- Extend info about final step in installer about verification if first test suite run was recorded correctly.
Add support for test files in directory with spaces.
- Show at the end of
knapsack_pro:queue:rspeccommand the example how to run all tests executed for the CI node in the development environment.
- Show for each intermediate request to Knapsack Pro API queue how to run a subset of tests fetched from API.
- Save build subset to API even when no test files were executed on CI node. Add warnings to notify why the test files were not executed on CI node in particular mode: regular or queue mode.
- Add info how to allow FakeWeb to connect with Knapsack Pro API in install rake task.
Queue mode retry (remember queue split on retry CI node).
Fix bug in queue mode with recording test files time execution data. Previously the same test files time execution data where multiple times send to Knapsack Pro API.
- Send client name and version in headers for each request to Knapsack Pro API.
- Add info about Queue Mode to install rake task.
- Add more info how to set up VCR and webmock to
- Improve VCR config documentation so it's more clear that ignore_hosts takes arguments instead of array
- Wait a few seconds before retrying failed request to API. With each retry wait a bit longer. Retry at most 5 times.
- Change timeout to 30s for requests to API.
- Retry failed request to API at most 3 times.
Add support for knapsack_pro queue mode
- Enable fallback mode for SocketError when failed to open TCP connection to http or https API endpoint.
Add KNAPSACK_PRO_LOG_LEVEL option
Related PR: https://github.com/ArturT/knapsack/pull/49
- Cache API response test file paths to fix problem with double request to get test suite distribution for the node.
Fix support for turnip >= 2.x
Related PR: https://github.com/ArturT/knapsack/pull/47
Handle case when API returns no test files to execute on the node.
- Use rake invoke for rspec and cucumber tasks.
- Add installer to get started with the knapsack_pro gem.
Add support for Minitest::SharedExamples
Related PR: https://github.com/ArturT/knapsack/pull/46
- Add test file names encryption
Add new environment variable
KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT. The default value is true.
It means when you run test suite again for the same commit hash and total number of nodes and for the same branch then you will get exactly the same test suite split. This is the new default behavior for the knapsack_pro gem. Thanks to that when tests on one of your node failed you can retry the node with exactly the same subset of tests that were run on the node in the first place.
There is one edge case. When you run tests for the first time and there is no data collected about time execution of your tests then we need to collect data to prepare the first test suite split. The second run of your tests will have fixed test suite split. To compare if all your CI nodes are running based on the same test suite split seed you can check the value for seed in knapsack logging message before your test starts. The message looks like:
[knapsack_pro] Test suite split seed: 8a606431-02a1-4766-9878-0ea42a07ad21
- Show test suite split seed in logger based on
build_distribution_idfrom Knapsack Pro API.
fixed_test_suite_splitparam to build distribution Knapsack Pro API endpoint.
- Show test suite split seed in logger based on
Add https support for Knapsack Pro API endpoint
Add Spinach support
Preserve cucumber latest error message with exit code to fix problem with false positive cucumber failed tests
Don't fail when there are no tests to run on a node
Add support for older cucumber versions than 1.3
Changed rake task in minitest_runner.rb to have no warnings output
- Add support for Cucumber 2
- Remove active support dependency so knapsack_pro gem can be used with rails 2.
- Add support for snap-ci.com
- Remove keyword arguments in order to add support for old ruby versions.
- TestFileFinder should find unique files without duplicates when using test file pattern supporting symlinks
- Update test file pattern to support symlinks in specs and readme examples
- Backwards compatibility with knapsack gem old rspec adapter name
Change file path patterns to support 1-level symlinks by default
Fix Travis CI environment variables support
Make knapsack_pro backwards compatible with earlier version of minitest
Related PR from knapsack gem repository: https://github.com/ArturT/knapsack/pull/26
First working release on rubygems.org.