Skip to content

Commit

Permalink
chore(actions): dynamic test scheduler / balancer (#12180)
Browse files Browse the repository at this point in the history
This commit adds an automatic scheduler for running busted tests.  It
replaces the static, shell script based scheduler by a mechanism that
distributes the load onto a number of runners.  Each runner gets to
work on a portion of the tests that need to be run.  The scheduler
uses historic run time information to distribute the work evenly
across runners, with the goal of making them all run for the same
amount of time.  With the 7 runners configured in the PR, the overall
time it takes to run tests is reduced from around 30 minutes to around
11 minutes.

Previously, the scheduling for tests was defined by what the
run_tests.sh shell script did.  This has now changed so that the new
JSON file `test_suites.json` is instead used to define the tests that
need to run.  Like before, each of the test suites can have its own
set of environment variables and test exclusions.

The test runner has been rewritten in Javascript in order to make it
easier to interface with the declarative configuration file and to
facilitate reporting and interfacing with busted.  It resides in the
https://github.com/Kong/gateway-test-scheduler repository and
provides its functionality through custom GitHub Actions.

A couple of tests had to be changed to isolate them from other tests
better.  As the tests are no longer run in identical order every time,
it has become more important that each test performs any required
cleanup before it runs.

KAG-3196
  • Loading branch information
hanshuebner committed Dec 15, 2023
1 parent 7f93a92 commit ac59ffd
Show file tree
Hide file tree
Showing 11 changed files with 249 additions and 343 deletions.
154 changes: 0 additions & 154 deletions .ci/run_tests.sh

This file was deleted.

34 changes: 34 additions & 0 deletions .ci/test_suites.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
[
{
"name": "unit",
"exclude_tags": "flaky,ipv6",
"specs": ["spec/01-unit/"]
},
{
"name": "integration",
"exclude_tags": "flaky,ipv6,off",
"environment": {
"KONG_TEST_DATABASE": "postgres"
},
"specs": ["spec/02-integration/"]
},
{
"name": "dbless",
"exclude_tags": "flaky,ipv6,postgres,db",
"specs": [
"spec/02-integration/02-cmd/",
"spec/02-integration/05-proxy/",
"spec/02-integration/04-admin_api/02-kong_routes_spec.lua",
"spec/02-integration/04-admin_api/15-off_spec.lua",
"spec/02-integration/08-status_api/01-core_routes_spec.lua",
"spec/02-integration/08-status_api/03-readiness_endpoint_spec.lua",
"spec/02-integration/11-dbless/",
"spec/02-integration/20-wasm/"
]
},
{
"name": "plugins",
"exclude_tags": "flaky,ipv6",
"specs": ["spec/03-plugins/"]
}
]
Loading

0 comments on commit ac59ffd

Please sign in to comment.