Split test jobs in Travis #3613
Conversation
|
Did few runs on my fork and have following observation
As Apache has a pool of 30 instances would see how much time it takes as part of PR build which uses same pool |
Codecov Report
@@ Coverage Diff @@
## master #3613 +/- ##
==========================================
- Coverage 74.57% 74.49% -0.09%
==========================================
Files 126 126
Lines 5982 5990 +8
Branches 397 390 -7
==========================================
+ Hits 4461 4462 +1
- Misses 1521 1528 +7
Continue to review full report at Codecov.
|
|
Current build timings for this PR. Build time reduces from ~44 mins to ~35 mins. Next would try splitting and optimizing the build steps |
|
Post splitting of
Build Scan
So total 1242 tests. Current master has 1247 tests. So looking into what is missing |
.travis.yml
Outdated
| - ansible-playbook -i ansible/environments/local ansible/logs.yml # collect logs regardless of build exit | ||
| - ./tools/travis/box-upload.py "$TRAVIS_BUILD_DIR/logs" "$TRAVIS_BUILD_ID-$TRAVIS_BRANCH.tar.gz" | ||
| env: DESCRIPTION="Execute test suite." | ||
| - tools/build/checkLogs.py logs db |
There was a problem hiding this comment.
Not consistent ./tools starting with ./
|
This looks awesome @chetanmeh good progress. ^^ @rabbah |
| //The value can be specified either via env variable | ||
| // ORG_GRADLE_PROJECT_testSetName | ||
| //Or via property -PtestSetName | ||
| if (!project.hasProperty("testSetName")) { |
There was a problem hiding this comment.
I think we still need to offer a way to run all tests.
In our downstream we run ./gradlew tests:test to run all tests.
I tried this and it doesn't run all tests, only LEAN
🔥 $ ./gradlew tests:test
> Configure project :tests
Using testSet LEAN - {
"excludes": [
"**/*Swift*",
"**/*Python*",
"**/*Java*",
"**/*ThrottleTests*",
"**/MaxActionDurationTests*",
"**/*ApiGwRestBasicTests*",
"**/*WskCli*",
"**/*Swift*",
"**/*Python*",
"**/*Java*",
"**/*ThrottleTests*",
"**/MaxActionDurationTests*",
"**/*ApiGwRestBasicTests*",
"**/*WskCli*"
]
}
There was a problem hiding this comment.
This was a misleading log which was getting logged due to build task evaluation triggering print message to be logged. Despite this log all tests would have been run.
This has been fixed now
Split the build.sh into separate scripts and invoke selective scripts based on tests to be run
Earlier the log collection was being done in build.sh prior to check. So retain that order
|
Added a new task to just run unit tests which do not require even a running CouchDB server locally. This task runs 823 tests in 2.5 mins!. So developers can run a large portion of test suite locally and can test changes locally for early feedback and faster development |
|
I tested this internally and |
Splits the Travis build to have 2 separate jobs for running tests
Description
This PR splits the travis setup in 2 parts
See discussions in #3572 for details
Implementation Details
This PR introduces a notion of
testSetswhich define set of filters for which test to execute. It defines 3 sets for nowREQUIRE_ONLY_DB- Test which just need database to be upREQUIRE_SYSTEM- Test which require whole OpenWhisk runtime up. Excludes tests under 1LEAN- Include all lean test i.e. current masterChanges in checkLogs.py
checkLogs.pycurrently performs some log sanity check like checking if db, controller logs are non empty. With this PR forREQUIRE_ONLY_DBcontroller logs would not be there. So it has been modified to supporttags. Each check specify set of tags to which it belongs likedbandsystem. And thencheckLogs.pywould specify the tags which it wants to enforce checks forTODO
Related issue and scope
My changes affect the following components
Types of changes
Checklist: