Run subtests #10862
Merged
Run subtests #10862
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hmlanigan
requested changes
Nov 5, 2019
| echo "" | ||
| echo " $(green 'cmd -h') Display this help message" | ||
| echo " $(green 'cmd -v') Verbose and debug messages" | ||
| echo " $(green 'cmd -V') Very verbose and debug messages" | ||
| echo " $(green 'cmd -s') Skip tests using a comma seperated list" | ||
| echo " $(green 'cmd -a') Create an atifact file" | ||
| echo " $(green 'cmd -x') Output file from streaming the output" | ||
| echo " $(green 'cmd -r') Reuse bootstrapped controller" | ||
| echo " $(green 'cmd -r') Reuse bootstrapped controller between testing suites" |
| To run the tests, they can be broken down into steps: | ||
|
|
||
| ```sh | ||
| ./main.sh deploy # Runs deploy test suite |
There was a problem hiding this comment.
Is it "cd tests && ./main.sh deploy"? Maybe it's just me, but if I read this, I'll never remember the "cd tests &&" part. And tests will fail due to pathing issues.
| } | ||
| ``` | ||
|
|
||
| ## Exit codes / Success |
Every person seems to hit the same issue, in that they attempt to run a subtest directly. This wasn't originally the main idea, instead they where meant to run the test that runs the subtests and then see if they broke anything whilst doing that. I've been persuaded otherwise, a subtest should be able be run as you want to iterate hard on that and instead of blocking people from doing fast iterations we should enable it. ---- Improved the documentation around the test setup.
Add the change directory before running main.sh
When reusing a local controller, ensure that we update the bootstrap provider.
Remove the ability to run all tests suites as default. This prevents mistakes when attempting to run the test suites and causes the bootstrap of juju in substraits that can be expensive. To prevent the feature from being removed completely, we've enabled -A, which allows the developer to say, they know what they're doing.
We encounter issues where we depend on the order of the keys in an object to be sort and sane. This does this.
This allows the test to handle localhost as lxd
SimonRichardson
force-pushed
the
run_subtests
branch
from
ac4ad33
to
6e63b5b
Compare
hmlanigan
approved these changes
Nov 6, 2019
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Every person seems to hit the same issue, in that they attempt to run
a subtest directly. This wasn't originally the main idea, instead they
where meant to run the test that runs the subtests and then see if they
broke anything whilst doing that.
I've been persuaded otherwise, a subtest should be able be run as you
want to iterate hard on that and instead of blocking people from doing
fast iterations we should enable it.
QA steps