-
Notifications
You must be signed in to change notification settings - Fork 82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KEP-0007: Assertions for CLI commands #245
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work :) Some open questions though
keps/0007-command-asserts.md
Outdated
```yaml | ||
apiVersion: kuttl.dev/v1beta1 | ||
type: TestAssert | ||
commands: | ||
- command: ./parse-logs.sh name-of-pod-0 string-to-search-for | ||
namespaced: true | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
```yaml | |
apiVersion: kuttl.dev/v1beta1 | |
type: TestAssert | |
commands: | |
- command: ./parse-logs.sh name-of-pod-0 string-to-search-for | |
namespaced: true | |
``` | |
```yaml | |
apiVersion: kuttl.dev/v1beta1 | |
type: TestAssert | |
commands: | |
- command: ./parse-logs.sh name-of-pod-0 string-to-search-for | |
namespaced: true | |
timeout: 60 | |
delay: 10 |
I would suggest an addition of an "assertDelay". This is currently fixed at 1 second, meaning the assert is checked after a delay of 1 second.
When we add commands, we might want to increase this delay, maybe because of very expensive checks.
An `initialDelay` might also make sense, although adding delays to check is often bad design...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not a fan of delay... If we have one... we should be clear a delay from what... I would rather have the declarative asserts successful as a criteria of when to start commands... time delays are a common source of flaky tests :(
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, we currently have a fixed delay of 1 second between rechecks. I.e. when an assert fails, there is a one-second delay until it is checked again. Might be worth to make that configurable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets have a round of discussions and edits... love that this moving! thank you!
#### User Story 2 | ||
|
||
An operator can be configured to encrypt it's API. To assert that the API endpoints are indeed encrypted we can run a specific `curl` command and check that it returns successfully once the operator has been deployed. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it might be interesting to parse the test logs... I'm not sure how that would be done.. but we have the ability to have background processes which are used for in development operators... you may want to parse the controller output which is in the test stdout.
If not... we should add this to the list of future considerations at the bottom of this kep
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
worth considering "shouldFail" for commands... and lets add it to a goal or non-goal and future considerations if not a current goal.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm generally ok with the KEP - I still think it can be extended quite a bit, but that doesn't need to be done in the first implementation.
I'd like to see a couple of kens questions answered somwhere in the KEP, for example things about log output if a command fails, if the report output changes, and if so how.
By asserting CLI commands, kuttl can support many additional test scenarios that examing the internal state of an operator. Signed-off-by: Jan Schlicht <jan@d2iq.com>
Signed-off-by: Jan Schlicht <jan@d2iq.com>
Signed-off-by: Jan Schlicht <jan@d2iq.com>
Signed-off-by: Jan Schlicht <jan@d2iq.com>
Signed-off-by: Ken Sipe <kensipe@gmail.com>
7ad6fa1
to
ddf3085
Compare
rebased to main |
What this PR does / why we need it:
By asserting CLI commands, kuttl can support many additional test scenarios that examining the internal state of an operator.