Make Client E2E tests more flexible #150
Conversation
willbeason
force-pushed
the
add-tests
branch
from
003d780
to
f2440f2
Compare
Codecov Report
@@ Coverage Diff @@
## master #150 +/- ##
==========================================
+ Coverage 43.95% 44.60% +0.65%
==========================================
Files 56 54 -2
Lines 3133 2717 -416
==========================================
- Hits 1377 1212 -165
+ Misses 1381 1259 -122
+ Partials 375 246 -129
Flags with carried forward coverage won't be shown. Click here to find out more. Continue to review full report at Codecov.
|
willbeason
changed the title
|
GitHub isn't properly calculating the diffs. In all cases I've preserved the original test logic, and have only changed how the results are validated. |
|
I wonder if there's some sort of non-determinism in the test coverage |
Yeah, test coverage is nondeterministic |
| ` | ||
| ) | ||
|
|
||
| var denyAllCases = []struct { |
There was a problem hiding this comment.
Why not have positive cases as well? I can imagine a scenario where a bug is introduced (perhaps in the rego library) that causes all templates be rejected. These tests might still all/mostly pass.
There was a problem hiding this comment.
I agree. This PR is only for refactoring test cases so that adding such tests (which are indeed absolutely necessary) is easier. Having those in this PR would have made it too large.
Signed-off-by: Will Beason <willbeason@google.com>
Refactor tests so that: - it is possible to specify rego and libraries per-test - all tests use their own Client - the Probe type is no longer necessary Signed-off-by: Will Beason <willbeason@google.com>
This makes test output on failure more consistent and clearer. Use per-test Context instead of one shared between all tests. This removes another avenue of test crosstalk and allows us to test things like behavior with cancellable contexts. Make response order more, but not completely, deterministic. We don't need response order to be more deterministic for tests than what I've put here, and as-is will make the output of reviewing objects more consistent. There are common edge cases where ordering is randomized, but this can be fully fixed later. Comment out the contents of TestRemoteClientE2E. This test is currently skipped until we have a working remote to connect the remote Driver to. I've added clarifying comments to explain what we'll want once this exists. Signed-off-by: Will Beason <willbeason@google.com>
Signed-off-by: Will Beason <willbeason@google.com>
willbeason
force-pushed
the
add-tests
branch
from
7bf0722
to
8e4b375
Compare
| c := &unstructured.Unstructured{} | ||
| c.SetGroupVersionKind(k8schema.GroupVersionKind{ | ||
| Group: "constraints.gatekeeper.sh", | ||
| Version: "v1alpha1", |
There was a problem hiding this comment.
do we want to update apiVersions?
There was a problem hiding this comment.
Yes. There's actually a few changes I want to make related to string constants, but I'd like to have a PR just for that:
Refactor tests so that:
This makes tests able to run hermetically. Additionally, In the future I'll be writing tests that use different Client configurations, which requires using different Clients for different tests (especially handler Rego).
Make response order more, but not completely, deterministic. We don't
need response order to be more deterministic for tests than what I've
put here, and as-is will make the output of reviewing objects more
consistent. There are common edge cases where ordering is still randomized,
but this can be fully fixed later.
Signed-off-by: Will Beason willbeason@google.com