Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flake fix: validate each expression once in primary CEL correctness suite #109238

Merged

Conversation

jpbetz
Copy link
Contributor

@jpbetz jpbetz commented Apr 1, 2022

What type of PR is this?

/kind cleanup
/kind flake

What this PR does / why we need it:

The test suite is compiling and validating each of the many rules it tests three times. The 2nd two times are just to do a sanity check of the cost system, which isn't particularly valuable to test here given all the other tests that cover that.

I've reduced this to a single compile/validate call which reduces the CPU cost of the test by more than half.

Which issue(s) this PR fixes:

Fixes #109225

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/flake Categorizes issue or PR as related to a flaky test. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 1, 2022
@jpbetz
Copy link
Contributor Author

jpbetz commented Apr 1, 2022

/priority critical-urgent
/triage accepted
/assign @cici37

@k8s-ci-robot k8s-ci-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 1, 2022
@jpbetz jpbetz changed the title Only validate each expression once in primary CEL correctness suite Flake fix: validate each expression once in primary CEL correctness suite Apr 1, 2022
@cici37
Copy link
Contributor

cici37 commented Apr 1, 2022

We have test on expression cost but miss the test for hitting limit. Maybe worth to add a separate small test aside for limit check?

@cici37
Copy link
Contributor

cici37 commented Apr 1, 2022

/lgtm
The separate test will be added in following up pr.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 1, 2022
@jpbetz
Copy link
Contributor Author

jpbetz commented Apr 1, 2022

/assign @liggitt
Would you be willing to approve? This is a pretty simple ~3x speed boost to this test suite.

@cici37
Copy link
Contributor

cici37 commented Apr 1, 2022

FYI: We have existing test covering runtime cost budget exceed in k8s.io/apiextensions-apiserver/pkg/apiserver/schema/defaulting/validation_test.go so removing the similar test from cel/validation_test should not be a big concern

@liggitt
Copy link
Member

liggitt commented Apr 1, 2022

/lgtm
/approve

#109239 is likely to resolve the tragically high integration test flake rate

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 1, 2022
@liggitt
Copy link
Member

liggitt commented Apr 1, 2022

We have existing test covering runtime cost budget exceed in k8s.io/apiextensions-apiserver/pkg/apiserver/schema/defaulting/validation_test.go so removing the similar test from cel/validation_test should not be a big concern

mmm... that's specific to defaulting, though, right? that doesn't seem like the same coverage this is dropping

We have test on expression cost but miss the test for hitting limit. Maybe worth to add a separate small test aside for limit check?

that's a good point. a single test of the limit (rather than exercising N times on N expressions) seems important
/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2022
@liggitt
Copy link
Member

liggitt commented Apr 1, 2022

/milestone v1.24
test-only change, improving test signal before test freeze
/retest

@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Apr 1, 2022
@liggitt
Copy link
Member

liggitt commented Apr 1, 2022

that's a good point. a single test of the limit (rather than exercising N times on N expressions) seems important

(if the flake is bad, feel free to unhold and re-add that solo test next week)

@cici37
Copy link
Contributor

cici37 commented Apr 1, 2022

@liggitt I have the PR open to separating the test here: #109242

@jpbetz
Copy link
Contributor Author

jpbetz commented Apr 1, 2022

/hold cancel
We'll merge @cici37's #109242 PR shortly and merge this PR now since this flake is significant.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2022
@k8s-ci-robot k8s-ci-robot merged commit 0184d77 into kubernetes:master Apr 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TestValidationExpressions timeout 10m
4 participants