Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

explicitly fail if no images are found when running remote tests #91470

Merged
merged 1 commit into from Jun 2, 2020

Conversation

MHBauer
Copy link
Contributor

@MHBauer MHBauer commented May 26, 2020

The previous implementation succeeds if no images are run. This causes
silent failures when image matchers are provided that do not match any image.

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:
Some of the root cause of this existing in the first place.
#91292
In this issue we can see that in april some images were updated, and tests stopped running at all for 2 of 3 images.

Special notes for your reviewer:
With a list of no images, it turns a bunch of the later for loops into nops, because we for _,_ := range over an empty container.

Ya'll looked at the issue:
/cc @spiffxp @vpickard @bart0sh @dims

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
I think tests should fail if they don't run.

I'm not sure how to test this.

This has potential to cause anything using image-regex, image_regex, image-family, image_family matchers to fail loudly.
/hold

$ rg 'image_regex|image-regex|image-family|image_family' ./

./community/contributors/devel/sig-node/e2e-node-tests.md
163:     image_regex: cos-stable-60-9592-84-0

# I don't think this runs anymore?
./kubernetes/test/e2e_node/jenkins/image-config.yaml
16:    image_regex: cos-stable-59-9460-64-0 # docker 1.11.2
20:    image_regex: cos-stable-60-9592-84-0 # docker 1.13.1

# I don't think this runs anymore?
./kubernetes/test/e2e_node/jenkins/image-config-serial.yaml
16:    image_regex: cos-stable-59-9460-64-0 # docker 1.11.2
24:    image_regex: cos-stable-60-9592-84-0 # docker 1.13.1

./test-infra/jobs/e2e_node/image-config.yaml
9:    image_regex: cos-stable-81-12871-103-0 # docker v19.03.6
13:    image_regex: cos-stable-77-12371-227-0 # docker v19.03.1, deprecated after 2020-12-17

./test-infra/jobs/e2e_node/image-config-serial-cpu-manager.yaml
11:    image_regex: cos-stable-73-11647-510-0 # docker v18.09.7, deprecated after 2020-06-19

./test-infra/jobs/e2e_node/image-config-serial.yaml
9:    image_regex: cos-stable-73-11647-510-0 # docker v18.09.7, deprecated after 2020-06-19
17:    image_regex: cos-stable-77-12371-227-0 # docker v19.03.1, deprecated after 2020-12-17

./test-infra/jobs/e2e_node/containerd/image-config.yaml
3:    image_family: pipeline-2
7:    image_family: cos-77-lts

./test-infra/jobs/e2e_node/containerd/containerd-master/image-config.yaml
7:    image_regex: cos-stable-73-11647-510-0

./test-infra/jobs/e2e_node/containerd/containerd-release-1.3/image-config.yaml
7:    image_regex: cos-stable-73-11647-510-0

./test-infra/jobs/e2e_node/containerd/cri-master/image-config-pr.yaml
7:    image_regex: cos-stable-73-11647-510-0

./test-infra/experiment/bump_e2e_image.sh
24:bazel run //experiment/image-bumper -- --image-regex gcr.io/k8s-testimages/kubekins-e2e "${TREE}/experiment/generate_tests.py" "${TREE}/experiment/test_config.yaml" "${TREE}/config/prow/config.yaml"
25:find "${TREE}/config/jobs/" . -name "*.yaml" | xargs bazel run //experiment/image-bumper -- --image-regex gcr.io/k8s-testimages/kubekins-e2e

./test-infra/jobs/e2e_node/containerd/containerd-release-1.3/#image-config.yaml#
7:    image_family: cos-stable-73-11647-510-0

./test-infra/jobs/e2e_node/containerd/containerd-release-1.2/image-config.yaml
7:    image_regex: cos-stable-73-11647-510-0

./test-infra/jobs/e2e_node/containerd/cri-master/image-config.yaml
7:    image_regex: cos-stable-73-11647-510-0

./test-infra/releng/test_config.yaml
493:    - --image-family=cos-shielded-lts-1
499:    - --image-family=cos-lts-2
505:    - --image-family=pipeline-1
511:    - --image-family=pipeline-2
635:    - --image-family=cos-shielded-lts-1
641:    - --image-family=cos-lts-2
647:    - --image-family=pipeline-1
652:    - --image-family=pipeline-2

./test-infra/config/jobs/kubernetes/sig-node/containerd.yaml
210:      - --image-family=ubuntu-gke-1604-lts
923:      - --image-family=ubuntu-gke-1604-lts
1102:      - --image-family=cos-73-lts
1129:      - --image-family=pipeline-2

./test-infra/config/jobs/kubernetes/generated/generated.yaml
21:      - --image-family=pipeline-1
53:      - --image-family=pipeline-1
85:      - --image-family=pipeline-1
117:      - --image-family=pipeline-1
149:      - --image-family=pipeline-1
181:      - --image-family=pipeline-1
213:      - --image-family=pipeline-1
245:      - --image-family=pipeline-1
277:      - --image-family=pipeline-2
309:      - --image-family=pipeline-2
341:      - --image-family=pipeline-2
373:      - --image-family=pipeline-2
405:      - --image-family=pipeline-2
437:      - --image-family=pipeline-2
469:      - --image-family=pipeline-2
501:      - --image-family=pipeline-2
531:      - --image-family=pipeline-1
564:      - --image-family=pipeline-1
597:      - --image-family=pipeline-1
630:      - --image-family=pipeline-1
663:      - --image-family=pipeline-1
696:      - --image-family=pipeline-1
729:      - --image-family=pipeline-1
762:      - --image-family=pipeline-1
795:      - --image-family=pipeline-1
828:      - --image-family=pipeline-1
861:      - --image-family=pipeline-1
894:      - --image-family=pipeline-1
927:      - --image-family=pipeline-2
960:      - --image-family=pipeline-2
993:      - --image-family=pipeline-2
1026:      - --image-family=pipeline-2
1059:      - --image-family=pipeline-2
1092:      - --image-family=pipeline-2
1125:      - --image-family=pipeline-2
1158:      - --image-family=pipeline-2
1191:      - --image-family=pipeline-2
1224:      - --image-family=pipeline-2
1257:      - --image-family=pipeline-2
1290:      - --image-family=pipeline-2

./test-infra/config/jobs/kubernetes/node-problem-detector/node-problem-detector-ci.yaml
268:      - name: IMAGE_FAMILY

./test-infra/config/jobs/kubernetes/sig-network/ci-e2e-gce-netd.yaml
994:      - --image-family=ubuntu-gke-1604-lts
1022:      - --image-family=ubuntu-gke-1604-lts

./test-infra/config/jobs/kubernetes/sig-storage/sig-storage-gce-config.yaml
230:        - --image-family=ubuntu-gke-1804-lts-1
271:        - --image-family=ubuntu-gke-1804-lts-1
338:      - --image-family=ubuntu-gke-1804-lts-1
363:      - --image-family=ubuntu-gke-1804-lts-1

The previous implementation succeeds if no images are run. This causes
silent failures when image matchers are provided that do not match any image.
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 26, 2020
@MHBauer
Copy link
Contributor Author

MHBauer commented May 26, 2020


W0526 21:43:24.857] F0526 21:43:24.856934    5046 run_remote.go:248] No matching images retrieved on image prefix "cos-stable-77-12371-227-0" and family "": <nil>
W0526 21:43:24.857] goroutine 1 [running]: 

Nice, that's what I expected to see. Looks like I can debug by whack-a-mole if necessary.

@bart0sh
Copy link
Contributor

bart0sh commented May 27, 2020

@MHBauer Thank you for the PR.

2 observations:

  • remaining references to nonexistent images (see the list below) should be fixed and tested before merging this PR. Otherwise it can break some jobs. Here is the list:
$ git grep image_regex
jobs/e2e_node/containerd/containerd-master/image-config.yaml:    image_regex: cos-stable-73-11647-510-0
jobs/e2e_node/containerd/containerd-release-1.2/image-config.yaml:    image_regex: cos-stable-73-11647-510-0
jobs/e2e_node/containerd/containerd-release-1.3/image-config.yaml:    image_regex: cos-stable-73-11647-510-0
jobs/e2e_node/containerd/cri-master/image-config-pr.yaml:    image_regex: cos-stable-73-11647-510-0
jobs/e2e_node/containerd/cri-master/image-config.yaml:    image_regex: cos-stable-73-11647-510-0
jobs/e2e_node/image-config-serial-cpu-manager.yaml:    image_regex: cos-stable-73-11647-510-0 # docker v18.09.7, deprecated after 2020-06-19
jobs/e2e_node/image-config-serial.yaml:    image_regex: cos-stable-73-11647-510-0 # docker v18.09.7, deprecated after 2020-06-19
jobs/e2e_node/image-config-serial.yaml:    image_regex: cos-stable-77-12371-227-0 # docker v19.03.1, deprecated after 2020-12-17
jobs/e2e_node/image-config.yaml:    image_regex: cos-stable-81-12871-103-0 # docker v19.03.6
jobs/e2e_node/image-config.yaml:    image_regex: cos-stable-77-12371-227-0 # docker v19.03.1, deprecated after 2020-12-17
  • even If we relax image_regex to something like 'cos-stable-81-' this PR can cause test failures when GCP stops providing cos-stable-81- images. We need to decide how to avoid that. Alternative solution would be not to fail jobs if regex doesn't match, but inform job maintainers that the test doesn't run anymore.

Thoughts?

@MHBauer
Copy link
Contributor Author

MHBauer commented May 27, 2020

I think failing is informing.

Can we configure test-infra image bumper to look at cos?

@derekwaynecarr
Copy link
Member

i agree failing loudly is preferred.

/retest

@derekwaynecarr
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, MHBauer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 1, 2020
@spiffxp
Copy link
Member

spiffxp commented Jun 1, 2020

/hold cancel
I'm totally fine with this failing to help us root out missing images

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 1, 2020
@@ -244,6 +244,10 @@ func main() {
klog.Fatalf("Could not retrieve list of images based on image prefix %q and family %q: %v",
imageConfig.ImageRegex, imageConfig.ImageFamily, err)
}
if len(images) == 0 { // if we have no images we can't run anything
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be cleaner to put this into the function rather than outside. See #91543

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 5592b5d into kubernetes:master Jun 2, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Jun 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

7 participants