Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1952457: Re-enable crictl node test #26320

Merged

Conversation

saschagrunert
Copy link
Member

The corresponding upstream PR has been merged which should make the test
more robust: kubernetes/kubernetes#101866

This means we can now re-enable the test for the Kubernetes master.

@saschagrunert saschagrunert force-pushed the crictl-on-the-node branch 3 times, most recently from cb3a6e5 to b4f328c Compare July 12, 2021 10:15
@saschagrunert
Copy link
Member Author

/test e2e-metal-ipi-ovn-ipv6

@saschagrunert
Copy link
Member Author

The test is using Kubernetes v1.21.1, so we may have to wait until v1.22.0 has been released as well as the dependency has been updated.

@bparees
Copy link
Contributor

bparees commented Jul 12, 2021

/approve

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 12, 2021
@saschagrunert
Copy link
Member Author

Test still use 1.21.1, waiting a bit more until we run on 1.22 🙃

The corresponding upstream PR has been merged which should make the test
more robust: kubernetes/kubernetes#101866

This means we can now re-enable the test for the Kubernetes `master`.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
@saschagrunert
Copy link
Member Author

/test e2e-agnostic-cmd

@saschagrunert
Copy link
Member Author

/retest

@saschagrunert
Copy link
Member Author

/test e2e-agnostic-cmd

@saschagrunert
Copy link
Member Author

The test gets now automatically skipped, see: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26320/pull-ci-openshift-origin-master-e2e-gcp/1432249603821932544

This is ready for review and merge.

@saschagrunert
Copy link
Member Author

PTAL @smarterclayton

@saschagrunert saschagrunert changed the title Re-enable test for running crictl on a node Bug 1952457: Re-enable test for running crictl on a node Aug 30, 2021
@openshift-ci openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Aug 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 30, 2021

@saschagrunert: This pull request references Bugzilla bug 1952457, which is invalid:

  • expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1952457: Re-enable test for running crictl on a node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@saschagrunert
Copy link
Member Author

/bugzilla refresh

@rphillips
Copy link
Contributor

/retest-required

@saschagrunert saschagrunert changed the title Bug 1952457: Re-enable test for running crictl on a node Bug 1952457: Re-enable crictl node test Aug 30, 2021
@rphillips
Copy link
Contributor

/lgtm

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 30, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, rphillips, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 30, 2021

@saschagrunert: An error was encountered querying GitHub for users with public email (schoudha@redhat.com) for bug 1952457 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message. non-200 OK status code: 403 Forbidden body: "{\n \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@rphillips
Copy link
Contributor

/retest

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@saschagrunert
Copy link
Member Author

saschagrunert commented Aug 31, 2021

e2e-gcp test passed but prow does not report the result back

@saschagrunert
Copy link
Member Author

/override ci/prow/e2e-gcp

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 31, 2021

@saschagrunert: saschagrunert unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

In response to this:

e2e-gcp test passed but prow does not report the result back
/override e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@saschagrunert
Copy link
Member Author

/test e2e-gcp

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 31, 2021

@saschagrunert: saschagrunert unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

In response to this:

/override ci/prow/e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@saschagrunert
Copy link
Member Author

/test e2e-gcp

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@saschagrunert
Copy link
Member Author

/test e2e-gcp

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 1, 2021

@saschagrunert: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic-cmd bf7f113 link /test e2e-agnostic-cmd

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@saschagrunert
Copy link
Member Author

/test e2e-aws-fips

@openshift-merge-robot openshift-merge-robot merged commit eb7ac8a into openshift:master Sep 1, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 1, 2021

@saschagrunert: All pull requests linked via external trackers have merged:

Bugzilla bug 1952457 has been moved to the MODIFIED state.

In response to this:

Bug 1952457: Re-enable crictl node test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@saschagrunert saschagrunert deleted the crictl-on-the-node branch September 1, 2021 12:17
@saschagrunert
Copy link
Member Author

Hm, what's the difference between the CI running here as presubmit and the periodic? Because the test passed (got excluded) on the PR.

@stbenjam
Copy link
Member

stbenjam commented Sep 2, 2021

I can't see much difference, presubmits installs Loki for logging but I don't see what that changes.

presubmits here: https://github.com/openshift/release/blob/master/ci-operator/config/openshift/origin/openshift-origin-master.yaml#L86-L89

CI: https://github.com/openshift/release/blob/master/ci-operator/config/openshift/release/openshift-release-master__ci-4.9.yaml#L182-L186

Looking at the test code I can't figure out why the IP is available in crictl.go ("Ran x on 10.0.0.5:22") but the IP is missing in the SSH method -- unless we're somehow hitting the kube bastion code on the CI builds? But I'm not sure how to tell.

if bastion := os.Getenv("KUBE_SSH_BASTION"); len(bastion) > 0 {
stdout, stderr, code, err := runSSHCommandViaBastion(cmd, result.User, bastion, host, signer)
result.Stdout = stdout
result.Stderr = stderr
result.Code = code
return result, err
}

@saschagrunert
Copy link
Member Author

Usually the test should skip earlier because no SSH'able nodes are found. But the result now assumes that ssh is running via the internal node IPs. Hm, hm… we made the test host-local in kubernetes/kubernetes#103640, but this will land in v1.23.0.

STEP: Getting all nodes' SSH-able IP addresses
Sep  2 10:36:43.638: INFO: No external IP address on nodes, falling back to internal IPs
Sep  2 10:36:43.640: INFO: Assuming SSH on host 10.0.0.5
Sep  2 10:36:43.640: INFO: Assuming SSH on host 10.0.0.3
Sep  2 10:36:43.640: INFO: Assuming SSH on host 10.0.0.4
Sep  2 10:36:46.640: INFO: Skipping host 10.0.128.4 because it does not run anything on port 22
Sep  2 10:36:46.640: INFO: Skipping host 10.0.128.2 because it does not run anything on port 22
Sep  2 10:36:46.640: INFO: Skipping host 10.0.128.3 because it does not run anything on port 22
STEP: SSH'ing to node "10.0.0.5:22" to run "sudo crictl version"
error dialing core@:22: 'dial tcp :22: connect: connection refused', retrying

@stbenjam
Copy link
Member

stbenjam commented Sep 2, 2021

Is there a way to fix it, or can we re-disable the test? I filed https://bugzilla.redhat.com/show_bug.cgi?id=2000589 for this problem if you need a BZ

@saschagrunert
Copy link
Member Author

Yes, let's revert for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants