Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

:running: 馃洜 fix prow presubmits #1201

Merged
merged 1 commit into from Nov 25, 2019

Conversation

@alexeldeib
Copy link
Contributor

alexeldeib commented Nov 14, 2019

Fixes prow e2e optional presubmit (maybe?)

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 14, 2019

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 14, 2019

/hold

@alexeldeib alexeldeib changed the title 馃洜 fix test_e2e GOBIN 馃洜 fix prow e2e Nov 14, 2019
Copy link
Contributor

mengqiy left a comment

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Nov 15, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 15, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexeldeib, mengqiy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mengqiy mengqiy changed the title 馃洜 fix prow e2e :running: 馃洜 fix prow e2e Nov 15, 2019
@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 15, 2019

/test pull-kubebuilder-e2e

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 15, 2019

Hmm, seems like they moved the runner in test-infra? Looking at Kind presumbits, they use runner.sh rather than /usr/local/bin/runner. Might need to switch this one more time :/

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 15, 2019

oh, the actual runner is /usr/local/bin/runner.sh -_-

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 15, 2019

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 17, 2019

Copy link
Contributor

camilamacedo86 left a comment

馃憤

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 17, 2019

/retest

1 similar comment
@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 19, 2019

/retest

@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 20, 2019

@alexeldeib Any update?
If you need more time to fix pull-kubebuilder-e2e, I'd suggest merging the fix for pull-kubebuilder-test first (maybe in a separate PR)

@alexeldeib alexeldeib changed the title :running: 馃洜 fix prow e2e :running: 馃洜 fix prow presubmit pt 1 Nov 20, 2019
@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 20, 2019

@mengqiy sorry, I've been at Kubecon since the test-infra PR merged and it seems the test runner requires more adjustments on the invocation side. Happy to merge this as is to fix the vanilla test and finish e2e separately.

If that sounds ok to you, feel free to cancel the hold.

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 20, 2019

Oh, actually I think it's because the files changed to test_e2e_v1.sh and test_e2e_v2.sh. PR incoming...

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 20, 2019

kubernetes/test-infra#15342

If this PR still doesn't pass the e2e after that, I would go ahead and remove hold + merge anyway to get the other fix done.

@mengqiy mengqiy closed this Nov 20, 2019
@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 21, 2019

@mengqiy tested pull-kubebuilder-e2e-1-16-2 prowjob locally with this PR + kubernetes/test-infra#15346 and things seem to be working now.

Apologies again for all the trial and error -- the new test-infra tooling should help avoid this kind of mess in the future.

@alexeldeib alexeldeib changed the title :running: 馃洜 fix prow presubmit pt 1 :running: 馃洜 fix prow presubmits Nov 21, 2019
@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 21, 2019

/retest

@alexeldeib alexeldeib force-pushed the alexeldeib:ace/gobin branch from e164926 to 3412515 Nov 21, 2019
@mengqiy mengqiy closed this Nov 21, 2019
@mengqiy mengqiy reopened this Nov 21, 2019
@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 21, 2019

@alexeldeib It seems we have some problem with kubeadm

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 21, 2019

few additional references for kind using DinD:

kubernetes-sigs/kind#303
kubernetes-sigs/kind#759 --> think prow runner already handles this
kubernetes/test-infra#11510 --> preset for volumes

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 21, 2019

I can't repro this locally, but it seems even after fixing the immediate issue the clusters still fails to come up. Hoping kubernetes/test-infra#15348 will get the last bits fixed or else need to get some additional debug help from Ben w.r.t kind inside a pod using dind.

I don't want to make the CI more and more red with each attempt to fix it...I keep thinking I have it, but it's been a few rounds of the same. Open to direction on how to proceed without making a bigger mess 馃檨

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 23, 2019

/retest

1 similar comment
@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 23, 2019

/retest

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 23, 2019

@alexeldeib: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubebuilder-e2e-v1 a47bed1 link /test pull-kubebuilder-e2e-v1
pull-kubebuilder-e2e 0b747e2 link /test pull-kubebuilder-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 23, 2019

/retest

@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 23, 2019

It is great that this is finally passing!
Thanks for your hard work!
To merge it, can you please drop the change related to debugging and squashed the commits to a reasonable number?

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 23, 2019

Will do, I鈥檓 also relieved this finally works :)

I鈥檓 traveling for personal reasons over the weekend post-Kubecon, but will clean this up for merge Monday if that is okay?

@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 23, 2019

Sure. Let's merge this on Monday.

@alexeldeib alexeldeib force-pushed the alexeldeib:ace/gobin branch from d5a3a9b to 3f961fd Nov 25, 2019
@alexeldeib alexeldeib force-pushed the alexeldeib:ace/gobin branch from 3f961fd to d2c8166 Nov 25, 2019
@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 25, 2019

/hold cancel

Copy link
Contributor

camilamacedo86 left a comment

It shows fine for me 馃憤
/lgtm /approved

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 25, 2019

/test pull-kubebuilder-e2e-k8s-1-15-3

@alexeldeib

This comment has been minimized.

Copy link
Contributor Author

alexeldeib commented Nov 25, 2019

@camilamacedo86 I think the bots didn't like the two commands in one line.


# You can use --image flag to specify the cluster version you want, e.g --image=kindest/node:v1.13.6, the supported version are listed at https://hub.docker.com/r/kindest/node/tags
kind create cluster --config test/kind-config.yaml --image=kindest/node:$K8S_VERSION
kind create cluster -v 4 --retain --wait=1m --config test/kind-config.yaml --image=kindest/node:$K8S_VERSION

This comment has been minimized.

Copy link
@mengqiy

mengqiy Nov 25, 2019

Contributor

-v 4 --retain --wait=1m

I wonder if these are related to debugging?
Re.--wait, do we have to wait for 1 minute? Is it causing any flakes? It seems the default wait time is 0s.

This comment has been minimized.

Copy link
@mengqiy

mengqiy Nov 25, 2019

Contributor

kind cluster is known to be fast to bring up.
Adding 1 minute wait time for bringing up the cluster slow it down ~10%. Look at https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kubebuilder/1201/pull-kubebuilder-e2e-k8s-1-16-2/1198845676235001858, the e2e test only takes <10 minutes.

This comment has been minimized.

Copy link
@alexeldeib

alexeldeib Nov 25, 2019

Author Contributor

I can remove this if desired, but I think it should stay. I matched it to what kind itself does in e2e: https://github.com/kubernetes-sigs/kind/blob/36eae75e067681658d59a31ed679e67e39d9cfc5/hack/ci/e2e-k8s.sh#L88-L93

Without this, kind won't wait at all for the nodes to come up: https://github.com/kubernetes-sigs/kind/blob/6a0de6124de068cd55b24025cf05e8e4ea172641/pkg/internal/cluster/create/actions/waitforready/waitforready.go#L45-L48

If anything, I can reduce -v from 4 to 3 I think.

This comment has been minimized.

Copy link
@alexeldeib

alexeldeib Nov 25, 2019

Author Contributor

It's not adding a static 1 minute wait. It's only checking that the nodes actually come up.

This comment has been minimized.

Copy link
@alexeldeib

alexeldeib Nov 25, 2019

Author Contributor

note that it's also a fairly tight loop so there is not really additional wait time

@mengqiy

This comment has been minimized.

Copy link
Contributor

mengqiy commented Nov 25, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Nov 25, 2019
@k8s-ci-robot k8s-ci-robot merged commit 74f69d6 into kubernetes-sigs:master Nov 25, 2019
7 of 8 checks passed
7 of 8 checks passed
tide Not mergeable. Needs lgtm label.
Details
cla/linuxfoundation alexeldeib authorized
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
netlify/kubebuilder/deploy-preview Deploy preview ready!
Details
pull-kubebuilder-e2e-k8s-1-14-1 Job succeeded.
Details
pull-kubebuilder-e2e-k8s-1-15-3 Job succeeded.
Details
pull-kubebuilder-e2e-k8s-1-16-2 Job succeeded.
Details
pull-kubebuilder-test Job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can鈥檛 perform that action at this time.