Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pull-kubernetes-kubemark-e2e-gce is failing #60870

Closed
krzyzacy opened this Issue Mar 7, 2018 · 19 comments

Comments

Projects
None yet
8 participants
@krzyzacy
Copy link
Member

commented Mar 7, 2018

http://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking#pull-kubernetes-kubemark-e2e-gce
starting this afternoon

/kind bug
/priority failing-test
/priority critical-urgent
/milestone v1.10

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

/sig testing
/sig scalability

cc @jdumars @jberkus @shyamjvs @wojtek-t

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

hummmmmmm
this might be the issue -

W0306 23:44:59.363] 2018/03/06 23:44:59 e2e.go:77: Calling kubetest -v --check-version-skew=false --test --test_args=--e2e-verify-service-account=false --dump-logs-on-failure=false --gather-resource-usage=true --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/14 --ginkgo.focus=\[Feature:Empty\]...
W0306 23:44:59.711] 2018/03/06 23:44:59 main.go:269: Flag parse failed: invalid argument "--check-version-skew=false" for "-v, --v" flag: strconv.Atoi: parsing "--check-version-skew=false": invalid syntax
W0306 23:44:59.712] 2018/03/06 23:44:59 e2e.go:79: err: exit status 1

but cc @BenTheElder our last kubekins bump was yesterday at kubernetes/test-infra#7145, and everything works fine, includes http://k8s-testgrid.appspot.com/sig-testing-canaries#kubemark-canary

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

ok... kubemark-100-canary was not using :latest kubekins?
/shrug
fix incoming
but it still does not explain why things start to fail ~4pm today :-\

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

... my hypothesis: config updater did not actually updated prow config for kubernetes/test-infra#7145, until kubernetes/test-infra#7162

cc @cjwagner and I'll look at that code again

@jdumars

This comment has been minimized.

Copy link
Member

commented Mar 7, 2018

Thanks @krzyzacy !

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

the test-infra fix is actually for skew tests not for kubemark... I'll need to make the fix in k/k...

and for @Q-Lee lol, see, we have places need to purge :-)

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

I think the root issue is actually because kubemark testing script pulling another latest kubetest if it's older than 24h.. the k/k fix will bring kubemark presubmit back to green, k/t-i fix is not relevant to this job, but will be needed whenever we choose to bump kubekins again.

#60872 (comment) explains it better 💣

k8s-github-robot pushed a commit that referenced this issue Mar 7, 2018

Kubernetes Submit Queue
Merge pull request #60872 from krzyzacy/fix-kubemark
Automatic merge from submit-queue (batch tested with PRs 60872, 60808). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

purge all the -v references from e2e.go

cf for #60870

probably need to work together with kubernetes/test-infra#7164
@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

okay that's healthy now, we'll still need to fix the kubemark script asap

@krzyzacy krzyzacy closed this Mar 7, 2018

@brancz

This comment has been minimized.

Copy link
Member

commented Mar 8, 2018

I believe I'm getting the same failure here (based on and merging into 1.9 branch): #60921

@shyamjvs

This comment has been minimized.

Copy link
Member

commented Mar 8, 2018

The same is being seen for our kubemark-small-last-release job - https://k8s-testgrid.appspot.com/sig-scalability-kubemark#kubemark-small-last-release
Reopening the issue.. @krzyzacy could you please send cherrypick PR for 1.9 (and maybe also 1.8, depending on if we're running kubemark presubmit against it)?

@shyamjvs shyamjvs reopened this Mar 8, 2018

@brancz

This comment has been minimized.

Copy link
Member

commented Mar 8, 2018

Not sure if that's the right procedure, but I already cherry-picked this commit into #60921 to see if that resolves my issue, I'm happy to remove the commit again if that was wrong.

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Mar 8, 2018

Normally we follow https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md for cherry picks, we need to do this for any affected branches.

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 8, 2018

I'll send the cherrypicks shortly

@jberkus

This comment has been minimized.

Copy link

commented Mar 13, 2018

At this point, should this issue be a blocker for 1.10 release? Or is this open just for cherry-picks?

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2018

it's not blocking 1.10

Cherrypicks has been open for older branches, they need to be merged to unblock those branches.

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2018

/milestone v1.9

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.10, v1.9 Mar 13, 2018

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Mar 13, 2018

[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process

@krzyzacy

Issue Labels
  • sig/scalability sig/testing: Issue will be escalated to these SIGs if needed.
  • priority/critical-urgent: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels.
  • kind/bug: Fixes a bug discovered during the current release.
Help
@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2018

/milestone v1.8

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.9, v1.8 Mar 13, 2018

@krzyzacy

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2018

all cherrypicks went in, closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.