Using optr.queue.ShutDown directly as upstream K8s is fixed now #339

LalatenduMohanty · 2020-03-20T19:04:41Z

Upstream pull request kubernetes/kubernetes#77170. It is available in upstream v1.15

Signed-off-by: Lalatendu Mohanty lmohanty@redhat.com

Upstream pull request kubernetes/kubernetes#77170 Signed-off-by: Lalatendu Mohanty <lmohanty@redhat.com>

wking · 2020-03-24T20:36:36Z

Upstream PR vendored in here with 0672262 (#249).

/lgtm

openshift-bot · 2020-03-24T22:47:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-24T23:00:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T00:57:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T01:10:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T03:07:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T05:16:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T05:29:33Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T07:26:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T09:36:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T11:46:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T11:58:56Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T13:55:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T16:17:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T18:29:04Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T18:42:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T20:39:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T20:52:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T22:48:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-25T23:01:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-26T00:58:47Z

/retest

Please review the full test history for this PR and help us cut down flakes.

abhinavdahiya · 2020-03-26T18:34:36Z

/approve

openshift-ci-robot · 2020-03-26T18:34:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, LalatenduMohanty, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-03-26T20:28:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-26T23:17:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T00:22:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T01:27:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T01:40:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T03:37:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T03:50:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T05:47:31Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T07:58:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T08:10:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T10:33:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T12:43:33Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T14:53:40Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T17:03:38Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-03-27T19:13:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

sdodson · 2020-03-27T19:20:31Z

/hold
@LalatenduMohanty can you please look into why the integration tests started failing on March 20th and haven't passed since? There was one pass on this commit so I don't know if it's this PR or a general failure.

LalatenduMohanty · 2020-03-30T16:16:04Z

@LalatenduMohanty can you please look into why the integration tests started failing on March 20th and haven't passed since? There was one pass on this commit so I don't know if it's this PR or a general failure.

Last time I checked the failure looked like a flake, but this time it is

2020/03/27 19:16:22 Container bin-cli in pod integration completed successfully
2020/03/27 19:16:35 Container bin-hyperkube in pod integration completed successfully
2020/03/27 21:16:18 error: unable to signal to artifacts container to terminate in pod integration, triggering deletion: could not run remote command: unable to upgrade connection: container not found ("artifacts")
2020/03/27 21:16:19 error: unable to retrieve artifacts from pod integration: could not read gzipped artifacts: unable to upgrade connection: container not found ("artifacts")
2020/03/27 21:16:19 error: could not wait for pod 'integration': it is no longer present on the cluster (usually a result of a race or resource pressure. re-running the job should help)
2020/03/27 21:16:19 No custom metadata found and prow metadata already exists. Not updating the metadata.
E0327 21:16:19.074027      13 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".160045c47c0d14b3", GenerateName:"", Namespace:"ci-op-gtgb1drn", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"", Namespace:"ci-op-gtgb1drn", Name:"", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"CiJobFailed", Message:"Running job pull-ci-openshift-cluster-version-operator-master-integration for PR https://github.com/openshift/cluster-version-operator/pull/339 in namespace ci-op-gtgb1drn from author LalatenduMohanty", Source:v1.EventSource{Component:"ci-op-gtgb1drn", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf97b928c1e176b3, ext:7317917608909, loc:(*time.Location)(0x271e5e0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf97b928c1e176b3, ext:7317917608909, loc:(*time.Location)(0x271e5e0)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events ".160045c47c0d14b3" is forbidden: unable to create new content in namespace ci-op-gtgb1drn because it is being terminated' (will not retry!)
2020/03/27 21:16:20 Ran for 2h1m58s
error: could not run steps: step integration failed: template pod "integration" failed: pod was deleted while ci-operator step was waiting for it

looking in to it

LalatenduMohanty · 2020-03-31T14:47:23Z

/retest

LalatenduMohanty · 2020-03-31T17:27:44Z

Another run, but the issue is same https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/339/pull-ci-openshift-cluster-version-operator-master-integration/826

2020/03/31 14:48:56 Container bin-hyperkube in pod integration completed successfully
2020/03/31 16:48:34 error: unable to signal to artifacts container to terminate in pod integration, triggering deletion: could not run remote command: unable to upgrade connection: container not found ("artifacts")
2020/03/31 16:48:34 error: unable to retrieve artifacts from pod integration: could not read gzipped artifacts: unable to upgrade connection: container not found ("artifacts")
2020/03/31 16:48:34 error: could not wait for pod 'integration': it is no longer present on the cluster (usually a result of a race or resource pressure. re-running the job should help)
2020/03/31 16:48:34 No custom metadata found and prow metadata already exists. Not updating the metadata.
E0331 16:48:34.053496      15 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".1601717a55d8cd21", GenerateName:"", Namespace:"ci-op-2kbnlkt3", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"", Namespace:"ci-op-2kbnlkt3", Name:"", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"CiJobFailed", Message:"Running job pull-ci-openshift-cluster-version-operator-master-integration for PR https://github.com/openshift/cluster-version-operator/pull/339 in namespace ci-op-2kbnlkt3 from author LalatenduMohanty", Source:v1.EventSource{Component:"ci-op-2kbnlkt3", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf98faf8830f7921, ext:7245935993107, loc:(*time.Location)(0x271e640)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf98faf8830f7921, ext:7245935993107, loc:(*time.Location)(0x271e640)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events ".1601717a55d8cd21" is forbidden: unable to create new content in namespace ci-op-2kbnlkt3 because it is being terminated' (will not retry!)
2020/03/31 16:48:35 Ran for 2h0m46s
error: could not run steps: step integration failed: template pod "integration" failed: pod was deleted while ci-operator step was waiting for it

LalatenduMohanty · 2020-04-16T16:17:42Z

/retest

LalatenduMohanty · 2020-04-16T18:34:51Z

@sdodson The e2e tests has passed now, we should remove the hold

sdodson · 2020-04-16T18:40:56Z

/hold cancel
No need to block on me if the tests are passing now.

LalatenduMohanty · 2020-04-17T09:30:13Z

/retest

LalatenduMohanty · 2020-04-17T18:01:09Z

/retest

LalatenduMohanty · 2020-04-17T19:11:17Z

/retest

Using optr.queue.ShutDown directly as upstream K8s is fixed now

94be9ea

Upstream pull request kubernetes/kubernetes#77170 Signed-off-by: Lalatendu Mohanty <lmohanty@redhat.com>

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 20, 2020

openshift-ci-robot requested review from crawford and jottofar March 20, 2020 19:04

LalatenduMohanty changed the title ~~[WIP] Using optr.queue.ShutDown directly as upstream K8s is fixed now~~ Using optr.queue.ShutDown directly as upstream K8s is fixed now Mar 24, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 24, 2020

openshift-ci-robot assigned wking Mar 24, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 24, 2020

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 16, 2020

openshift-merge-robot merged commit b8af7e4 into openshift:master Apr 17, 2020

Using optr.queue.ShutDown directly as upstream K8s is fixed now #339

Using optr.queue.ShutDown directly as upstream K8s is fixed now #339

Conversation

LalatenduMohanty commented Mar 20, 2020 • edited

wking commented Mar 24, 2020

openshift-bot commented Mar 24, 2020

openshift-bot commented Mar 24, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 25, 2020

openshift-bot commented Mar 26, 2020

abhinavdahiya commented Mar 26, 2020

openshift-ci-robot commented Mar 26, 2020

openshift-bot commented Mar 26, 2020

openshift-bot commented Mar 26, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

openshift-bot commented Mar 27, 2020

sdodson commented Mar 27, 2020

LalatenduMohanty commented Mar 30, 2020

LalatenduMohanty commented Mar 31, 2020

LalatenduMohanty commented Mar 31, 2020 • edited

LalatenduMohanty commented Apr 16, 2020

LalatenduMohanty commented Apr 16, 2020

sdodson commented Apr 16, 2020

LalatenduMohanty commented Apr 17, 2020

LalatenduMohanty commented Apr 17, 2020

LalatenduMohanty commented Apr 17, 2020

LalatenduMohanty commented Mar 20, 2020 •

edited

LalatenduMohanty commented Mar 31, 2020 •

edited