Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.14-SNAPSHOT is generating WaiterException: CleaningProject #308

Closed
mchoma opened this issue Sep 6, 2019 · 5 comments
Closed

0.14-SNAPSHOT is generating WaiterException: CleaningProject #308

mchoma opened this issue Sep 6, 2019 · 5 comments

Comments

@mchoma
Copy link
Contributor

mchoma commented Sep 6, 2019

We are missing way to specify deletion policy in kubernetes client [1].
(And probably with 0.13 default becomes orphan policy??)

Anyway, currently we are experiencing *-deploy pods not beeing deleted because they have finalizer: orphan and ownerReference filled to parent object (I think it was ReplicaSet) in same time.

So either

  1. OpenShift.clean() does not clean parent objects properly. Should we return to cascading(true) ?
  2. Or there is some bug in kubernetes GC and ownerReference is not cleared properly from child objects

Shouldnt we apply cleanFinalizers workaround [2] again to cover this scenario?

[1] fabric8io/kubernetes-client#1614
[2] caa00a4

@Crumby
Copy link
Contributor

Crumby commented Sep 23, 2019

Are you using same fabric8 client version as the xtf release? -deploy orphans shouldn't be a problem with 4.3.1.

@mchoma
Copy link
Contributor Author

mchoma commented Sep 23, 2019

With aligning client versions this problem is gone. What are we experiencing now is much more longer testsuite run - it has nearly doubled. From nearly 5 hours to 9 hours. Have you noticed something similar?

@mchoma mchoma closed this as completed Sep 23, 2019
@mchoma mchoma changed the title 0.13 is generating WaiterException: CleaningProject 0.14-SNAPSHOT is generating WaiterException: CleaningProject Oct 3, 2019
@mchoma mchoma reopened this Oct 3, 2019
@mchoma
Copy link
Contributor Author

mchoma commented Oct 3, 2019

@dsimansk Now with 0.14-SNAPSHOT I run into situation that ReplicaSet was not deleted Replica Sets jaeger-7c76d4c55, thus jaeger pod keep restarting.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  annotations:
    deployment.kubernetes.io/desired-replicas: '1'
    deployment.kubernetes.io/max-replicas: '1'
    deployment.kubernetes.io/revision: '1'
  creationTimestamp: '2019-10-03T07:43:01Z'
  generation: 1
  labels:
    app: jaeger
    jaeger-infra: jaeger-pod
    pod-template-hash: '373280711'
  name: jaeger-7c76d4c55
  namespace: eap72-tests-13-mx
  resourceVersion: '304197'
  selfLink: /apis/apps/v1/namespaces/eap72-tests-13-mx/replicasets/jaeger-7c76d4c55
  uid: 6d4a5667-e5b1-11e9-bd82-fa163e733bcb

After manual delete through web console problem was away.

@Crumby
Copy link
Contributor

Crumby commented Oct 3, 2019

@mchoma The issue is that clean() doesn't clean ReplicaSet at all. It's quite simple fix however.

mchoma pushed a commit to mchoma/xtf that referenced this issue Oct 11, 2019
mchoma pushed a commit to mchoma/xtf that referenced this issue Oct 16, 2019
mchoma pushed a commit to mchoma/xtf that referenced this issue Oct 16, 2019
dsimansk added a commit that referenced this issue Oct 16, 2019
[#308] Clean does not remove ReplicaSets
@dsimansk
Copy link
Contributor

Resolved in #321.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants