Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle non zero deletionTimestamp for child objects. #953

Closed
f41gh7 opened this issue May 15, 2024 · 2 comments
Closed

handle non zero deletionTimestamp for child objects. #953

f41gh7 opened this issue May 15, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@f41gh7
Copy link
Collaborator

f41gh7 commented May 15, 2024

Currently, if by mistake ServiceAccount ( or any other object) was deleted, operator doesn't remove finalizer for it. Since parent object wasn't marked as deleted. It blocks kubernetes control-plain operations.

Proposed solution:

  1. check for deletion timestamp.
  2. if timestamp is not zero, remove finalizer and raise an error
  3. object must be recreated and the next reconcile loop.
@f41gh7 f41gh7 added the bug Something isn't working label May 15, 2024
@jmleddy
Copy link

jmleddy commented May 15, 2024

Here's the reproducer if needed:

james@M-D19FP4G4QL ~/src/kube-resources $ kubectl get sa vmagent-az1
NAME          SECRETS   AGE
vmagent-az1   0         95d
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl get sa vmagent-az1 -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2024-02-09T15:55:25Z"
  finalizers:
  - apps.victoriametrics.com/finalizer
<snip>
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl delete sa vmagent-az1
serviceaccount "vmagent-az1" deleted
^Z
[1]+  Stopped                 kubectl delete sa vmagent-az1
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl get sa vmagent-az1  -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2024-02-09T15:55:25Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-05-15T12:56:27Z"
  finalizers:
  - apps.victoriametrics.com/finalizer
<snip
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl get pods
NAME            READY   STATUS    RESTARTS   AGE
vmagent-az1-0   3/3     Running   0          6d15h
vmagent-az2-0   3/3     Running   0          6d15h
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl rollout restart sts vmagent-az1
statefulset.apps/vmagent-az1 restarted
james@M-D19FP4G4QL ~/src/kube-resources $ kubectl get pods
NAME            READY   STATUS               RESTARTS      AGE
vmagent-az1-0   1/3     PostStartHookError   0 (13s ago)   89s
vmagent-az2-0   3/3     Running              0             6d16h
james@M-D19FP4G4QL ~/src/kube-resources $

f41gh7 added a commit that referenced this issue May 20, 2024
… DeletionTimestamp

Kubernetes performs soft delete and waits for object hard delete until finalizers == 0.
During that period any actions for soft deleted objects aren't performend. It caues weird behavior for service accounts, deployments and etc.
When kubernetes-controller manager ignores actions that must be performend with it. For instance, pod creation for soft deleted deployment.

 Operator now detects soft delete and free objects. An object ll be recreate at the next reconcile loop and error message ll be logged.

#953
f41gh7 added a commit that referenced this issue May 21, 2024
f41gh7 added a commit that referenced this issue May 21, 2024
… DeletionTimestamp (#956)

* controllers/finalize: Remove finalizer for child objects for non zero DeletionTimestamp

Kubernetes performs soft delete and waits for object hard delete until finalizers == 0.
During that period any actions for soft deleted objects aren't performend. It caues weird behavior for service accounts, deployments and etc.
When kubernetes-controller manager ignores actions that must be performend with it. For instance, pod creation for soft deleted deployment.

 Operator now detects soft delete and free objects. An object ll be recreate at the next reconcile loop and error message ll be logged.

#953

* apply review comments
@f41gh7
Copy link
Collaborator Author

f41gh7 commented Jun 11, 2024

Changes was included to v0.45.0 release.

@f41gh7 f41gh7 closed this as completed Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants