The secret-generator pod occasionally fails to start after the pod is being deleted

[eranreshef]

Describe the bug
The secret-generator container sometime restarts after the pod is being replaced.

To Reproduce

1. Deploy the secret-generator with the manifests supplied in this repo.
2. After the pod is up, delete it.
3. The new single container in the new pod will sometime restart due to the wait period when it tries to be the leader.

Expected behavior
A new pod will start and be ready immediately.

Environment:

• Kubernetes version: 1.18.16
• kubernetes-secret-generator version: 3.2.0

Additional context
The generator logs:

kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644831.4539402,"logger":"cmd","msg":"Operator Version: 0.0.1"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644831.4539738,"logger":"cmd","msg":"Go Version: go1.15.5"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644831.4539793,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644831.453984,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644831.4541202,"logger":"leader","msg":"Trying to become the leader."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644832.172396,"logger":"leader","msg":"Found existing lock","LockOwner":"kubernetes-secret-generator-68d6f67df7-bs2lj"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644832.1823065,"logger":"leader","msg":"Not the leader. Waiting."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644833.3122263,"logger":"leader","msg":"Not the leader. Waiting."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644835.699852,"logger":"leader","msg":"Not the leader. Waiting."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644840.244563,"logger":"leader","msg":"Not the leader. Waiting."}
- kubernetes-secret-generator-68d6f67df7-dghwg › kubernetes-secret-generator
+ kubernetes-secret-generator-68d6f67df7-dghwg › kubernetes-secret-generator
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644841.9950554,"logger":"cmd","msg":"Operator Version: 0.0.1"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644841.9950805,"logger":"cmd","msg":"Go Version: go1.15.5"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644841.9950848,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644841.9950886,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644841.9952486,"logger":"leader","msg":"Trying to become the leader."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644842.7113423,"logger":"leader","msg":"Found existing lock","LockOwner":"kubernetes-secret-generator-68d6f67df7-bs2lj"}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644842.723332,"logger":"leader","msg":"Not the leader. Waiting."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644843.855035,"logger":"leader","msg":"Leader pod has been deleted, waiting for garbage collection do remove the lock."}
kubernetes-secret-generator-68d6f67df7-dghwg kubernetes-secret-generator {"level":"info","ts":1617644846.2358556,"logger":"leader","msg":"Became the leader."}

[mittwald-machine]

There has not been any activity to this issue in the last 30 days. It will automatically be closed after 7 more days. Remove the stale label to prevent this.