Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Uninstallation job stuck #2945

Closed
khushboo-rancher opened this issue Aug 27, 2021 · 2 comments
Closed

[BUG] Uninstallation job stuck #2945

khushboo-rancher opened this issue Aug 27, 2021 · 2 comments
Assignees
@PhanLe1010 @khushboo-rancher
Labels
kind/bug reproduce/rare < 50% reproducible severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact)
Milestone
v1.2.0

Comments

@khushboo-rancher
Copy link
Contributor

khushboo-rancher commented Aug 27, 2021 •
edited

Describe the bug
The uninstallation job stuck with Found 4 instance managers remaining" controller=longhorn-uninstall

To Reproduce
Steps to reproduce the behavior:

  1. Deploy Longhorn v1.1.2
  2. Perform some operations like create a volume, create backup, restore backup etc
  3. Upgrade longhorn to v1.2.0-rc2.
  4. Uninstall Longhorn using kubectl.
  5. See error in the pod logs
Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:11Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:12Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:12Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall
time="2021-08-27T18:01:12Z" level=info msg="Found 4 instance managers remaining" controller=longhorn-uninstall

Expected behavior
Uninstall job should not stuck

Log
container (24).log

Environment:

  • Longhorn version: v1.2.0-rc2
  • Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl

Additional context
One of the support bundle taken in the cluster.
longhorn-support-bundle_27b87f7a-0484-4102-b36b-bc3c8a118182_2021-08-27T06-43-42Z.zip
@khushboo-rancher khushboo-rancher added kind/bug reproduce/rare < 50% reproducible severity/3 Function working but has a major issue w/ workaround severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact) and removed severity/3 Function working but has a major issue w/ workaround labels Aug 27, 2021
@innobead innobead added this to the v1.2.1 milestone Aug 27, 2021
@PhanLe1010
Copy link
Contributor

PhanLe1010 commented Aug 27, 2021 •
edited

It is a bug in the upgrade path. A race condition that sometime we forget to remove the finalizer from all instance managers. More specifically we should replace this line with continue

@PhanLe1010 PhanLe1010 mentioned this issue Aug 27, 2021
Merged
@innobead innobead modified the milestones: v1.2.1, v1.2.0 Aug 27, 2021
@PhanLe1010 PhanLe1010 self-assigned this Aug 27, 2021
@khushboo-rancher khushboo-rancher self-assigned this Aug 29, 2021
@khushboo-rancher
Copy link
Contributor Author

Verified with Longhorn-master 08/28/2021

Validation - Pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@PhanLe1010 PhanLe1010

@khushboo-rancher khushboo-rancher

Labels
kind/bug reproduce/rare < 50% reproducible severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact)
Projects
None yet
Milestone
v1.2.0
Development

No branches or pull requests
3 participants
@innobead @PhanLe1010 @khushboo-rancher