Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Upgrade the four nodes airgapped Harvester stop with context deadline exceeded error #3729

Closed
TachunLin opened this issue Mar 28, 2023 · 3 comments
Closed

[BUG] Upgrade the four nodes airgapped Harvester stop with context deadline exceeded error #3729

TachunLin opened this issue Mar 28, 2023 · 3 comments
Labels
area/upgrade kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/rare Reproducible less than 10% of the time severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact)

Comments

@TachunLin
Copy link

TachunLin commented Mar 28, 2023 •
edited

Describe the bug

When upgrade the 4 nodes airgapped Harvester cluster from v1.1.1 to v1.1.2-rc4
After Download Upgrade Image complete and enter Creating Upgrade Repository stage

The upgrade process stopped with error

Get "http://upgrade-repo-hvst-upgrade-mldzx.harvester-system/harvester-iso/harvester-release.yaml": 
context deadline exceeded (Client.Timeout exceeded while awaiting headers)

image

To Reproduce
Steps to reproduce the behavior:

  1. Use ipxe-example to provision an airgapped four nodes v1.1.1 Harvester cluster
  2. After cluster ready, vagrant ssh to pxe_server
  3. Setup vlan network to the dhcpd service of pxe_server
  4. Install and setup nfs server on pxe_server
  5. Create the vlan1 network on management interface of Harvester
  6. Create a new cluster network and related network config
  7. Create a vlan network 101 with manual route
  8. Create three images from File
  9. Create three virtual machines separate on each node
  10. Backup each virtual machine
  11. Get harvester v1.1.2-rc4 iso file on pxe_server
  12. Perform offline upgrade to v1.1.2-rc4

Expected behavior

Should be able to complete all upgrade process without error

Support bundle

supportbundle_143ea7c3-25c1-4d61-8f42-c4ff10856943_2023-03-28T14-08-57Z.zip

Environment

  • Harvester ISO version: v1.1.1 to v1.1.2-rc4
  • Underlying Infrastructure: 4 nodes Harvester on local kvm machines

Additional context

  • Nodes resource usage
    image
@TachunLin TachunLin added kind/bug Issues that are defects reported by users or that we know have reached a real release severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact) area/upgrade reproduce/rare Reproducible less than 10% of the time labels Mar 28, 2023
@bk201
Copy link
Member

bk201 commented Mar 29, 2023

Same issue as #3550 and fix in #3593. For v1.1.1 clusters, the workaround is to start the upgrade again.

@TachunLin
Copy link
Author

After delete the upgrade and start the upgrade again.

This time we can proceed the creating upgrade repository stage and completely upgrade to v1.1.2-rc4

image

@bk201
Copy link
Member

bk201 commented May 12, 2023

Duplicated with #3550

@bk201 bk201 closed this as completed May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/upgrade kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/rare Reproducible less than 10% of the time severity/2 Function working but has a major issue w/o workaround (a major incident with significant impact)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bk201 @TachunLin