Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Unable to login harvester dashboard after upgrading from v0.3.0 to v1.0.0-rc1 #1666

Closed
Tracked by #1665
TachunLin opened this issue Dec 13, 2021 · 5 comments
Closed
Tracked by #1665

[BUG] Unable to login harvester dashboard after upgrading from v0.3.0 to v1.0.0-rc1 #1666

TachunLin opened this issue Dec 13, 2021 · 5 comments
Assignees
@bk201 @guangbochen @TachunLin
Labels
area/aaa Authentication, authorization and accounting kind/bug Issues that are defects reported by users or that we know have reached a real release require/doc Improvements or additions to documentation require/release-note severity/1 Function broken (a critical incident with very high impact)
Milestone
v1.0.1

Comments

@TachunLin
Copy link

TachunLin commented Dec 13, 2021 •
edited

Describe the bug

In a 3 nodes harvester cluster with VIP from fixed IP. After upgrade from v0.3.0 to v1.0.0-rc1
User can't login harvester using their original password, show up An error occurred logging in

image

Where to find
This issue discovered during the verification of upgrading harvester from v0.3.0 to v1.0.0-rc1

Steps to reproduce the behavior:

  1. Download harvester v0.3.0 iso and do checksum
  2. Download harvester v1.0.0 iso and do checksum
  3. Use ISO Install a 4 nodes harvester cluster
  4. Use fixed IP as the VIP address in create harvester mode
  5. ISO install the rest of the three harvester node in join mode
  6. Configure a 12 digits password in the first time login
  7. Create ssh key
  8. Enable vlan network with harvester-mgmt
  9. Create virtual network vlan1 with id 1
  10. Create 2 virtual machines
  • ubuntu-vm: 2 core, 4GB memory, 30GB disk
  • centos-vm: 2 core, 4GB memory, 30GB disk
  1. Run upgrade process to v1.0.0-rc1
  2. Since VIP related issue, we can't access harvester dashboard by VIP
  3. SSH to the management node (e.g node1)
  4. Run kubectl port-forward --address <node1 ip> service/rancher 30443:443 -n cale-system
  5. Access harvester dashboard via url https:// {node1 ip}:30443
  6. Login with original password

upgrade process
Follow the manual upgrade steps to upgrade from v0.3.0 to v1.0.0-rc1
https://github.com/harvester/docs/pull/67/files

Expected behavior

Can manual upgrade harvester from v0.3.0 to v1.0.0-rc1 with vm backup.
Can login harvester with original password without error

Support bundle

bundle_2nd_loginFail.zip

Environment:

  • Harvester ISO version before upgrade: v0.3.0
  • Harvester ISO version after upgrade: v1.0.0-rc1
  • Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): 4 nodes harvester cluster on local kvm machines

Harvester network information

  • VIP: 192.168.122.66
  • node1: 192.168.122.45
  • node2: 192.168.122.40
  • node3: 192.168.122.78
  • node4: 192.168.122.2

Additional context
After check, harvester have upgraded to v1.0.0-rc1

harvester-node01-upgrade-rc1:/home/rancher # helm history harvester -n harvester-system
REVISION	UPDATED                 	STATUS    	CHART              	APP VERSION	DESCRIPTION     
16      	Sat Dec 11 04:27:14 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
17      	Sat Dec 11 08:31:14 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
18      	Sat Dec 11 08:31:35 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
19      	Sat Dec 11 14:27:15 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
20      	Sat Dec 11 18:31:14 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
21      	Sat Dec 11 18:31:34 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
22      	Sat Dec 11 18:42:58 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
23      	Sun Dec 12 00:28:30 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
24      	Sun Dec 12 14:29:20 2021	superseded	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
25      	Mon Dec 13 00:29:20 2021	deployed  	harvester-1.0.0-rc1	v1.0.0-rc1 	Upgrade complete
@TachunLin TachunLin added kind/bug Issues that are defects reported by users or that we know have reached a real release area/aaa Authentication, authorization and accounting severity/1 Function broken (a critical incident with very high impact) area/dashboard Harvester/Rancher dashboard labels Dec 13, 2021
@TachunLin TachunLin added this to the v1.0.0 milestone Dec 13, 2021
@bk201 bk201 mentioned this issue Dec 13, 2021
Closed
7 tasks
@bk201
Copy link
Member

bk201 commented Dec 13, 2021

Panic in rancher pod:

2021-12-10T02:35:57.408422927Z E1210 02:35:57.408235      33 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
2021-12-10T02:35:57.408467774Z goroutine 316696 [running]:
2021-12-10T02:35:57.408475779Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39e5e00, 0x482b750)
2021-12-10T02:35:57.408482033Z 	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x95
2021-12-10T02:35:57.408487342Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-12-10T02:35:57.408492749Z 	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x86
2021-12-10T02:35:57.408497561Z panic(0x39e5e00, 0x482b750)
2021-12-10T02:35:57.408502794Z 	/usr/local/go/src/runtime/panic.go:965 +0x1b9
2021-12-10T02:35:57.408509661Z github.com/rancher/rancher/pkg/auth/tokens.(*Manager).UserAttributeCreateOrUpdate(0xc0011917a0, 0xc00a1dd500, 0xa, 0x41aa741, 0x5, 0x6a15bd8, 0x0, 0x0, 0xc009634270, 0x6b636f64225b3a22, ...)
2021-12-10T02:35:57.408516187Z 	/go/src/github.com/rancher/rancher/pkg/auth/tokens/manager.go:598 +0x285
2021-12-10T02:35:57.408520966Z github.com/rancher/rancher/pkg/auth/tokens.(*Manager).NewLoginToken.func1(0x6f68676e6f6c2f6f, 0x67616e616d2d6e72, 0x2e322e31763a7265)
2021-12-10T02:35:57.408526055Z 	/go/src/github.com/rancher/rancher/pkg/auth/tokens/manager.go:659 +0x8e
2021-12-10T02:35:57.408545914Z k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc001f7cc90, 0x616c662d64656e00, 0x0, 0x0)
2021-12-10T02:35:57.408554718Z 	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:211 +0x69
2021-12-10T02:35:57.408560041Z k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x5f5e100, 0x4000000000000000, 0x3fc999999999999a, 0x5, 0x0, 0xc001f7cc90, 0x6568636e61722f6f, 0x6e65647261682f72)
2021-12-10T02:35:57.408566303Z 	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:399 +0x55
2021-12-10T02:35:57.408571078Z github.com/rancher/rancher/pkg/auth/tokens.(*Manager).NewLoginToken(0xc0011917a0, 0xc00a1dd500, 0xa, 0x0, 0x0, 0x0, 0x0, 0xc0076a57d0, 0x12, 0x0, ...)
2021-12-10T02:35:57.408575468Z 	/go/src/github.com/rancher/rancher/pkg/auth/tokens/manager.go:658 +0x1b7
2021-12-10T02:35:57.408578376Z github.com/rancher/rancher/pkg/auth/providers/publicapi.(*loginHandler).createLoginToken(0xc001236fc0, 0xc00735ea20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2021-12-10T02:35:57.408581298Z 	/go/src/github.com/rancher/rancher/pkg/auth/providers/publicapi/login.go:229 +0xe85
2021-12-10T02:35:57.408585383Z github.com/rancher/rancher/pkg/auth/providers/publicapi.(*loginHandler).login(0xc001236fc0, 0xc00659af57, 0x5, 0xc00771ee40, 0xc00735ea20, 0xc0028a0000, 0xc001107f20)
2021-12-10T02:35:57.408588303Z 	/go/src/github.com/rancher/rancher/pkg/auth/providers/publicapi/login.go:69 +0x189
2021-12-10T02:35:57.408591393Z github.com/rancher/norman/api.handleAction(0xc00771ee40, 0xc00735ea20, 0x0, 0x0)

rancher.log

@gitlawr
Copy link
Contributor

gitlawr commented Dec 13, 2021

@bk201 See #1549 (comment)

@bk201
Copy link
Member

bk201 commented Dec 13, 2021

@gitlawr Thanks. Does this mean we need to remove the UserAttribute CRD in the manual upgrade path?

@gitlawr
Copy link
Contributor

gitlawr commented Dec 13, 2021

remove or apply. Both sound good to me but it needs testing to verify

@guangbochen guangbochen added require/release-note require/doc Improvements or additions to documentation and removed area/dashboard Harvester/Rancher dashboard labels Dec 14, 2021
This was referenced Dec 17, 2021
Closed
Closed
@guangbochen guangbochen modified the milestones: v1.0.0, v1.0.1 Dec 19, 2021
@TachunLin TachunLin self-assigned this Dec 24, 2021
@TachunLin
Copy link
Author

Verified fixed on v1.0.0 (GA). Close this issue.

Result

After manual upgrade from v0.3.0 to v1.0.0 GA
With the u
User can use original admin password to login harvester dashboard after upgrade to v1.0.0

Test Information

  • Test Environment: 4 nodes harvester on local kvm machine
  • Harvester version: v1.0.0 GA

Verify Steps

  1. Download harvester v0.3.0 iso and do checksum

  2. Download harvester v1.0.0 GA iso and do checksum

  3. Use ISO Install a 4 nodes harvester cluster

  4. Use fixed IP as the VIP address in create harvester mode

  5. ISO install the rest of the three harvester node in join mode

  6. Configure a 12 digits password in the first time login

  7. Create ssh key

  8. Enable vlan network with harvester-mgmt

  9. Create virtual network vlan1 with id 1

  10. Create 2 virtual machines

  11. ubuntu-vm: 2 core, 4GB memory, 30GB disk

  12. centos-vm: 2 core, 4GB memory, 30GB disk

  13. Run upgrade process to v1.0.0 GA
    https://github.com/harvester/docs/blob/a4be9a58441eeee3b5564b70e499dc69c6040cc8/docs/upgrade.md

  14. Since VIP related issue, we can't access harvester dashboard by VIP

  15. SSH to the management node (e.g node1)

  16. Run kubectl port-forward --address service/rancher 30443:443 -n cale-system

  17. Access harvester dashboard via url https:// {node1 ip}:30443

  18. Login with original password

Additional Context

Support bundle
supportbundle_305b7d49-9118-4d65-a01c-0b66b483790e_2021-12-24T02-44-53Z.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bk201 bk201

@guangbochen guangbochen

@TachunLin TachunLin

Labels
area/aaa Authentication, authorization and accounting kind/bug Issues that are defects reported by users or that we know have reached a real release require/doc Improvements or additions to documentation require/release-note severity/1 Function broken (a critical incident with very high impact)
Projects
None yet
Milestone
v1.0.1
Development

No branches or pull requests
4 participants
@bk201 @guangbochen @gitlawr @TachunLin