[BACKPORT][v1.6.1][BUG] The activated DR volume do not contain the latest data.

[github-actions]

backport #7945

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at controller: DR volume should wait for the latest backup restored before activation (backport #2611) longhorn-manager#2615
  controller: Fix issue that regular restore volume won't be detached (backport #2621) longhorn-manager#2622

[roger-ryao]

Verified on v1.6.x-head 20240223

• longhorn v1.6.x-head 3382579
• longhorn-manager v1.6.x-head longhorn/longhorn-manager@bf3edb2

The test steps

Test Method 1 : #7945 (comment)

1. Launch 2 clusters and both have Longhorn installed
2. Set up a backup target.
3. Create two volumes and write data in the 1st cluster. Then create volumes' backups.
4. Restore two backup files as DR volumes in the 2nd cluster.
5. Modify the backup poll interval to a large value.
6. Write more data for the volumes in the 1st cluster, and create the 2nd backup.
7. Activate one of the DR volumes in the 2nd cluster. Then verify the data

Test Method 2 refer:#7947 (comment)

Result
Hi @shuo-wu
The activated DR volume does not contain the latest data. However, manually updating backupVolume.Spec.SyncRequestedAt causes the DR volume to sync up with the latest backup.
Could you help to check it?

supportbundle_91df4b65-3a16-4cb1-8d8f-d901ffb280da_2024-02-23T10-10-56Z.zip
volume name : pvc-3da73260-6e9d-4a18-80ac-ae63864ed6c2

[shuo-wu]

Weird...
The activation is triggered (and finished) at 2024-02-23T10:06:23Z. backupVolume.Spec.SyncRequestedAt should be the same timestamp. But its value in the support bundle is 2024-02-23T10:06:25Z. I have not found that which one updates it again. BTW, the 2nd backup backup-8fefd2d832114dcb creation time is 2024-02-23T10:06:24Z

2024-02-23T10:06:23.531811509Z time="2024-02-23T10:06:23Z" level=info msg="Activating volume pvc-3da73260-6e9d-4a18-80ac-ae63864ed6c2 with frontend blockdev" func="manager.(*VolumeManager).Activate" file="volume.go:422"

2024-02-23T10:06:23.849213385Z time="2024-02-23T10:06:23Z" level=info msg="Restore/DR volume finished with the last restored backup backup-c0d99731ad034d59" func="controller.(*VolumeController).checkAndFinishVolumeRestore" file="volume_controller.go:3214" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=ryao-16x-w1-29d1f2d8-bwq5j owner=ryao-16x-w1-29d1f2d8-bwq5j shareEndpoint= shareState=stopped state=attached volume=pvc-3da73260-6e9d-4a18-80ac-ae63864ed6c2

Is this case always reproducible? Can this be reproduced in other versions (master or v1.5.x)?

[roger-ryao]

Weird... The activation is triggered (and finished) at 2024-02-23T10:06:23Z. backupVolume.Spec.SyncRequestedAt should be the same timestamp. But its value in the support bundle is 2024-02-23T10:06:25Z. I have not found that which one updates it again. BTW, the 2nd backup backup-8fefd2d832114dcb creation time is 2024-02-23T10:06:24Z

2024-02-23T10:06:23.531811509Z time="2024-02-23T10:06:23Z" level=info msg="Activating volume pvc-3da73260-6e9d-4a18-80ac-ae63864ed6c2 with frontend blockdev" func="manager.(*VolumeManager).Activate" file="volume.go:422"

2024-02-23T10:06:23.849213385Z time="2024-02-23T10:06:23Z" level=info msg="Restore/DR volume finished with the last restored backup backup-c0d99731ad034d59" func="controller.(*VolumeController).checkAndFinishVolumeRestore" file="volume_controller.go:3214" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=ryao-16x-w1-29d1f2d8-bwq5j owner=ryao-16x-w1-29d1f2d8-bwq5j shareEndpoint= shareState=stopped state=attached volume=pvc-3da73260-6e9d-4a18-80ac-ae63864ed6c2

Is this case always reproducible? Can this be reproduced in other versions (master or v1.5.x)?

I built up the test environment 3 times, and the issue could be reproduced on v1.6.x. However, I didn't observe the issue on v1.5.4-rc3. I think I should test it on v1.5.4-rc4. I didn't test it with the master-head yet.

[roger-ryao]

Verified on v1.6.x-head 20240301

• longhorn v1.6.x-head 3382579
• longhorn-manager v1.6.x-head longhorn/longhorn-manager@654ee92

The test steps

Test Method 1 : #7945 (comment)

1. Launch 2 clusters and both have Longhorn installed
2. Set up a backup target.
3. Create two volumes and write data in the 1st cluster. Then create volumes' backups.
4. Restore two backup files as DR volumes in the 2nd cluster.
5. Modify the backup poll interval to a large value.
6. Write more data for the volumes in the 1st cluster, and create the 2nd backup.
7. Activate one of the DR volumes in the 2nd cluster. Then verify the data

Test Method 2 refer:#7947 (comment)

Result Passed

• 1. Test Method 1.
• 2. Test Method 2
• 3. require/manual-test-plan : test(manual): Sync up with backup target during DR volume activation longhorn-tests#1732