[BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate

[github-actions]

backport #8114

[innobead]

ref: #8167

cc @ejweber

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: Track replica mode transition times with replicaTransitionTimeMap longhorn-manager#2685 (comment).
  Run the following test again: Be more careful while cleaning up stale replicas longhorn-manager#2432 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  @krohrsb managed to mitigate it by increasing the capability of his API server ([BUG] Longhorn api-server PUT request rate #8114 (comment)), but there is nothing a user can do to tweak Longhorn to avoid the issue.

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Track replica mode transition times with replicaTransitionTimeMap (backport #8151) #8167.
  The PR for the chart change is at: Track replica mode transition times with replicaTransitionTimeMap (backport #8151) #8167.

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Track replica mode transition times with replicaTransitionTimeMap (backport #2685) longhorn-manager#2694.

[roger-ryao]

Verified on v1.6.1-rc1 20240314

• longhorn v1.6.1-rc1 6e96e27

The test steps
longhorn/longhorn-manager#2432 (comment)
#7425 (comment)

Result Passed

1. If you wait any length of time, ReplicaA is never cleaned up.
2. After deleting ReplicaB and waiting for the volume to rebuild from ReplicaA, the data is consistent.

[ejweber]

Hello @roger-ryao, can you please confirm that you also ran longhorn/longhorn-manager#2685 (comment) to test this fix? Sorry for the confusion. I do think longhorn/longhorn-manager#2432 (comment) should be run again (as you have done), but the former are the primary test steps.

Moving this back to Ready for Testing. Please feel free to close it again directly if you have already run these steps.

[roger-ryao]

Verified on v1.6.1-rc1 20240318

• longhorn v1.6.1-rc1 6e96e27

The test steps
longhorn/longhorn-manager#2685 (comment)
There are four cases to verify:

1. Ensure that the upgrade correctly populates replicaTransitionTimeMap
2. Confirm that the rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica deletion
3. Validate that the rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica failure
4. Ensure that replicas are not updated unnecessarily when the cluster is stable.

Result

1. We observed that the replicaTransitionTimeMap is not visible on v1.6.1-rc1, but the test passed on the master-head.
2. Attaching the screenshot, PUT requests to replica resources persist. Additionally, after updating Longhorn from v1.6.1-rc1 to the master-head, no PUT requests to replica resources were observed.

cc. @ejweber

[ejweber]

This is correct. The necessary changes did not make it to longhorn-manager in https://github.com/longhorn/longhorn-manager/commits/v1.6.1-rc1. They are only in https://github.com/longhorn/longhorn-manager/commits/v1.6.x/.

cc @roger-ryao

[roger-ryao]

Verified on v1.6.x-head 20240319

• longhorn v1.6.x-head 6e96e27
• longhorn-manager v1.6.x-head https://github.com/longhorn//longhorn-manager/commit/0dd766f7376b13bc6ecc1c94f6b7a311d72e84e8

Result Passed

• 1. We observed that the replicaTransitionTimeMap is visible on v1.6.x-head.
• 2. The rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica deletion
• 3. The rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica failure
• 4. After updating Longhorn from v1.6.1-rc1 to the v1.6.x-head, no PUT requests to replica resources were observed.