Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

taking more time than expected to update the Resync status after marking VR as secondary #120

Closed
Madhu-1 opened this issue Oct 18, 2021 · 1 comment · Fixed by #123
Closed

taking more time than expected to update the Resync status after marking VR as secondary #120

Madhu-1 opened this issue Oct 18, 2021 · 1 comment · Fixed by #123
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@Madhu-1
Copy link
Member

Madhu-1 commented Oct 18, 2021 •
edited 

kubectl logs po/csi-rbdplugin-provisioner-5f899665f4-4fcv4 -nrook-ceph -c csi-rbdplugin -f | grep -i "ResyncVolume"
I1018 06:48:31.973060       1 utils.go:176] ID: 34 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:32.170678       1 utils.go:176] ID: 37 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:32.360827       1 utils.go:176] ID: 40 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:32.576044       1 utils.go:176] ID: 43 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:32.791402       1 utils.go:176] ID: 46 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:33.020634       1 utils.go:176] ID: 49 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:33.354465       1 utils.go:176] ID: 52 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:33.854856       1 utils.go:176] ID: 55 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:34.641120       1 utils.go:176] ID: 58 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:36.053590       1 utils.go:176] ID: 61 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:38.761398       1 utils.go:176] ID: 64 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:48:44.050990       1 utils.go:176] ID: 67 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:49:56.144036       1 utils.go:176] ID: 73 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:51:18.334437       1 utils.go:176] ID: 78 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:54:02.363541       1 utils.go:176] ID: 84 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 06:59:30.336139       1 utils.go:176] ID: 92 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 07:10:25.989196       1 utils.go:176] ID: 106 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 07:27:06.238203       1 utils.go:176] ID: 126 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 07:43:46.501844       1 utils.go:176] ID: 145 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 08:00:26.754842       1 utils.go:176] ID: 165 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume
I1018 08:17:07.005895       1 utils.go:176] ID: 185 Req-ID: 0001-0009-rook-ceph-0000000000000001-8a30bf5b-2fdd-11ec-bf5a-0242ac110005 GRPC call: /replication.Controller/ResyncVolume

Due to exponential backoff of Requeue, it's taking more time than expected to update the conditions in the VR. Even if the storage is recovered after a certain amount of time. due to exponential backoff, the request is not sent to the drivers very frequently. this behavior can affect the VR update on time and also it can affect the RTO time of the resources.

The suggestion is to use RequeueAfter: time.Duration(time. Second * 30) for the Reque condition as it makes sure that the request will be required for every 30 seconds.
@Madhu-1
Copy link
Member Author

Madhu-1 commented Oct 18, 2021

CC @ShyamsundarR

@Madhu-1 Madhu-1 added enhancement New feature or request good first issue Good for newcomers labels Oct 18, 2021
Madhu-1 added a commit to Madhu-1/operator that referenced this issue Oct 25, 2021
@Madhu-1
replication: set RequeueAfter to 30 secondary for requeue
e9bd094 
Setting Requeue time for 30 seconds as the
resync can take time and having default
Requeue exponential backoff time can affect
the RTO time

closes: csi-addons#120

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
@Madhu-1 Madhu-1 mentioned this issue Oct 25, 2021
Merged
@mergify mergify bot closed this as completed in #123 Oct 25, 2021
mergify bot pushed a commit that referenced this issue Oct 25, 2021
@Madhu-1 @mergify
replication: set RequeueAfter to 30 secondary for requeue
8c7d3f8 
Setting Requeue time for 30 seconds as the
resync can take time and having default
Requeue exponential backoff time can affect
the RTO time

closes: #120

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Madhu-1 added a commit to red-hat-storage/volume-replication-operator that referenced this issue Oct 27, 2021
@Madhu-1
replication: set RequeueAfter to 30 secondary for requeue
cd87973 
Setting Requeue time for 30 seconds as the
resync can take time and having default
Requeue exponential backoff time can affect
the RTO time

closes: csi-addons#120

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 8c7d3f8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

replication: set RequeueAfter to 30 secondary for requeue Madhu-1/operator
1 participant
@Madhu-1