New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use workqueue model in expand_controller #71760

Open
jingxu97 opened this Issue Dec 5, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@jingxu97
Contributor

jingxu97 commented Dec 5, 2018

What would you like to be added:
Currently, expand_controller for resizing a volume is using desired/actual state model. The controller watches PVC add/update/delete events and then for each event, it updates its in-memory cache to store the resize request information. A reconciler periodically compares desired state and actual state and take actions in order to move actual state to the desired one.

This model might cause some extra overhead for controller to periodically sync the desired and actual state and store them. We are considering to use workqueue model which has some build-in features such as setting rate limit (e.g., exponentialFailureRateLimiter) and exposing metrics.

/kind feature

@gnufied

This comment has been minimized.

Member

gnufied commented Dec 6, 2018

Just for background - the reason we chose not to use workqueue is because, resize is level based. So resize of a PVC from 2GB to 4GB and then to 6GB and finally to 8GB - need not visit all the nodes in transaction. If while 2GB to 4GB resize was in progress and other resizes arrive, then for that PVC only final resize will be done ignoring intermediate ones. So, we will have to internally manage some state anyways, so as we don't process all resize operations on same PVC.

The rate limiting feature supported by workqueue is on per item basis, which I doubt will be useful for volume expansion where if a user is submitting multiple resizes for same PVC, intermediate resize requests are simply ignored.

@jingxu97

This comment has been minimized.

Contributor

jingxu97 commented Dec 6, 2018

I think depending on how fast user sends out their request and how fast controller processes, both desired/actual state and workqueue model might end up processing every request or just one final request. They should not have much differences.

For example, if user changes from 2GB to 4GB, and then change from 4GB to 6GB in the next second. In desired/actual state model, sync_volume_resize controller resync very 400 Millisecond. So it will process first request and then second request.

For workqueue model, we are not going to put every request into the queue. Instead, we put PVC unique name into the queue and worker get an item from the queue. If user put multiple PVC change requests very fast, the worker gets the PVC name and then get the lastest PVC object from the cache, it might resize directly from 2GB to 4GB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment