[WFL-Operator-100] adding a way to report heuristic transaction state during scaledown #295
Conversation
|
|
||
| When transaction scale down process reviews the state of transactions in the pods it checks if there are no unfinished prepared transactions. | ||
| If there are found some then it does not permit the pod being terminated and the pod is marked with the _state_ `SCALING_DOWN_RECOVERY_DIRTY`. | ||
| It's marked as dirty until all transactions are safely recovered. |
There was a problem hiding this comment.
Is pod fully working in dirty state? Is load balancer forwarding traffic to such pod is it handling requests?
What happens when I scale up to origin number of pods again? I understand dirty state of pod is not changed.
Can I scale up to bigger number of pods?
I am thinking of usecase where Autoscaler is involved based on current cluster utilization. So pods get scaledown and scaleup randomly. How it will happen in this situation?
There was a problem hiding this comment.
This summarizes how the current functionality is already working.
- When in dirty state the pod is not capable to start a new transaction. But otherwise it's working. There is a issue to manage that in a better way here: https://issues.redhat.com/browse/EAP7-1355
- No, load balancer is not forwarding the traffic to the pod
- The pod should be moved to the state ACTIVE. The pod is not dirty anymore as it's normally operating and normally operating pod may process recovery.
- Yes, you can scale up as you wish. The processing of scaledown does not touch it
All these situations should be part of the eap xpaas testsuite.
I assume the scaledown and scaleup processing in a faster pace should be working fine. Just it could happen that the pod won't be really never scaleddown as scaling down really takes time. That needs to be checked.
| The goal of this feature change is that the WildFly Operator needs to report better the transactions in heuristic to user/administrator. | ||
|
|
||
| * a new _pod state_ should be created. This _state_ would mean that there are some transactions in heuristic state which needs a manual recovery (because otherwise the pod is never terminated) | ||
| * when the pod is marked with this heuristic _state_ an event about this should be emitted (event should be bound to the pod and to the operator object as well). The event announces that there is a pod in scaledown process which requires a manual assistance. |
There was a problem hiding this comment.
Thinking how to notify user something is going on. For me as administrator it would be nice to have OpenShift Alert fired. Don't know how feasible that is though.
With new state and event, administrator must get notify something happens which needs its attention. In case of manual scaledown this can be included in his operations steps. But in case of Autoscaler he must be notified somehow.
There was a problem hiding this comment.
I don't know about some other Kubernetes primitives that could be used for announcing that something happens. I know only about events thus they are expected to be used here.
Do you have some more ideas how to announce the user?
There was a problem hiding this comment.
I meant to provide alert rule. But as I am reading [1] Currently you cannot add custom alerting rules So it is up to administrator to add pod state to monitoring which they are using.
| == Test Plan | ||
|
|
||
| The test means to automatize the checks for newly created state and counter and to check if the flag is working. | ||
| The EAP QE testsuite for OpenShift should be used for this: https://gitlab.mw.lab.eng.bos.redhat.com/jbossqe-eap/openshift-eap-tests |
… during scaledown
ochaloup
force-pushed
the
WFLY-OPERATOR-100-report-better-heuristic-txn-state
branch
from
f0b421a
to
6d7943e
Compare
wildfly/wildfly-operator#100
https://issues.redhat.com/browse/EAP7-1394