Implement online enable/disable monitor support. #591
Conversation
| } | ||
|
|
||
| NodeState initialState = | ||
| pgSetup->is_in_recovery ? WAIT_STANDBY_STATE : SINGLE_STATE; |
There was a problem hiding this comment.
There might be a primary and a demoted primary when the monitor is enabled. How does the initialState affect things?
There was a problem hiding this comment.
You are pointing exactly to why we don't have a fully automated way to register nodes again to a new monitor. To solve the problem, we ask the user to be careful about the node registration ordering. Once a primary has been registered, the other primary might be more interesting to support.
I suppose on this case we are back to the previous situation where the current state needs to be dropped and a new node created from scratch again:
$ pg_autoctl stop
$ pg_autoctl drop node --pgdata ...
$ pg_autoctl create postgres --pgdata ... ...
But that's only when you're in the unfortunate position of losing the monitor node just after having lost a primary node and then you now have all the nodes back online and you're dealing with all the pieces.
There was a problem hiding this comment.
I think to avoid this issue we should not to use is_in_recovery to determine in which state the node should be registered. I think it makes more sense to do this based on the current state of the node.
- if state is
primary,wait_primaryorsingle(and maybedraining) we should try to join as single. - any other state we should try to join as
wait_standby
There was a problem hiding this comment.
I think there is a scenario where it's better to use is_in_recovery: when deploying to two regions for disaster recovery purposes, and the first (primary) region has just been lost, then we want to promote the only node left in the DR region. It's possible to do so with:
- pg_autoctl disable monitor
- pg_ctl promote (or call the SQL function)
- start a new monitor
- pg_autoctl enable monitor
At this point the FSM state on-disk is not relevant anymore. It can be argued that one could use pg_autoctl do fsm assign single in step 2, though this is not documented at the moment, and required PG_AUTOCTL_DEBUG=1 in the environment.
| } | ||
|
|
||
| NodeState initialState = | ||
| pgSetup->is_in_recovery ? WAIT_STANDBY_STATE : SINGLE_STATE; |
There was a problem hiding this comment.
I think to avoid this issue we should not to use is_in_recovery to determine in which state the node should be registered. I think it makes more sense to do this based on the current state of the node.
- if state is
primary,wait_primaryorsingle(and maybedraining) we should try to join as single. - any other state we should try to join as
wait_standby
DimCitus
force-pushed
the
feature/reset-state
branch
from
c6804e0 to
8e72f48
Compare
DimCitus
force-pushed
the
feature/reset-state
branch
from
8e72f48 to
b55426f
Compare
DimCitus
force-pushed
the
feature/reset-state
branch
from
14d548f to
dc4e057
Compare
This allows for adjusting to a lost monitor while keeping our services online. At run-time, we can now $ pg_autoctl disable monitor $ pg_autoctl enable monitor --monitor ... Taking care of the order of operations, the cluster can continue behaving correctly with a minimum amount of disturbance.
Reduce each matrix job to a smaller work unit, it seems Travis is having trouble again with too many things in a single work unit.
DimCitus
force-pushed
the
feature/reset-state
branch
from
8eca60c to
a869445
Compare
This allows for adjusting to a lost monitor while keeping our services
online. At run-time, we can now
Taking care of the order of operations, the cluster can continue behaving
correctly with a minimum amount of disturbance.