Failover when a node goes down.

[lgammo]

Starting with the tutorial docker images (so monitor, node1 and node2). I have these running and I can monitor their status. So far so good. I have node1 as primary:

node2 |     1 | node2:5432 |   8: 0/F0000D8 |    read-only |           secondary |           secondary
node1 |     2 | node1:5432 |   8: 0/F0000D8 |   read-write |             primary |             primary

Now suppose node1 dies for some reason. On a production system, the node H/W or VM can have a kernel panic, network issues, whatever. Here I am simulating it with:

    docker stop tutorial-node1-1

soon the status is:

node2 |     1 | node2:5432 |   9: 0/F000350 |    read-only |    stop_replication |    stop_replication
node1 |     2 | node1:5432 |   8: 0/F000110 | read-write ! |             primary |      demote_timeout

and then a few seconds later:

node2 |     1 | node2:5432 |   9: 0/F000350 |   read-write |        wait_primary |        wait_primary
node1 |     2 | node1:5432 |   8: 0/F000110 | read-write ! |             primary |             demoted

Node2 stays as wait_primary and seems to be stuck there. I tried to wait 15 minutes and it stayed there.

I then started the container to simulate the VM rebooting or the network restoration:

   docker start tutorial-node1-1

and quickly I see:

node2 |     1 | node2:5432 |   9: 0/11000000 |   read-write |        wait_primary |        wait_primary
node1 |     2 | node1:5432 |   9: 0/11000000 |  read-only ! |          catchingup |          catchingup

then:

node2 |     1 | node2:5432 |   9: 0/11000110 |   read-write |             primary |             primary
node1 |     2 | node1:5432 |   9: 0/11000110 |    read-only |           secondary |           secondary

But while node1 is down, node2 was stuck.

That can't be right as it should have assumed primacy on its own.

On another attempt, I tried drop node --name node1 (after I stopped it), and node2 because a 'single' node right away without waiting for node1 (wait_primary).

I think I am missing a step. Any suggestions? Appreciated!

Thanks,

[DimCitus]

Hi @lgammo ; please take some time to actually read the documentation. Specifically, we have full coverage for the Failover State Machine including a glossary that details what the state names mean, including wait_primary.

I won't copy paste the docs contents in here. In short, the primary state embeds the idea that we have a trustworthy secondary to failover to. Otherwise, the applicable state is wait_primary when there is (at least) another node registered but only one node is available at this time.

In other words, it's all working exactly as designed. And documented...

[lgammo]

The ask is: I need a node to become a primary if a node dies unexpectedly. I can monitor it externally and fix it myself. I wanted to know if I missed something.

Having worked on the design of a high-availability platform many years ago, to me the concept of a failover is distinct from a switchover as one is a response to an error condition and the latter is a controlled changed. The documentation seems to conflate these concepts.

Thanks.

[DimCitus]

A node in the state wait_primary is a primary. The connection is read-write. The state means that it is a primary that does not have a secondary to fail over to. Please, if you have questions about the specifics of the docs and suggestions to make it easier to understand, consider opening a PR or an issue with a specific review.

If you open a discussion because you're too lazy to understand the docs and the pg_auto_failover concepts, instead trying to find your own concepts in our docs, then I can't see how to make this exchange productive.

[lgammo]

I have read the documentation, which have:

"Wait_primary
Applied to a node intended to be the primary but not yet in that position. The primary-to-be at this point knows the secondary’s node name or IP address, and has granted the node hot standby access in the pg_hba.conf file."

I read the 'no yet in that position' to indicate it is not in fact a primary.

By the way, you have a great product @DimCitus.

[DimCitus]

I think the confusion comes from what you want to name a “primary” as in the Postgres docs compared to what the state “primary” embeds as a meaning in the pg_auto_failover docs. When dealing with a pg_auto_failover node, the primary state means it's both a Postgres primary and also it is known to have a standby ready for failover (in the state secondary then).

Your use of primary for me is related to Postgres. I'm trying to make you understand that a pg_auto_failover primary is its own thing, and that in the state wait_primary the node is a Postgres primary node -- though with no node to failover to...

[lgammo]

"Your use of primary for me is related to Postgres. I'm trying to make you understand that a pg_auto_failover primary is its own thing, and that in the state wait_primary the node is a Postgres primary node -- though with no node to failover to..."

Yes there is an overlap in terminology. I will make a few more experiments to understand the implications of the state transitions.

Thanks for the replies.

[Rajeevshar]

if our 3 nodes went in bad state then how we can identify which will take primary role if our 3 nodes show read-only! or None!