OraOperator v1.2.0 automatic failover issues when reinstating new standby

[andbos]

I'm trying out the new Automatic failover feature. Could you perhaps suggest a good way of testing it? I deleted the primary SIDB pod and this triggered automatic failover alright but reinstation of the new standby fails with error:

2025-07-03T13:04:37.906+00:00
Initiating reinstatement for database "ncnfdb11"...
Reinstating database "ncnfdb11", please wait...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "ncnfdb11" failed
2025-07-03T13:04:37.965+00:00
[W000 2025-07-03T13:04:38.911+00:00] Failed to reinstate database ncnfdb11. Will retry later.
[W000 2025-07-03T13:04:54.936+00:00] Try to connect to the new standby ncnfdb11.
[W000 2025-07-03T13:04:55.936+00:00] Connection to the new standby restored!
[W000 2025-07-03T13:04:57.941+00:00] Try to connect to the primary ncnfdb12:1521/ncnfdb12.
[W000 2025-07-03T13:04:58.942+00:00] Connection to the primary restored!
[W000 2025-07-03T13:04:58.942+00:00] Wait for new primary to be ready to reinstate.
[W000 2025-07-03T13:04:58.946+00:00] New primary is now ready to reinstate.
[W000 2025-07-03T13:04:59.946+00:00] Issuing REINSTATE command.

2025-07-03T13:04:59.946+00:00
Initiating reinstatement for database "ncnfdb11"...
Reinstating database "ncnfdb11", please wait...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "ncnfdb11" failed
2025-07-03T13:05:00.017+00:00
[W000 2025-07-03T13:05:00.946+00:00] Failed to reinstate database ncnfdb11. Will retry later.

$ kubectl -n oracle-database get pods
NAME                         READY   STATUS    RESTARTS   AGE
cnfdb-dgbroker-rb0po   0/1     Running   0          23m
cnfdb11-ck99a          0/1     Running   0          8m41s
cnfdb12-e8a8l          1/1     Running   0          45m

$ kubectl -n oracle-database logs -f cnfdb11-ck99a
SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted


SQL>    alter database open
*
ERROR at line 1:
ORA-16649: possible failover to another database prevents this database from
being opened


SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 10 sec(s) before restarting Oracle processes and opening the database
The Oracle base remains unchanged with value /opt/oracle
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
   alter database mount
ORA-1100 signalled during:    alter database mount...
   alter database open
2025-07-03T13:00:34.231335+00:00
Data Guard Broker initializing...
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
Data Guard: broker startup completed
Data Guard determines a failover has occurred - instance will not be opened
ORA-16649 signalled during:    alter database open...

[andbos]

A better way to test automatic failover proved to be to exec in to the primary SIDB pod, then log in to the SIDB using SQLplus and finally executing "shutdown abort". Automatic failover was triggered successfully and also reinstation of new standby succeeded.

kubectl -n oracle-database logs -f cnfdb-dgbroker-iyecf
The following output is now a tail of the observer.log:
Observer 'cnfdb-dgbroker' started
[W000 2025-07-04T11:55:27.836+00:00] Observer trace level is set to USER
[W000 2025-07-04T11:55:27.836+00:00] Try to connect to the primary.
[W000 2025-07-04T11:55:27.836+00:00] Try to connect to the primary cnfdb11:1521/ncnfdb11.
[W000 2025-07-04T11:55:27.846+00:00] The standby ncnfdb12 is ready to be a FSFO target
[W000 2025-07-04T11:55:28.846+00:00] Connection to the primary restored!
[W000 2025-07-04T11:55:30.846+00:00] Disconnecting from database cnfdb11:1521/ncnfdb11.
[W000 2025-07-04T12:06:45.801+00:00] Primary database cannot be reached.
[W000 2025-07-04T12:06:45.802+00:00] Fast-Start Failover threshold has not exceeded. Retry for the next 30 seconds
[W000 2025-07-04T12:06:46.802+00:00] Try to connect to the primary.
[W000 2025-07-04T12:06:47.854+00:00] Primary database cannot be reached.
[W000 2025-07-04T12:06:48.854+00:00] Try to connect to the primary.
[W000 2025-07-04T12:07:12.477+00:00] Primary database cannot be reached.
[W000 2025-07-04T12:07:12.477+00:00] Fast-Start Failover threshold has not exceeded. Retry for the next 3 seconds
[W000 2025-07-04T12:07:13.477+00:00] Try to connect to the primary.
[W000 2025-07-04T12:07:14.525+00:00] Primary database cannot be reached.
[W000 2025-07-04T12:07:14.525+00:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second
[W000 2025-07-04T12:07:15.526+00:00] Try to connect to the primary.
[W000 2025-07-04T12:07:16.573+00:00] Primary database cannot be reached.
[W000 2025-07-04T12:07:16.573+00:00] Fast-Start Failover threshold has expired.
[W000 2025-07-04T12:07:16.573+00:00] Try to connect to the standby.
[W000 2025-07-04T12:07:16.573+00:00] Check if the standby is ready for failover.
[S002 2025-07-04T12:07:16.586+00:00] Fast-Start Failover started...

2025-07-04T12:07:16.586+00:00
Initiating Fast-Start Failover to database "ncnfdb12"...
[S002 2025-07-04T12:07:16.586+00:00] Initiating Fast-start Failover.
Performing failover NOW, please wait...
Failover succeeded, new primary is "ncnfdb12"
2025-07-04T12:07:36.769+00:00
[S002 2025-07-04T12:07:36.769+00:00] Fast-Start Failover finished...
[W000 2025-07-04T12:07:36.769+00:00] Failover succeeded. Restart pinging.
[W000 2025-07-04T12:07:36.778+00:00] Primary database has changed to ncnfdb12.
[W000 2025-07-04T12:07:36.780+00:00] Try to connect to the primary.
[W000 2025-07-04T12:07:36.780+00:00] Try to connect to the primary cnfdb12:1521/ncnfdb12.
[W000 2025-07-04T12:07:36.848+00:00] Connection to the primary restored!
[W000 2025-07-04T12:07:36.856+00:00] The standby ncnfdb11 needs to be reinstated
[W000 2025-07-04T12:07:36.856+00:00] Try to connect to the new standby ncnfdb11.
[S006 2025-07-04T12:07:37.128+00:00] Failed to attach to cnfdb11:1521/ncnfdb11.
ORA-12541: TNS:no listener

Unable to connect to database using cnfdb11:1521/ncnfdb11
[W000 2025-07-04T12:07:37.856+00:00] Disconnecting from database cnfdb12:1521/ncnfdb12.
[W000 2025-07-04T12:07:51.877+00:00] Try to connect to the new standby ncnfdb11.
[W000 2025-07-04T12:07:52.881+00:00] Connection to the new standby restored!
[W000 2025-07-04T12:08:10.910+00:00] Try to connect to the primary cnfdb12:1521/ncnfdb12.
[W000 2025-07-04T12:08:11.910+00:00] Connection to the primary restored!
[W000 2025-07-04T12:08:12.910+00:00] Wait for new primary to be ready to reinstate.
[W000 2025-07-04T12:08:12.915+00:00] New primary is now ready to reinstate.
[W000 2025-07-04T12:08:13.915+00:00] Issuing REINSTATE command.

2025-07-04T12:08:13.916+00:00
Initiating reinstatement for database "ncnfdb11"...
Reinstating database "ncnfdb11", please wait...
[W000 2025-07-04T12:08:36.955+00:00] The standby ncnfdb11 is ready to be a FSFO target
Reinstatement of database "ncnfdb11" succeeded
2025-07-04T12:08:48.433+00:00
[W000 2025-07-04T12:08:48.976+00:00] Successfully reinstated database ncnfdb11.