After a certificate expiration, no possibility to get out of the situation

[pmengal]

Hello,

We have a cluster of two, both on linux (A+B). For some reason, one of the two (B) did not renew the certificate that eventually expired. Nobody noticed until recently. The issue is that we can't get out of it. B refuse to accept the new certificate and give this error:

Exception of type 'Raven.Client.Exceptions.Cluster.NoLeaderException' was thrown.

Even worse, A is refusing to do manual backup. So we can't migrate to a new instance. No error is provided. The files are just not where we specified they should go.

We have been looking for 2 days now and we are desperate to find a solution. Can anyone point us to the good solution?

[mcrio]

Maybe the problem is that you are in a cluster of 2 where there is no way to achieve a majority decision. You should have a odd number of nodes in the cluster because of the majority vote. Someone from RavenDb will know better.

[pmengal]

Yes I forgot to mention that I have no way to add a third node...

[georgiosd]

You can just replace the certificate on the disk and restart the service, no?

…

On Sat, 2 Dec 2023 at 17:43, pmengal ***@***.***> wrote: Yes I forgot to mention that I have no way to add a third node... — Reply to this email directly, view it on GitHub <#17802 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMYG5R2PG5CKXFEU7MXLMTYHNLC5AVCNFSM6AAAAABAEDR342VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TOMZYHEZTG> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ayende]

I assume that in this case, both nodes are using different certificates?

At any rate, there are explicit manual ways to resolve this.
You can update the certificate on the disk, and either restart the B node or wait up to 30 minutes for it to pick up and use the new certificate.

Beyond that, there in the admin channel as well, which can be used. See the discussion here:
https://ravendb.net/learn/inside-ravendb-book/reader/4.0/18-operational-recipes

[pmengal]

Yes that's right. B node is manual and also in a docker image. That's was an experiment that we forgot about and the dev that did it doesn't work for us anymore.

What happens if that image/machine goes down forever? No way to recover the situation?

The most troubling thing is the impossibility to actually backup the database, scratch everything, and start over. That's what we finally decided to do, but with custom code that fetched all the document (we had only one collection there) and then bulkinsert them back in some new RavenDb Cloud cluster so save our weekend :)

[16:35:25 INFO] Saving the backup status in the cluster
[16:35:39 ERROR] Failed to save the backup status in the cluster
[16:35:39 ERROR] Error saving the periodic backup status
System.TimeoutException: Could not send command Raven.Server.ServerWide.Commands.PeriodicBackup.UpdatePeriodicBackupStatusCommand from A to leader because there is no leader, and we timed out waiting for one after 00:00:15
---> System.TimeoutException: Could not send command Raven.Server.ServerWide.Commands.PeriodicBackup.UpdatePeriodicBackupStatusCommand from A to leader because there is no leader, and we timed out waiting for one after 00:00:15
at Raven.Server.ServerWide.ServerStore.ThrowTimeoutException(CommandBase cmd, Exception requestException) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 3089
at Raven.Server.ServerWide.ServerStore.SendToLeaderAsyncInternal(TransactionOperationContext context, CommandBase cmd) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 3078
--- End of inner exception stack trace ---
at Raven.Server.ServerWide.ServerStore.ThrowTimeoutException(CommandBase cmd, Exception requestException) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 3089
at Raven.Server.ServerWide.ServerStore.SendToLeaderAsyncInternal(TransactionOperationContext context, CommandBase cmd) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 3078
at Raven.Server.ServerWide.ServerStore.SendToLeaderAsyncInternal(CommandBase cmd) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 3015
at Raven.Server.ServerWide.ServerStore.SendToLeaderAsync(CommandBase cmd) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\ServerWide\ServerStore.cs:line 2849
at Raven.Server.Utils.BackupUtils.<>c__DisplayClass12_0.<b__2>d.MoveNext() in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\Utils\BackupUtils.cs:line 444
--- End of stack trace from previous location ---
at Raven.Server.Documents.PeriodicBackup.BackupHelper.RunWithRetriesAsync(Int32 maxRetries, Func1 action, String infoMessage, String errorMessage, SmugglerResult smugglerResult, Action1 onProgress, OperationCancelToken operationCancelToken) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\Documents\PeriodicBackup\BackupHelper.cs:line 74
at Raven.Server.Documents.PeriodicBackup.BackupHelper.RunWithRetriesAsync(Int32 maxRetries, Func1 action, String infoMessage, String errorMessage, SmugglerResult smugglerResult, Action1 onProgress, OperationCancelToken operationCancelToken) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\Documents\PeriodicBackup\BackupHelper.cs:line 74
at Raven.Server.Utils.BackupUtils.<>c__DisplayClass12_0.<b__0>d.MoveNext() in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\Utils\BackupUtils.cs:line 440
--- End of stack trace from previous location ---
at Raven.Client.Util.AsyncHelpers.RunSync(Func1 task) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Client\Util\AsyncHelpers.cs:line 81 at Raven.Server.Utils.BackupUtils.SaveBackupStatus(PeriodicBackupStatus status, String databaseName, ServerStore serverStore, Logger logger, BackupResult backupResult, Action1 onProgress, OperationCancelToken operationCancelToken) in C:\Builds\RavenDB-Stable-5.4\54049\src\Raven.Server\Utils\BackupUtils.cs:line 438

[ayende]

You can do a one time backup, for sure. You cannot define a new periodic backup without a cluster.

There are also emergency operations mode, where you can change the cluster to be a single node directly.

return server.ServerStore.Engine.HardResetToNewCluster();

See the details in the operation recipes section in the book

[pmengal]

Thanks Oren, hope you are safe where you are.

FYI the error I copied is from one time backup using button in Tasks > Backups > Manual Backup > [Create Backup]

[ayende]

You are correct, I created an issue for that here. Note that the backup itself completes successfully, but errors on trying to set its state.

https://issues.hibernatingrhinos.com/issue/RavenDB-21827/Manual-backup-fails-when-the-cluster-is-not-available