Database NoChange status

[olibanjoli]

How do you get the database out of this state?

• backup tasks can't be created because there are no nodes which can be selected as responsible node
• otherwise db is running fine (reading/writing data)
• single node
• runs in docker
• error appeared after db crashed a few weeks ago (container had to be restarted)

Node in rehabilitation due to last report status being 'Error'.
. System.InvalidOperationException: We have databases with 'NoChange' status, but our last report from this node is 'Timeout'
   at Raven.Server.ServerWide.Maintenance.ClusterMaintenanceSupervisor.ClusterNode.UpdateNodeReportIfNeeded(ClusterNodeStatusReport nodeReport, List`1 unchangedReports) in C:\Builds\RavenDB-Stable-5.4\54027\src\Raven.Server\ServerWide\Maintenance\ClusterMaintenanceSupervisor.cs:line 356
   at Raven.Server.ServerWide.Maintenance.ClusterMaintenanceSupervisor.ClusterNode.ListenToMaintenanceWorker() in C:\Builds\RavenDB-Stable-5.4\54027\src\Raven.Server\ServerWide\Maintenance\ClusterMaintenanceSupervisor.cs:line 352

[ayende]

What is your cluster configuration? Are you running on a single node?
Does that persist after a restart?

[olibanjoli]

it's single node and the error is still there after a restart (restart of container)

[georgiosd]

I am having the same problem, but with a connection refused error.

I managed to resolve it by adding a 127.0.0.1 <public domain> in /etc/hosts in the container but I am not quite sure why it would be needed.

Any ideas what causes this? Or a more elegant solution?

Doesn't Raven detect that it's trying to send to itself (A => A) and use a local binding instead of the public URL?

[ayende]

No, we always use the URL that the client will try to use. The reason for that is that it ensures that the server will have the same connectivity as the client.

[georgiosd]

Okay, but it seems to create a problem in this instance. You think it's down to the host networking configuration or?

[ayende]

That usually indicates that the network is setup in such a way that the server cannot talk to itself.
Maybe the hostname changed ? Or some firewall rules via the public IP?

[georgiosd]

Neither is true in my case, I did check the firewall. Does Raven do any "sender" related checks when receiving requests? I did get different behavior on the port when trying from different levels (inside container, outside, outside the host).

I guess I'll have to do a traceroute as well

[olibanjoli]

problem was ufw on the node messing up with docker-compose. using an external firewall now and turned ufw off. all good.