fix: test_replicaof_reject_on_load crash on stop

[kostasrim]

resolves #2769

The problem is that we are shutting down master. During this flow, we iterate and shutdown all of the listeners and then we give a chance for the currently executing commands to finish with DispatchTracker. However, this is problematic, because internally it traverses all of the connections in all shards and sends SendCheckPoint messages. When this happens, it forces the connection that handles the replica to activate its dispatch_fiber_. If this happens before we Migrate the connection in the replication flow (dflycmd) then we will trigger the CHECK that asserts that the dispatch_fiber is not actve (it was never launched).

Unfortunately this is based on timing and therefore it requires two things:

1. Migrate might fail if dispatch_fiber is active. In that case do no crash but return false to indicate that the migration was not successful.
2. After we migrate, we might find ourselves with the socket closed (because of the shutdown signal process/flow). We need to check that the socket is open. If it's not, it means that it was shutdown by the signal flow (because after we finish with DispatchTracker, we start iterating over all of the connections in all of the shards and shut them down.

[kostasrim]

@adiholden I did not supply a fix for pytests and the crashed instance. We need to think about this because proc.kill() && proc.terminate() do not return a value so we need another chore. I will fix this in a separate PR and I created an issue here: #2819

[adiholden]

what is triggering the dispatch_fb_ creation on shutdown?

[kostasrim]

This https://github.com/dragonflydb/dragonfly/blob/main/src/facade/dragonfly_listener.cc#L301 which calls SendCheckpoint. See also: https://github.com/dragonflydb/dragonfly/blob/main/src/facade/dragonfly_listener.cc#L301 and https://github.com/dragonflydb/dragonfly/blob/main/src/facade/dragonfly_listener.cc#L455

Moreover, this is what I was telling you about migrations. In this flow, we migrate and it happens that the traversal fiber happens before the migration and so when we migrate the dispatch fiber is already started. Now, if we remove this check and cancel the migration, it could also be the case that another fiber run which killed the socket (after PreShutdown and DispatchTracker) and I have seen this being triggered and that's why I checked for the socket to be active.

Of course, it could be the case that traversal happened after migration but before the connection is relinked to the thread local and in that case we would probably deadlock or loose some important message being send from the traversal.

[adiholden]

@kostasrim can you merge this?