Swap order of starting up RELAY/SOCKET processes in multi-master tests

[mistydemeo]

The tests should fail here since they're missing the fix from #65. cc @aroben

[aroben]

Hm, I put the tests in this order in 96b6712 on purpose. Maybe we need a second set of tests that boots the servers in the other order?

[mistydemeo]

After looking at it more in #65 (comment), we probably want to make sure we're checking the exit status/stdout of both processes? Sorry, that's exactly what the commit you linked addressed. So, yeah, we might want to add extra tests to cover both cases.

[aroben]

The only tricky thing with checking the exit code of both processes is that you don't know which master will run the failing test. If the central master runs the failing test, then the remote master will exit 0. If the remote master runs the failing test, the remote master will exit 1.

[mistydemeo]

Modified to test in both orders.

[aroben]

Did you mean to leave this debug output in?

[aroben]

I think this will sometimes fail due to the issue I mentioned. There's no guarantee that the remote master will run the failing test. If it doesn't, its exit code will be 0 (and it won't contain any failure output). We need a way to ensure that the right master fails. Something like this might work inside a test:

def test_foo
  whoami =
    if ENV["TEST_QUEUE_SOCKET"]
      "central"
    elsif ENV["TEST_QUEUE_RELAY"]
      "remote"
    end
  refute ENV["MULTI_MASTER_FAIL"] == whoami
end

I.e., if you set MULTI_MASTER_FAIL=central, the central master will fail. If you set MULTI_MASTER_FAIL=remote, the remote master will fail. If you put this new test inside the MiniTestSleep suites then it will be highly likely to run in both masters and fail or pass appropriately.

[mistydemeo]

Ben suggested an alternate approach, sleeping the order of the masters to induce the desired one to pick up the test. Pushed that version.

[aroben]

I don't think this will guarantee that a particular master's workers will run the failing suite. around_filter gets called inside each worker just after receiving the suite to run, so it seems like workers will be handed an initial suite to run and then will sleep. I.e., imagine the following execution when we're trying to get the remote master to run the failing suite:

1. SLEEP_AS_MASTER=1
2. Central master boots and spawns workers
3. Central master hands the failing suite to one of its own workers
4. Central master's workers sleep
5. Remote master boots and connects to central master
6. Central master hands all the rest of the suites to the remote master's workers
7. Remote master's workers run those suites and remote master exits with success
8. Central master's workers wake up and run their suites, including the failing one

For the sleeping to be effective I think it needs to happen before suites are assigned, i.e., before the workers send their first POP messages.

[bhuga]

For the sleeping to be effective I think it needs to happen before suites are assigned, i.e., before the workers send their first POP messages.

Nuts, this makes total sense. My bad.

Do we need to make changes to the runner itself to give us a place to inject this behavior for testing?

[aroben]

Putting the sleep in Runner#after_fork should do the trick I think.

[mistydemeo]

Good point. 😅 Swapped.

[aroben]

This is great. Thank you!

[aroben]

CI is red, but that is what we want. #65 will fix it.