Two test servers, one lock

[dfandrich]

The test servers use a single file serverlogs.lock to indicate when a test server is still writing its logs to prevent the test harness from looking at them too soon. However, many tests use more than one server running in parallel (e.g. http and http-proxy) and the first one to finish clears the lock for both of them. This means that the test harness can look at the logs before they are all ready. I believe this is what causes many of the flaky test failures we often see in the CI tests.

For example, in one particular test failure I looked at in test 1904, the proxy.input file started with the line [DISCONNECT]. This should never happen because there is no client connected when the test starts. Furthermore, test 1904 does NOT set connection-monitor without which [DISCONNECT] should not be printed. What must have happened was that a previous test set connection-monitor but the other server cleared the log lock before it had a chance to write that final message and only write it after the log files were deleted and the next test (1904) had already started.

Many other flaky test failures occur because a final QUIT command is not logged. This may also be explained if another server clears the log lock (causing the test harness to look at all the log files too soon) before that server gets a chance to write the final command.

More evidence for this is the server log line:

Error removing lock file log9/serverlogs.lock error: 2 No such file or directory

indicating that the lock file is expected to exist but was somehow deleted without the server doing it. The most likely explanation is that the other server deleted it.

One problem with this theory in the general case is that this kind of problem also happens on tests that only have a single server (e.g. ftp test 135). Some of these use sockfilt in addition to the main test server, but it doesn't appear that sockfilt engages in the log file locking scheme.

One solution to this problem is to allow multiple lock files, probably one fixed lock file per server, and have the test harness look for and wait for them all. Another is to have all servers set fcntl(1) locks on a single file, but this may not be portable enough and is probably why the file approach was chosen from the start.

[bagder]

The fcntl() method does not seem to exist on Windows for example.

Maybe the easiest (?) approach is to have all test servers create their own "lock file" in a dedicated per-runner lock-directory and have runtests.pl verify that the entire locks-directory is empty before it proceeds?

[bagder]

@dfandrich do you want to take a stab at converting over to a multi-file system or should I?

[dfandrich]

Parallel testing changes the test timing so much that this issue causes many more flaky tests. Since this is basically blocking proper use of that feature, I'll tackle this one to get that unblocked. I'll still make a PR with parallel testing available shortly since it's still useful in this state, even if imperfect.