[tests] linkcheck: bind each test HTTP server to a unique port per-testcase.

[jayaddison]

Feature or Bugfix

• Refactoring

Purpose

• Reduce the possibility that linkcheck builder tests can interfere with each other.
• Allow the linkcheck builder tests to run in arbitrary order.

Detail

• Delegate network port binding to Python by requesting port number 0, as illustrated in the Python documentation's socketserver sample code for ThreadingMixIn.
• Return a localhost:{port} address for each test that it can make linkcheck builder requests to.
• Dynamically rewrites source hyperlinks that refer to the address localhost:7777 during linkcheck tests to remap these to the test-specific HTTP(S) server.
• Removes the filelock dependency from our test dependencies since as this should no longer be required with the above changes in place.

Relates

• Relates to [BUG] Tests have global side-effects. #11285.
• Resolves [tests] linkcheck builder unit tests fail sporadically when run using unpredictable orderings. #12122.

[picnixz]

Sometimes I want to use a specific port (either because I don't trust Python or my OS, or I want to check that it fails because I know the port is busy) so I'd suggest this.

I'm sorry I didn't suggest this before by the way.

[jayaddison]

Stick to common test ports (xxxx for x = 5,6,7,8) possibly with xxx(x+1) and xxx(x-1) but don't use other ports.

@picnixz going back to check that I haven't overlooked things: was there a compelling reason to prefer using only recognizable-as-maybe-test port numbers like these? I think the port 0 dynamic allocation is a neat solution, but it does conflict with that numbering scheme suggestion.

[picnixz]

was there a compelling reason to prefer using only recognizable-as-maybe-test port numbers like these

Not really. It's just that I think it would have been easier to debug in general.

[jayaddison]

was there a compelling reason to prefer using only recognizable-as-maybe-test port numbers like these

Not really. It's just that I think it would have been easier to debug in general.

In situations where an individual test is being troublesome, and repeating it and perhaps inspecting the network requests to try to debug that, this seems very true.

Perhaps an environment variable (TEST_HTTP_PORT) could override the default port number when set? It's a small amount of additional complexity, but would allow use of a static port number without changing the codebase locally.

[picnixz]

TEST_HTTP_PORT

Not really, if you have a port issue, you would just explicitly use the port (that's why I asked you that you could explicitly change the port number if needed). We could consider it in another PR if we see that we need it.

By the way, I'm updating the branch, waiting for everything to work and then LGTM.

[picnixz]

Thank you !

[jayaddison]

Thank you @picnixz for the code reviews and suggestions 👍