Test Version deployment

[anrossi]

Describe the bug

Version negotiation supports a complicated set of steps to deploy changes to the supported versions list of a fleet of servers. MsQuic provides the means to accomplish this, but there's no testing on the process.

Add a test against two servers behind a load balancer, and have the client connect while a version deployment is in progress and ensure the connection succeeds.
Given the timing-sensitive nature of this test, it should run in a loop multiple times, with multiple clients connecting concurrently to ensure the best coverage.

Affected OS

• All
• Windows Server 2022
• Windows 11
• Windows Insider Preview (specify affected build below)
• Ubuntu
• Debian
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

Run tests.

Expected behavior

Test should exist for this scenario.

Actual outcome

No tests available for this scenario.

Additional details

No response

[nibanks]

Also look into using LoadBalancedServer to test this in a virtual environment.

[anrossi]

I don't think this is going to be possible without some significant product code changes which makes it even more necessary to write tests.

The basic assumption of the server-side version settings is that they are global, since, at the time I thought that we'd only ever have fleet customers with a single MsQuic server instance per machine, and that still seems to be the case.

In order to write a single-machine test with the load balancer, I'd have to move the version settings down to the binding layer (so that stateless binding operations could still generate a VN packet), to enable different listeners in the same process to have different version settings.

This would change the behavior of the version settings API too: setting it globally could still work (I think, because the settings are copied from global to config, and from config to listener/connection), but then it would also be possible to set on the config/listener and have it applied down to the binding layer

Design notes:
Test cases for load balancer:

• add new version
  • test before adding
    • handshakes in a loop ensuring the version is expected
  • test during/after adding to 1 server
    • handshakes in a loop ensuring clients which support new version use it, and clients that don't, use the old version
  • test during/after adding to both servers
    • handshakes in a loop ensuring clients which support new version use it, and clients that don't, use the old version
• remove one of two versions
  • test before removal
    • handshakes in a loop ensuring clients are using both versions (one client for to-be-removed version, one for not)
  • test during/after removing from 1 server
    • handshakes in a loop ensuring clients using removed version sometimes fail, and clients not using removed version don't fail EVER.
  • test during/after removing from both servers
    • handshakes in a loop ensuring clients using removed version ALWAYS fail, and clients not using removed version don't fail EVER.

Implementation details:

• 2 threads: each thread is creating new clients as soon as a client reaches CONNECTED state (the clients in CONNECTED state immediately shutdown) in a loop. This should result in at least 2 handshakes active at any time.
• The clients will need a "timestamp" to know whether they were created before/after the version settings change timestamp. This is necessary for the validation logic to know which version(s) this client is expected to have.
• The version change will need a timestamp as well to know when it takes effect.

[anrossi]

An alternative would be to write the test with separate processes: a load balancer running as a separate process with a RAW socket to hand traffic to the appropriate server processes. The server processes then sending traffic back to the load balancer.