quic interop test failures

[nhorman]

quic interop test are failing in ci in overnight runs. Need to investigate and fix

See our latest overnight run for an example:
https://github.com/openssl/openssl/actions/runs/14962985597

This started happening recently, so a bisect might be called for here to identify what introduced the problem.

[nhorman]

@jogme some notes for you on this one:

• The interop runner tests are run using the test harness found here: https://github.com/quic-interop/quic-interop-runner
• You can see how we run these tests by examining the workflow in the openssl repo, under .github/workflows/run_quic_interop.yml
• Note that, because the upstream project is currently not accepting new implementations, you will have to modify the implementations.json file to include our client container using the following commands:

jq '.openssl = { image: "quay.io/openssl-ci/openssl-quic-interop"
                         , url: "https://github.com/openssl/openssl"
                         , role: "both"
                         }' ./implementations.json > ./implementations.tmp
          mv ./implementations.tmp implementations.json

• Note that docker will not pull a container for a remote repository if you have it built and tagged locally. You can do so by following the instructions in test/quic-openssl-docker/README.md
• The process for building locally can be simplified somewhat by modifying the Docker file as follows:

diff --git a/test/quic-openssl-docker/Dockerfile b/test/quic-openssl-docker/Dockerfile
index 348f43673d..280ee29707 100644
--- a/test/quic-openssl-docker/Dockerfile
+++ b/test/quic-openssl-docker/Dockerfile
@@ -26,8 +26,9 @@ RUN git clone --depth 1 https://github.com/ngtcp2/nghttp3.git && \
     rm -rf /nghttp3
 
 # download and build openssl 
-RUN git clone --depth 1 -b $OPENSSL_BRANCH $OPENSSL_URL && \
-    cd openssl && \
+RUN mkdir /openssl
+COPY . /openssl
+RUN /openssl && \
     ./Configure enable-sslkeylog enable-fips enable-demos enable-h3demo enable-hqinterop disable-docs --prefix=/usr --openssldir=/etc/pki/tls && \
     make -j 4 && make install && cp test/quic-openssl-docker/hq-interop/quic-hq-interop /usr/local/bin && \
     cp test/quic-openssl-docker/hq-interop/quic-hq-interop-server /usr/local/bin && \
@@ -43,7 +44,7 @@ RUN git clone --depth 1 https://github.com/curl/curl.git && \
     rm -rf /curl
 
 # copy run script and run it
-COPY run_endpoint.sh .
+COPY test/quic-openssl-docker/run_endpoint.sh .
 RUN chmod +x run_endpoint.sh

and building with the command:

docker build -t quay.io/openssl-ci/openssl-quic-interop:latest -f ./test/quic-openssl-docker/Dockerfile .

This will allow you to build the openssl container from your local tree sources rather than pulling from a remote branch, which accelerates your build/test cycle

• The goal I think, once you get the problem recreated locally, is to modify the openssl sources to gain better visibility on the problem(s) using printf/other debug techniques and then implementing and testing a fix.

• Please let me know if you have questions.

[jogme]

Putting my notes here from initial investigation:

• The tests are flaky (at least locally)

• The client does not send the "WHAT" version packet and sends version "1" right away. (But this is also true for success in the pcap files, only the server logs shows the WHAT packet) The server does not respond so the client keeps sending PING packets

• The behavior ^ shows up interesting in the pcap files. The client side does not have any QUIC protocol packets in it, but the server side shows this behavior. When the tests are successful the connection shows up in both of the pcap files.

• The Initial packets are send, but at failure the server does not respond.

• Both has DCID in the packets and no SCID

• DCID looks to be properly randomized

This kind of behavior shows up on all of the tests.

I tried to increase the delay on the client startup, but did NOT solve the problem.