mysterious failures and IPv6 problems for macOS GitHub actions jobs

[bagder]

I did this

Over the last several days, CI jobs running on GitHub actions have turned flaky. Sometimes one of the macOS jobs fails, sometimes 10 of them fail. Rerunning the same code again may result in a different outcome.

The failures seem IPv6 related somehow. example job, another example

The most obvious symptom seems to be that when it has problems, it can't "resolve" ::1. It can be noted that for macOS we call getaddrinfo even on plain numerical IP addresses

failures

When this problem happens, usually the following test cases fail: 1085 1400 1401 1402 1403 1404 1405 1406 1407 1420 1465 1467 1468 2100

Looking at test logs in a failed run , we can see that all HTTP-IPv6 tests are skipped because curl cannot verify the IPv6 server:

2024-04-02T19:43:36.1057020Z  * Added connection 0. The cache now contains 1 members
2024-04-02T19:43:36.1057900Z  * STATE: CONNECT => RESOLVING handle 0x7fc26980be08; line 1954
2024-04-02T19:43:36.1058690Z  * Could not resolve host: ::1

test 1085

Tries to use an IPv6 address for --interface but the resolving of the host (::1) fails, which makes the test return an unexpected return code.

test 1400-1468

These tests verify --libcurl - outputting generated libcurl using source code for a curl command line.

These tests mysteriously get an extra line of libcurl code added:

curl_easy_setopt(hnd, CURLOPT_IPRESOLVE, 1L);

The only way CURLOPT_IPRESOLVE is set by the curl command line tool is when the -4/--ipv4 option is used. And it is not used in any of these tests.

test 2100

Uses DoH. When failing, the request does not ask for an IPv6 address (only IPv4) which makes the protocol check fail. Because it deems IPv6 not working?

macOS runner image

Image: macos-12
  Version: 20240329.1
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20240329.1/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20240329.1

I expected the following

The tests should just work.

curl/libcurl version

current git

operating system

macOS only

[vszakats]

image changes on 20240329:
actions/runner-images@6cb51f9

[vszakats]

Possibly related:
actions/runner-images#9626
actions/runner-images#9628

[bagder]

aaaaaah

actions/runner-images#9586

This explains a lot:

echo "Configuring curl to resolve names with IPv4..."
echo '--ipv4' >> ~/.curlrc

[bagder]

What puzzles me is how that was not always causing those tests to fail...

[erik-bershel]

What puzzles me is how that was not always causing those tests to fail...

Do green tests shipped on updated images? It takes some time to update all the agents in prod. So you should have seen something like an ever-increasing number of errors until all new jobs only hit the updated agent with the curl config. If they took place on updated agents, then this would be a very interesting case.

[bagder]

Do green tests shipped on updated images?

Yes, that's exactly it - or rather it still worked fine on the old images. I went back to check a few of the older PRs and you are right. The green builds are still Version: 20240218.1. All the builds on the new image fail. It all makes sense now.