Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GHA: add MSYS, mingw-w64, Cygwin jobs #13599

Closed
wants to merge 51 commits into from

Conversation

vszakats
Copy link
Member

@vszakats vszakats commented May 11, 2024

  • re-implement autotools MSYS and Cygwin AppVeyor jobs in GHA.
    Now build with SSL and PSL to improve test coverage.
  • re-implement MSYS2 mingw-w64 gcc 13 AppVeyor job in GHA.
    CMake, mingw-w64, gcc 13, Debug, x64, Schannel, Static, Unicode
  • add new cmake Cygwin job (build-only).
  • enable -j14 parallelism when running tests.
  • delete the 5 migrated jobs from AppVeyor CI.
  • add 2 build-only mingw-w64 builds, gcc Release and clang OpenSSL.
  • also enable brotli, libssh2, nghttp2 for more test coverage.

These jobs offer better performance, more flexibility and
parallelization compared to the AppVeyor ones they replace. It also
offloads AppVeyor, allowing to iterate faster. They also appear more
reliable than e.g. Azure Windows jobs, where runners are prone to fail
1.

Closes #13599


  • Also migrate the mingw-w64 gcc 13 AppVeyor job?
  • Delete the 5 AppVeyor CI jobs if these ones work as expected?
  • Test mingw-w64 in Release CMake configuration.
  • Depends on lib: fix compiler warnings (gcc) #13643.

Footnotes

  1. Exit code 143 returned from process: file name 'C:\Windows\system32\docker.EXE', arguments 'exec -i 6b13a669c6dfe7fb9f59414369872fd64d61c7182f880c3d39c135cb4c115c8f C:\__a\externals\node\bin\node.exe C:\__w\_temp\containerHandlerInvoker.js'.

@vszakats vszakats added the CI Continuous Integration label May 11, 2024
@vszakats vszakats marked this pull request as draft May 11, 2024 15:18
@vszakats

This comment was marked as resolved.

@vszakats vszakats added Windows Windows-specific tests labels May 11, 2024
@vszakats vszakats changed the title GHA: add msys2 and cygwin jobs GHA: add MSYS2 and Cygwin jobs May 11, 2024
@vszakats vszakats changed the title GHA: add MSYS2 and Cygwin jobs GHA: add MSYS and Cygwin jobs May 11, 2024
@vszakats vszakats marked this pull request as ready for review May 11, 2024 17:25
@vszakats
Copy link
Member Author

vszakats commented May 11, 2024

Runs OK now, has OpenSSL + libpsl (plus a bunch of others it detected automaticall). Tests pass.
Runtimes compared to AppVeyor jobs (no SSL) (tested on a weekend):

                                           AppVeyor GHA
                                           no SSL¹  OpenSSL
                                                    default -j2 #1  -j2 #2 -j3    -j4    -j5    -j6    -j8    -j12   -j14³  -j16
                                           -------- ------- ------  ------ ------ ------ ------ ------ ------ ------ ------ ------
autotools, cygwin, Debug, x86_64           53m25s   26m25s  22m21s  21m57s 20m08s 20m21s 19m40s 19m12s 18m39s 18m08s 32m18s 18m17s
autotools, msys2, Debug, x86_64, no Proxy  38m30s   22m28s  18m46s  18m16s 17m04s 16m10s 16m03s 15m47s 15m21s 15m38s 22m46s 14m59s
autotools, msys2, Debug, x86_64            38m45s   23m48s  19m16s  19m10s 18m06s 16m43s 16m18s 16m01s 16m01s 15m25s 24m06s 15m20s
autotools, msys2, Release, x86_64          35m57s   18m29s  14m25s² 14m19s 13m05s 12m35s 11m33s 11m35s 11m16s 10m59s 18m49s 11m04s
cmake, mingw-w64, gcc 13, Debug, x64       34m39s                                                                    19m31s

¹ https://ci.appveyor.com/project/curlorg/curl/builds/49794352
² TESTFAIL: These test cases failed: 1542 https://github.com/curl/curl/actions/runs/9052581559/job/24870416402?pr=13609#step:5:7441
³ on a workday

edit: Is the 'no proxy' variant adding much value to be worth keeping it? Or perhaps more variation could be introduced to make it valuable?

edit: added parallel test results. (#13609)

@vszakats vszakats force-pushed the gha-windows-msys-cygwin branch 4 times, most recently from 4a1bc58 to b553c13 Compare May 13, 2024 08:55
@vszakats vszakats changed the title GHA: add MSYS and Cygwin jobs GHA: add MSYS, mingw-w64, Cygwin jobs May 13, 2024
@vszakats vszakats force-pushed the gha-windows-msys-cygwin branch 2 times, most recently from 62a789e to cb2025f Compare May 15, 2024 12:59
@vszakats
Copy link
Member Author

Last minute I enabled WebSockets in the mingw-w64 job and 2301 and 2305 started failing right away:
https://github.com/curl/curl/actions/runs/9133449443/job/25116951463#step:14:14502
https://github.com/curl/curl/actions/runs/9133227792/job/25116241601#step:14:18246

It's also the first time WebSocket is tested on native Windows in CI.
With MSYS and Cygwin I haven't seen this happening.

Ignoring these results for now as a workaround.

@vszakats vszakats closed this in 36fd2dd May 17, 2024
@vszakats vszakats deleted the gha-windows-msys-cygwin branch May 17, 2024 22:43
vszakats added a commit to vszakats/curl that referenced this pull request May 18, 2024
vszakats added a commit that referenced this pull request May 18, 2024
@vszakats
Copy link
Member Author

vszakats commented May 19, 2024

Post merge update: mingw-w64 (and sometimes also MSVC) tests seem to hang, which is new compared to AppVeyor CI and wasn't typical while testing this.

Is there a way to tell which test is hanging here?:

  1. https://github.com/curl/curl/actions/runs/9150447926/job/25155235098?pr=13694 (msys2, mingw-w64) MQTT
  2. https://github.com/curl/curl/actions/runs/9150573943/job/25155515772#step:14:3015 (msys2, mingw-w64) WebSockets
  3. https://github.com/curl/curl/actions/runs/9160634785/job/25183743521?pr=13718#step:14:3663 (msys2, mingw-w64) IMAP?
  4. https://github.com/curl/curl/actions/runs/9180118121/job/25243819164?pr=13718 (msys2, mingw-w64) MQTT
  5. https://github.com/curl/curl/actions/runs/9221334828/job/25370173056?pr=13606 (msys2, mingw-64, runner hang?)
  6. https://github.com/curl/curl/actions/runs/9229566918/job/25396055324?pr=13759#step:14:2837 (msys2, mingw-w64) WebSockets
  7. https://github.com/curl/curl/actions/runs/9233820883/job/25406708621?pr=13766#step:10:17395 (old-mingw-w64, 7.3.0) TFTP
  8. https://github.com/curl/curl/actions/runs/9246239130/job/25433558648?pr=13785#step:14:2840 (msys2, mingw-w64) WebSockets
  9. https://github.com/curl/curl/actions/runs/9253290860/job/25452674387?pr=13766#step:10:17226 (old-mingw-w64, 7.3.0) TFTP
  10. https://github.com/curl/curl/actions/runs/9260300300/job/25473819770?pr=13718#step:10:17518 (old-mingw-w64, 7.3.0) FTP
    (disabled TFTP, MQTT and WebSockets jobs at this point)
  11. https://github.com/curl/curl/actions/runs/9256858127/job/25463785898#step:6:18433 (MSVC) TFTP
  12. https://github.com/curl/curl/actions/runs/9346162475/job/25720437944?pr=13855#step:14:2838 (msys2 (cmake, mingw64, x86_64)) (test3006 SMTP maybe?)
  13. https://github.com/curl/curl/actions/runs/9758011305/job/26931678639?pr=14084#step:14:2834 (msys2 (cmake, mingw64, x86_64)) (test3005 SMTP maybe?)
  14. https://github.com/curl/curl/actions/runs/9380565803/job/25827882253#step:5:1450 (MSVC hang in "cmake build tests" step, last log message: "Building Custom Rule D:/a/curl/curl/tests/CMakeLists.txt")
  15. https://github.com/curl/curl/actions/runs/9384449682/job/25840281198?pr=13896#step:8:1442 (old-mingw-w64 (cmake, 7.3.0-x86_64)), completed 'build tests', then hang before starting up 'run tests' step.
  16. https://github.com/curl/curl/actions/runs/9775422731/job/26985795679 (old-mingw-w64, 9.5.0) location unknown, last executed was test2600 with a failure
  17. https://github.com/curl/curl/actions/runs/9788066896/job/27025540999?pr=14097#step:14:2878 (msys2, mingw-w64, x86_64) last executed: test3008. (this run has !SMTP TFLAGS already)
  18. https://github.com/curl/curl/actions/runs/9793324277/job/27041042967?pr=14100#step:12:4012 (msvc, no ssl) last test in log: 3205.
  19. https://github.com/curl/curl/actions/runs/9793324277/job/27041042764?pr=14100 (old-mingw-w64 7.3.0) 'cmake run tests' step failed to start.
  20. https://github.com/curl/curl/actions/runs/9796121067/job/27049861121?pr=14096#step:14:2887 (msys2 cmake, mingw-w64, x86_64) last test in log: 2603
  21. https://github.com/curl/curl/actions/runs/9830189344/job/27136088802?pr=14116#step:13:3178 (old-mingw-w64 9.5.0)
  22. https://github.com/curl/curl/actions/runs/9831904667/job/27139950536#step:9:5148 (msvc, libressl) last test: 2600
  23. https://github.com/curl/curl/actions/runs/9837611706/job/27155835549?pr=14117#step:9:5156 (msvc, boringssl) last test: 2600
  24. https://github.com/curl/curl/actions/runs/9841853460/job/27169458913?pr=14125#step:13:5265 (msvc, boringssl) last test: 2600
  25. https://github.com/curl/curl/actions/runs/9841771973/job/27169174607?pr=14124 (mingw, CM x86_64 schannel U) 'cmake run tests' step failed to start.
  26. https://github.com/curl/curl/actions/runs/9843527144/job/27174924074?pr=14129#step:23:3322 (mingw, CM x86_64 schannel U) last test: 2600
  27. https://github.com/curl/curl/actions/runs/9843961100/job/27176348727?pr=14097#step:23:3323 (mingw, CM x86_64 schannel U) last test: 2600
  28. https://github.com/curl/curl/actions/runs/9857526570/job/27216813548?pr=14137#step:13:5265 (msvc, boringssl) last test: 2600
  29. https://github.com/curl/curl/actions/runs/9918753161/attempts/1?pr=14174 (4 jobs) all test 2600
    --- disabled test 2600 as non-Windows compatible
  30. https://github.com/curl/curl/actions/runs/9955276247/job/27502681202?pr=14196 (msvc, boringssl) timeout

Other, non-hang, failures seen:

msys2 (autotools, msys, x86_64), HTTP:

TESTFAIL: These test cases failed: 1542

https://github.com/vszakats/curl/actions/runs/9271072193/job/25505602121#step:10:3107

msys2 (autotools, msys, x86_64), FTP:

TESTFAIL: These test cases failed: 130 138 520 588 1044 1103 1149 1389 1449 3027

https://github.com/curl/curl/actions/runs/9156257909/job/25170280392?pr=13717#step:10:3280

msys2 (autotools, msys, x86_64), FTP:

TESTFAIL: These test cases failed: 109 140 215 402 532 983 1103 1227 1387 1449 2001

Ref: https://github.com/curl/curl/actions/runs/9380565803/job/25827888259#step:10:3939

old-mingw-w64 7.3.0, FTP:

TESTFAIL: These test cases failed: 2083

https://github.com/curl/curl/actions/runs/9318039397/job/25649583674?pr=13838#step:10:4043
https://github.com/curl/curl/actions/runs/9297899553/job/25588833035#step:10:4024

old-mingw-w64 7.3.0, multi:

TESTFAIL: These test cases failed: 1557

https://github.com/curl/curl/actions/runs/9497439177/job/26174014283?pr=13936#step:9:3527

msvc (x64, windows, -DENABLE_DEBUG=ON -DCURL_USE_SCHANNEL=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_UNICODE=OFF), FTP:

TESTFAIL: These test cases failed: 1501

https://github.com/curl/curl/actions/runs/9366928852/job/25785522746?pr=13877#step:6:5006

old-mingw-w64 (cmake, 7.3.0-x86_64), FTP

TESTFAIL: These test cases failed: 2001

https://github.com/curl/curl/actions/runs/9384449682/job/25845487515#step:9:4724
https://github.com/curl/curl/actions/runs/9399096155/job/25885854456?pr=13897#step:9:4653

old-mingw-w64 (cmake, 7.3.0-x86_64), FTP

TESTFAIL: These test cases failed: 2039

https://github.com/curl/curl/actions/runs/9455592386/job/26045733061#step:9:4170
https://github.com/curl/curl/actions/runs/9752279449/job/26915502708?pr=14059#step:9:4119

msvc (x64, windows):

TESTDONE: 1342 tests out of 1343 reported OK: 99%
  Building Custom Rule D:/a/curl/curl/tests/CMakeLists.txt
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'D:\a\curl\curl\bld\CMakeFiles\621f80ddbb0fa48179f056ca77842ff0\test-ci.rule;D:\a\curl\curl\tests\CMakeLists.txt' exited with code -1. [D:\a\curl\curl\bld\tests\test-ci.vcxproj]
Error: Process completed with exit code 1.

Ref: https://github.com/curl/curl/actions/runs/9390969492/job/25862038785#step:6:4557

msvc (x64, windows, openssl), FTP

TESTFAIL: These test cases failed: 593

Ref: https://github.com/curl/curl/actions/runs/9749983645/job/26908366075?pr=14078#step:12:5453

msvc (x64, windows, schannel), FTP

TESTFAIL: These test cases failed: 593

Ref: https://github.com/curl/curl/actions/runs/9826913304/job/27129584190?pr=14077#step:12:5631

msvc, CM x64-windows schannel U:

TESTDONE: 1297 tests out of 1300 reported OK: 99%
  Building Custom Rule D:/a/curl/curl/tests/CMakeLists.txt
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'D:\a\curl\curl\bld\CMakeFiles\621f80ddbb0fa48179f056ca77842ff0\test-ci.rule;D:\a\curl\curl\tests\CMakeLists.txt' exited with code -1. [D:\a\curl\curl\bld\tests\test-ci.vcxproj]

Ref: https://github.com/curl/curl/actions/runs/9863858249/job/27237533696?pr=14140#step:13:5749

old-mingw-w64 (cmake, 7.3.0-x86_64), FTP

TESTFAIL: These test cases failed: 2039

Ref: https://github.com/curl/curl/actions/runs/9897168501/job/27341046058?pr=14160#step:13:4269

mingw, CM x86_64 schannel U, IMAP:

TESTFAIL: These test cases failed: 803

Ref: https://github.com/curl/curl/actions/runs/9916302331/job/27398414368?pr=14129#step:23:3836

vszakats referenced this pull request May 27, 2024
MQTT / OmniOS:
```
TESTFAIL: These test cases failed: 1190 1198 3017
```
Ref: https://github.com/curl/curl/actions/runs/9258522297/job/25468730731?pr=13694#step:3:10251

MQTT / OmniOS:
```
TESTFAIL: These test cases failed: 1194 2200 2203 2205
```
Ref: https://github.com/curl/curl/actions/runs/9150523540/job/25155409832#step:3:10233

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 1096
```
Ref: https://github.com/curl/curl/actions/runs/9150702711/job/25155793948#step:3:10247

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 381
```
Ref: https://github.com/curl/curl/actions/runs/9163863822/job/25193897640#step:3:10230

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 340
```
Ref: https://github.com/curl/curl/actions/runs/9233804752/job/25406671742?pr=13771#step:3:10245

Ref: #13583 (comment)
vszakats added a commit that referenced this pull request May 28, 2024
Shot in the dark trying to find out which tests are
hanging / going to an infinite loop.

The ones failing after 45 minutes (mingw-w64) or 30 minutes (MSVC).

Ref: #13599 (comment)
vszakats added a commit to vszakats/curl that referenced this pull request May 28, 2024
Following up on previous occurrences showing up as gcc warnings, replace
the remaining few `time(&var)` calls with `= time(NULL)`, though these
aren't specifically causing compiler warnings. All of these are in the
TFTP client code (`lib/tftp.c`), except one which is in a debug branch
in `lib/http_aws_sigv4`.

What's unexplainable is that this patch seems to stabilize the TFTP
tests regularly hanging or going into an infinite loop on GHA windows
workflows with MSYS2, mingw-w64 and MSVC (Cygwin is unaffected):
  curl#13599 (comment)

I figure it's something worth trying even if it turns out to be a dead
end.

`time()` docs:
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/time-time32-time64
https://manpages.debian.org/bookworm/manpages-dev/time.2.en.html

Follow-up to 58ca0a2 curl#13800
Follow-up to d0728c9 curl#13643
Closes #xxxxx
vszakats added a commit that referenced this pull request May 29, 2024
Following up on previous occurrences showing up as gcc warnings, replace
the remaining `time(&var)` calls with `var = time(NULL)`, though these
aren't specifically causing compiler warnings. These are in the TFTP
client code (`lib/tftp.c`), except one which is in a debug branch in
`lib/http_aws_sigv4.c`.

What's unexplainable is that this patch seems to mitigate TFTP tests
often hanging or going into an infinite loop on GHA windows workflows
with MSYS2, mingw-w64 and MSVC (Cygwin is unaffected):
  #13599 (comment)
TFTP hangs did not entirely disappear though, so could be unrelated.

`time()` docs:
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/time-time32-time64
https://manpages.debian.org/bookworm/manpages-dev/time.2.en.html

Follow-up to 58ca0a2 #13800
Follow-up to d0728c9 #13643
Closes #13815
vszakats added a commit to vszakats/curl that referenced this pull request Jul 2, 2024
On the suspicion of sometimes hanging.

Ref: curl#13599 (comment)
Closes #xxxxx
vszakats added a commit to vszakats/curl that referenced this pull request Jul 3, 2024
On the suspicion of sometimes hanging.

Ref: curl#13599 (comment)
Closes #xxxxx
vszakats added a commit that referenced this pull request Jul 3, 2024
- disable SMTP tests in MSYS2/mingw-w64 and MSVC jobs.
  On the suspicion of sometimes hanging:
  https://github.com/curl/curl/actions/runs/9346162475/job/25720437944?pr=13855#step:14:2838
  https://github.com/curl/curl/actions/runs/9758011305/job/26931678639?pr=14084#step:14:2834
  https://github.com/curl/curl/actions/runs/9774468536/job/26982805294#step:11:4731

- run TFTP, MQTT, WebSockets tests in MSYS2/msys jobs again.

- switch hanging old-mingw-w64 7.3.0 job to Release (from Debug).
  Guessing here, 9.5.0 is more solid, and one difference is
  Debug/Release mode. Let's match 7.3.0 with that and see how it changes
  hangs and flakiness.
  The other difference is Unicode ON in 7.3.0. Flaky 6.3.0 was also
  Debug, with Unicode OFF:
  217878b #13566.
  (Unicode unlikely to play a role here IMO.)
  If 7.3.0 keeps hanging / remains flaky I'll consider disabling its
  test runs.

- opt-out from vcpkg telemetry.

Ref: #13599 (comment)
Closes #14085
vszakats added a commit to vszakats/curl that referenced this pull request Jul 7, 2024
vszakats added a commit that referenced this pull request Jul 7, 2024
They are flaky.

E.g.:
- old-mingw-w64 7.3.0: 2001, 2039, 2083
- msvc: 1501, 593 (multiple)

Ref: #13599 (comment)
Cherry-picked from #14116
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration tests Windows Windows-specific
Development

Successfully merging this pull request may close these issues.

None yet

2 participants