Skip to content

Commit

Permalink
GHA: ignore flaky MQTT and FTP test results [ci skip]
Browse files Browse the repository at this point in the history 
MQTT / OmniOS:
```
TESTFAIL: These test cases failed: 1190 1198 3017
```
Ref: https://github.com/curl/curl/actions/runs/9258522297/job/25468730731?pr=13694#step:3:10251

MQTT / OmniOS:
```
TESTFAIL: These test cases failed: 1194 2200 2203 2205
```
Ref: https://github.com/curl/curl/actions/runs/9150523540/job/25155409832#step:3:10233

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 1096
```
Ref: https://github.com/curl/curl/actions/runs/9150702711/job/25155793948#step:3:10247

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 381
```
Ref: https://github.com/curl/curl/actions/runs/9163863822/job/25193897640#step:3:10230

FTP / OmniOS:
```
TESTFAIL: These test cases failed: 340
```
Ref: https://github.com/curl/curl/actions/runs/9233804752/job/25406671742?pr=13771#step:3:10245

Ref: #13583 (comment)
  • Loading branch information
@vszakats
vszakats committed May 27, 2024
1 parent c29a20d commit 8373783
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion .github/workflows/non-native.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -195,5 +195,5 @@ jobs:
gmake -j3 install
src/curl --disable --version
gmake -j3 examples
export TFLAGS='-j12'
export TFLAGS='-j12 ~MQTT ~FTP'
gmake check V=1

2 comments on commit 8373783

@dfandrich
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly confident these test failures are due to parallel tests, and aren't specific to FTP and MQTT. Trying to patch over them this way instead of addressing the fundamental issue with the tests or harness is a losing battle.

@vszakats
Copy link
Member Author

@vszakats vszakats commented on 8373783 May 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, but since I cannot fix the test nor the harness, this is the next best thing I can do to
mitigate it. This was meant for OmniOS, which is much flakier than others, and this is likely to do
with the runner env, which doesn't seem very stable. OmniOS is the only job where this particular
runner is used. If this doesn't improve, I plan to delete the OmniOS job as-is. It's not adding
enough value to be worth chasing this.

FWIW, observing the non-native and Windows tests while adding them to GHA, it seems that
MQTT, TFTP, FTP and WebSocket tests are somewhat more prone to flakiness than others.

Some Issues opened:
#13623
#13692

(with some non-flaky issues fixed already)

Tracking the flaky failures here:
#13583 (comment)

I'm more concerned about hangs that can be seen while executing Windows tests.
This is new compared to AppVeyor CI and I couldn't yet figure out which tests are
hanging, to disable (or fix) them.

Hangs are tracked here:
#13599 (comment)

Please sign in to comment.