Run CI (tests) on MacOS #3528
Run CI (tests) on MacOS #3528
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #3528 +/- ##
==========================================
- Coverage 72.54% 72.46% -0.08%
==========================================
Files 278 278
Lines 20891 20933 +42
==========================================
+ Hits 15155 15170 +15
- Misses 4773 4788 +15
- Partials 963 975 +12
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
joanlopez
force-pushed
the
ci-enable-macos
branch
2 times, most recently
from
f64e905
to
2814ddf
Compare
joanlopez
force-pushed
the
ci-enable-macos
branch
from
2814ddf
to
8047e70
Compare
joanlopez
requested review from
mstoykov and
olegbespalov
and removed request for
a team
| curl -fsSLO "https://raw.githubusercontent.com/codecov/codecov-bash/${CODECOV_BASH_VERSION}/codecov" | ||
| echo "$CODECOV_BASH_SHA512SUM codecov" | sha512sum -c - | ||
| echo "$CODECOV_BASH_SHA512SUM codecov" | sha512sum -c - |
There was a problem hiding this comment.
For some reason (I didn't fully get), without the extra space it doesn't work on MacOS, but with it, it works correctly in all platforms 🤷🏻
There was a problem hiding this comment.
I remember that we had issues with codecov (which later were resolved) and created the task, but one of the highlights was that codecov has the GH action. So maybe it's worth migrating to it
There was a problem hiding this comment.
Oh, nice! Thanks for letting me know, I could consider working on that right after, but I'd prefer to keep it for a separate PR, if that sounds good to you as well 🙏🏻
| @@ -37,8 +37,8 @@ jobs: | |||
| go version | |||
| export GOMAXPROCS=2 | |||
| args=("-p" "2" "-race") | |||
| # Run with less concurrency on Windows to minimize flakiness. | |||
| if [[ "${{ matrix.platform }}" == windows* ]]; then | |||
| # Run with less concurrency on Windows/MacOS to minimize flakiness. | |||
There was a problem hiding this comment.
I tried luck without this, and there has been no way to get all tests passing. I guess next step will be to take a look at that flakiness and try to correct it, but for now I guess it's fine, as it has been for Windows so far.
There was a problem hiding this comment.
The issue is that tests are flaky on all platforms, primarily GRPC-TLS related https://github.com/grafana/xk6-grpc/issues/39. Which ones do you constantly see?
There was a problem hiding this comment.
Yeah, there's nothing new, as I think all (or most) of them are documented at #2144.
It's just that it seems that Windows and MacOS runners are way more likely to fail (it's probably a matter of their performance), so I decided to add MacOS into that exception we already had for Windows, to avoid adding more flakiness.
What?
It enables
macos-latestfor test-related (test.yml) CI workflows.Why?
Ha, why not? Jokes aside, imho now that MacOS-related issues (#2578 and #1230) have been closed (fixed), and that it looks like MacOS runners have at the very least a performance equivalent to Windows runners' (in fact, test jobs seems to be taking less time in MacOS runners), I guess it's good idea to make the check of the good functioning of k6 on MacOS part of the CI checks.
Also, in the worst case we can disable it again, but why not giving it a try?
Checklist
make lint) and all checks pass.make tests) and all tests pass.Related PR(s)/Issue(s)
I haven't found any.