CI: Only run coverage jobs on master

[alamb]

Draft until #2212

Which issue does this PR close?

re #2149

Rationale for this change

The codecov job is long (example takes 20 minutes, and I haven't seen its results referred to in PRs myself.

It also has several unfortunate issues (like not accounting for coverage of doc tests), which we should probably file as a separate issue if anyone is looking at its results.

Thus, let's only run it on pushes to master

What changes are included in this PR?

1. Only run coverage job on pushes to master

Are there any user-facing changes?

no

[tustvold]

Won't this mean we lose information about test coverage in PRs? I personally don't regular inspect this, but some people might?

[alamb]

Won't this mean we lose information about test coverage in PRs?

Yes that is correct.

@nevi-me @HaoYang670 @jhorstmann @viirya @sunchao do you ever use the per PR code coverage report ? I am proposing to only run coverage jobs on master to improve our CI

[nevi-me]

Won't this mean we lose information about test coverage in PRs?

Yes that is correct.

@nevi-me @HaoYang670 @jhorstmann @viirya @sunchao do you ever use the per PR code coverage report ? I am proposing to only run coverage jobs on master to improve our CI

An alternative could be to run coverage as part of tests, via cargo--llvm-cov, which should also be faster. I'll make changes and PR them into this PR so we can discuss it.

I locally use llvm-cov so I do benefit from Rust coverage, but I'm inactive on arrow-rs at the moment, and would be fine if the decision was to disable coverage altogether if it isn't being used.

[nevi-me]

On second thoughts, I believe we can't use certain actions, I wanted to add taiki-e/install-action@cargo-llvm-cov, but that might need INFRA approval.

llvm-cov is generally faster than tarpaulin though, so I'll explore getting it to run without the action above

[alamb]

llvm-cov is generally faster than tarpaulin though, so I'll explore getting it to run without the action above

Thanks @nevi-me !

Note there is already one simple action in the arrow-rs repository https://github.com/apache/arrow-rs/tree/master/.github/actions/setup-builder in case you wanted to add another for coverage

[alamb]

Marking as a draft so we don't merge this PR accidentally

[nevi-me]

Okay, thanks, I'll work on this on the weekend, if I can't make llvm-cov work, we can move to using tarpaulin only on master.

[alamb]

For what it is worth I don't think this ticket is super high priority -- I was just going through all the CI checks in general (on the occasion of the object_store crate being merged and making the current intertwined ness just a bit too much to bear).

[viirya]

I personally don't use it too much, so the idea of running coverage jobs on master is okay for me. But @nevi-me's proposal is awesome if it works. Thanks @nevi-me for working on it.

[tustvold]

Shall we update and get this one in, there seems to be fairly broad consensus?

[alamb]

Shall we update and get this one in, there seems to be fairly broad consensus?

👍 I will do so. We'll save some CO2 and maybe help the other PRs to run a bit faster. We can always re-enable if someone wants to see it.

[alamb]

I don't really know how to test this other than on master, so YOLO here we go:

[alamb]

Well, this is promising -- the job is queued to run on the master commit:

5166a08

https://github.com/apache/arrow-rs/runs/7679145290?check_suite_focus=true

[ursabot]

Benchmark runs are scheduled for baseline = d87f6a4 and contender = 5166a08. 5166a08 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java