Meta-issue: current issues with the coverage job

[DilumAluthge]

This is a great step! But there seem to be a few issues still, looking at https://app.codecov.io/gh/JuliaLang/julia

• paths are missing base/ prefix, so the coverage tools can't find them in the repo. This means that you can't view the source code files on the Codecov website.
• there is a test file accidentally present https://codecov.io/gh/JuliaLang/julia/src/master/cache/build/default-amdci5-7/julialang/julia-master-scheduled/usr/share/julia/test/project/deps/Foo1/src/Foo.jl
• the stdlib files have their full path instead of being stdlib/ prefixed https://codecov.io/gh/JuliaLang/julia/tree/master/cache/build/default-amdci5-7/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.9
• Pkg (and other similar files that do not live in the JuliaLang/julia repo) are getting included in the stats
• coverage changed from ~89% to ~93%, which seems unlikely and might indicate the amend_coverage_from_src! step is being skipped, though it is hard to compare what changed currently do to previous issues

Originally posted by @vtjnash in #136 (comment)

[IanButterworth]

Are coverage files actually uploaded for PRs? None of these seem to be? https://app.codecov.io/gh/JuliaLang/julia/pulls

I want to see if JuliaLang/julia#46403 works

[DilumAluthge]

We do not upload coverage on PRs, because coverage takes 3-4 hours to run.

You can make a PR to this repo, and in the PR, you set it to use your custom branch of Julia instead of the master branch of Julia. I don't actually remember where you specify your custom branch name, but @staticfloat or @brenhinkeller will know.

[IanButterworth]

Makes sense! This is me trying. Although I think it might hit a build bug on non-macos that I can't figure out..
#194

[brenhinkeller]

Sounds like you may have figured it out but FWIW re @dilumalthuge's point IIRC one way to pick a custom branch might be to set UPSTREAM_BRANCH as an environment variable: https://github.com/JuliaCI/julia-buildkite/blob/main/.buildkite/hooks/post-checkout

[vtjnash]

Here is the old script that addressed most of these issues in buildbot https://github.com/JuliaCI/julia-buildbot/blob/96f24ff3fcfff05c5411662363483271d364ee59/master/coverage.py#L23-L24

[vtjnash]

bump?

[kshyatt]

Items 1 and 3 have been fixed.

[kshyatt]

2 and 4 I just confirmed are still problems.

[kshyatt]

Looking at https://app.codecov.io/gh/JuliaLang/julia/commit/c4fd8a48383235e818c44231d7507168c27f393b/tree/stdlib, 4 seems to have been fixed.

[kshyatt]

https://codecov.com/gh/julialang/julia/tree/2a9f4412a00cbaf366895083f651d5d7906cce1f the extraneous file is also gone

[kshyatt]

Coverage dropped back down to ~89%. Looks like we can close this!