[ci] Clean wheels directory before build, validate wheel commit strings

[krfricke]

Why are these changes needed?

Buildkite runners are re-used and sometimes have leftover state.

This is illustrated in these two builds:

https://buildkite.com/ray-project/ray-builders-branch/builds/3798#98bd4bff-8b77-484d-94a9-5e09ea5f13e2

and

https://buildkite.com/ray-project/ray-builders-branch/builds/3799#283c057c-6334-466b-805d-75535a16e169

(click on "Timeline").

In this case, wheels from the first build (2.0.0.dev0) still exist on the runner and are uploaded again in the next build (1.7.0). In this case this is obvious, but sometimes this happens for the same version string and can lead to confusion.

This PR clears the .whl directory in ci.sh build so that previous build wheels are deleted. We also introduce a sanity check that extracts the wheel commit string from each wheel and compares it to the expected commit.

Related issue number

Closes #18702

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jjyao]

Thanks for fixing the issue.

Besides .whl/, is it possible that we have other leftover from previous build that might mess up with the current build? Should we consider buildkite Clean checkout?

[krfricke]

I think this is a good idea - cc @simon-mo do we have to add this to the buildkite ci stack?

[scv119]

looks good to me. cc @simon-mo for sanity check.

[simon-mo]

LGTM

[simon-mo]

Should we consider buildkite clean checkout?

This might not work as .whl from linux actually comes from a shared component (docker-in-docker container) that's harder to clean. We are already doing clean checkout as a matter of fact.

[krfricke]

The good news: The sanity check works!

https://buildkite.com/ray-project/ray-builders-pr/builds/15243#47a50ee0-841c-4e5e-a6fa-eb21c683b24e

The bad news: Cleaning up the .whl directory does not seem to work. I'll look into this tomorrow.