Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Build] Another attempt at building Python 3.9 MacOS wheels #16347

Merged
merged 5 commits into from
Jun 10, 2021

Conversation

mwtian
Copy link
Member

@mwtian mwtian commented Jun 9, 2021

Why are these changes needed?

I was able to reproduce the issue of Python 3.9 wheels missing ray/python/ray/thirdparty_files on my MacBook. IIUC, the file copying logic in python/setup.py is broken in some cases. Logic in distutils.build_ext actually copied the psutil and setproctitle files. This behavior could have changed on MacOS Python 3.9, for these types of files. Fixing the logic in setup.py seem to fix the issue.

A more radical approach would be to not vendor psutil and setproctitle anymore, and require users to install themselves as Python dependencies. If Ray C++ uses these libraries, they could (might already?) be statically linked.

Related issue number

#11287

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • CI tests
    • Release tests
    • This PR is not tested :(

@mwtian mwtian changed the title [WIP - Test wheels][Build] Yet another attempt at building Python 3.9 MacOS wheels [Build] Another attempt at building Python 3.9 MacOS wheels Jun 10, 2021
Copy link
Contributor

@wuisawesome wuisawesome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine, let's wait until morning to merge it though, since the other attempts also looked fine until they broke master 😛

@mwtian
Copy link
Member Author

mwtian commented Jun 10, 2021

Fixed an issue in setup.py and reverted the change in build-wheel-macos.sh. MacOS Py 3.9 wheel passes validation: https://travis-ci.com/github/ray-project/ray/jobs/512659322#L34378-L34457

@wuisawesome
Copy link
Contributor

crossing fingers and merging now 🤞

@wuisawesome wuisawesome merged commit 2e8d8fb into ray-project:master Jun 10, 2021
@mwtian mwtian deleted the pyver branch June 10, 2021 17:32
@wuisawesome
Copy link
Contributor

This might've worked? There were some travis CI issues today, so we're 1/2 on the mac wheel builds. The failed build was due to a timeout (this also happened before thought). I'm cautiously excited, but we may need to make sure the mac wheel build doesn't timeout too often.

@mwtian
Copy link
Member Author

mwtian commented Jun 11, 2021

Yes, I hope it works too! From my observation, it seems every time we run out then refill Travis credits, all Bazel deps are rebuilt which can longer time. Hopefully we will see faster Mac builds tonight!

@mwtian mwtian added the 1.4.1 label Jun 14, 2021
DmitriGekhtman pushed a commit that referenced this pull request Jun 20, 2021
* Test rollforward python 3.9

* Upgrade setproctitle in thirdparty

* Update cython requirement to match those in wheel build scripts.

* Fix MANIFEST.in

* Fix file copying in setup.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants