cmake: allow to build pytorch as a CMake subproject

[anthonyalayo]

This is a re-attempt of fixing #53980, first submitted in #54978.

Quoting @SpaceIm

Fixes https://github.com/pytorch/pytorch/issues/53980

Maybe it would be nice to find why some files are generated in CMAKE_BINARY_DIR instead of CMAKE_CURRENT_BINARY_DIR or Torch_BINARY_DIR or PROJECT_BINARY_DIR, but there is a lot of indirection in the logic of pytorch build files, so I was not able to find where it comes from.

cc @mcarilli @ptrblck @leslie-fang-intel @jgong5

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110373

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 58408c2 with merge base 06464a3 ():

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

• trunk / linux-focal-rocm5.6-py3.8 / test (default, 1, 3, linux.rocm.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[anthonyalayo]

@pytorchbot label "topic: not user facing"

[anthonyalayo]

@pytorchbot drci

[anthonyalayo]

@malfet I wanted to revive this PR since it would be great to have, originally here:
#54978

Any idea why these CI checks are failing? I'm looking at them, but nothing looks related?

I tested this in a project I work on and it looked good there.

[anthonyalayo]

cc @albanD who helped me get my last PyTorch contribution in

[malfet]

It would be nice to add some trivial unittest (that just runs cmake on a project that lists pytorch as a subproject)

[malfet]

@pytorchbot rebase

[malfet]

Any idea why these CI checks are failing? I'm looking at them, but nothing looks related?

@anthonyalayo your branch is 3000 commits behind main, please rebase (if mergebot fails to do it for you)

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Successfully rebased aalayo-fix/cmake-allow-addsubdirectory-fetchcontent onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout aalayo-fix/cmake-allow-addsubdirectory-fetchcontent && git pull --rebase)

[anthonyalayo]

@anthonyalayo your branch is 3000 commits behind main, please rebase (if mergebot fails to do it for you)

Nice catch, didn't realize I didn't update my fork.

It would be nice to add some trivial unittest (that just runs cmake on a project that lists pytorch as a subproject)

I'll do that next, thanks @malfet

[anthonyalayo]

@pytorchbot drci

[anthonyalayo]

@malfet I started writing the test, but it takes a pretty long time to fetch and build since it pulls in PyTorch itself. Is that alright to have as something that runs on CI?

I'm basing the test off of this one:
4972cf0

Unlike that one though, since we are testing that ExternalProject_Add() works, we can't leverage the build that gets generated through the python test script.

[anthonyalayo]

@malfet any thoughts on that last comment?

[malfet]

Changes to the 3 CMakeLists.txt and Codegen.cmake looks fine to me

[malfet]

@anthonyalayo thank you for the updates, looks good to me if trunk and binary builds are passing.
As for the test, it can be done as followup PR(because otherwise, if not tested it will regress after a while), but yes, something similar to #41145 would do

[anthonyalayo]

Looks like everything test clean, good to merge?

[anthonyalayo]

@malfet can we start the merge? thanks!

[anthonyalayo]

@malfet any thoughts?

[malfet]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here