Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split the compiler_gym package into multiple packages #669

Open
sogartar opened this issue Apr 28, 2022 · 3 comments
Open

Split the compiler_gym package into multiple packages #669

sogartar opened this issue Apr 28, 2022 · 3 comments
Assignees
Labels
Enhancement New feature or request

Comments

@sogartar
Copy link

🚀 Feature

It would be beneficial to split the compiler_gym package into multiple packages. Currently, everything is built and installed together with the core runtime stuff + the environments. The environments should be installed separately per user's choice.

Motivation

I don't know if there is a goal for proliferation of environments and where they will be hosted, but the facilities for creating new environments should be available in a library form. This can mostly include the C++ stuff from compiler_gym/service and compiler_gym/util. Currently you could not easily reuse this code if the environment is not a part of the CompilerGym build.

The introduction of the MLIR environment #652 encountered the problem with conflicting multiple versions of LLVM. This is due to CMake's legacy how find_package works #568 (comment). This problem will be automatically solved if there is no mono build and each environment is built as a separate package.

Pitch

Split the compiler_gym package into packages each with a separate build.

compiler_gym # runtime
compiler_gym[dev] # development
compiler_gym[gcc]
compiler_gym[llvm]
compiler_gym[loop_tool]
compiler_gym[mlir]
@ChrisCummins
Copy link
Contributor

I think splitting the cmake build into separate packages is great for solving the problem of conflicting version of LLVM, and for making it possible to develop a new environment without needing to fork the main CompilerGym repo.

One clarification is that I think this split should only apply to the build system. The Python wheel (as installed by pip install compiler_gym) should include all of the GCC/LLVM/MLIR/loop_tool environments. I don't think there's any need to split the compiler_gym wheel unless the number of environments grows so large that it becomes prohibitively expensive to ship a single wheel, which I think is a long way off 🙂

Cheers,
Chris

@sogartar
Copy link
Author

sogartar commented May 9, 2022

Do you think we should add the C++ development library for the gRPC service to the python package?
The other option is the user to build it from source and install it through CMake directly.

@ChrisCummins
Copy link
Contributor

If I understand the question properly, I think the C++ development libraries should be kept separate from the python wheel. So in C++ / CMake land you can do:

ExternalProject_Add(
    compiler_gym
    URL
        "https://github.com/facebookresearch/CompilerGym/archive/refs/tags/v0.2.4.tar.gz"
    ...
)

and that would provide everything needed to build a new compiled service. For the python runtime you would install that independently using:

python -m pip install compiler_gym==0.2.4

Cheers,
Chris

@ChrisCummins ChrisCummins self-assigned this Nov 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants