Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package: llvm/13 - llvm/16 #17509

Closed
wants to merge 174 commits into from
Closed

package: llvm/13 - llvm/16 #17509

wants to merge 174 commits into from

Conversation

jusito
Copy link

@jusito jusito commented May 10, 2023

Specify library name and version: llvm/13.0.1, llvm/14.0.6, llvm/15.0.7, llvm/16.0.6

Current State 07.2023 waiting for reviews

follow up on previous pull request: #7613

My main focus in this pr is:

  • llvm/14.0.6 without any project/runtime should work on linux/mac
  • llvm/14.0.6 + clang / dynlib should be usable if supported by the project itself
  • keeping certain binaries, e.g. keeping clang-14 as binary to compile with this specific clang version to LLVM IR but using a different compiler for llvm/clang itself

What I added:

  • Conan2 migration
  • job pool calculation according to available memory, helps a lot in a shared ci environment as a trade of, could fix the previous timeout explained here
  • Linux/Ubuntu tested with clang-15 in many different configurations
    • beside lldb/libc/compiler-rt everything should work, this three are always a bit special
      • with libc enabled clang can crash in sin/cos/tan for llvm/13-14 in debug build
      • compiler-rt always requires x86 headers for compilation
      • lldb has circular deps which I removed by hand, probably thats not always a good idea
  • Windows & Mac ongoing development
  • lots of testing and tweaking

Tasks:

  • Windows testing
  • Mac testing, currently lack of hardware
  • compatibility settings, why rebuild llvm without clang if there is a package which contains clang?

Feedback and help welcome.


Copy link
Contributor

@jwillikers jwillikers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding LLVM 17!

I recommend looking over the CMake package template, as you'll want to align with it as much as possible.

@@ -0,0 +1,55 @@
from conan import ConanFile
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from conan import ConanFile
from conan import ConanFile
from conan.tools.env import VirtualBuildEnv, VirtualRunEnv

@@ -0,0 +1,55 @@
from conan import ConanFile
from conan.tools.cmake import cmake_layout, CMakeDeps, CMakeToolchain, CMake
from conan.tools.build import cross_building
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from conan.tools.build import cross_building
from conan.tools.build import can_run

Comment on lines +54 to +55
if not cross_building(self, self.settings):
self.run("./test_package")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should run under the VirtualRunEnv.

Suggested change
if not cross_building(self, self.settings):
self.run("./test_package")
if can_run(self):
bin_path = os.path.join(self.cpp.build.bindir, "test_package")
self.run(bin_path, env="conanrun")

deps.generate()
tc = CMakeToolchain(self)
tc.generate()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the tool_requires to take effect, use VirtualBuildEnv.

Suggested change
env = VirtualBuildEnv(self)
env.generate()

@@ -0,0 +1,32 @@
from conans import ConanFile, CMake, tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's probably alright to just drop the test_v1_package entirely now, assuming no other reviewers have a good reason for keeping it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I didn't have one in the first place but ci complained about.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the policy has changed since you opened your PR.


def source(self):
get(self, **self.conan_data["sources"][self.version], strip_root=True)
apply_conandata_patches(self)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this due to the no_copy_source option being set? You should apply patches in the build method instead of the source method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, nevermind. I see this is exactly what we do for the Qt package.

Comment on lines +175 to +182
def is_windows(self):
return self.settings.os == "Windows"

def is_macos(self):
return self.settings.os == "MacOS"

def is_linux(self):
return self.settings.os == "Linux"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be safe, you should prefix any custom methods or attributes with an _ to avoid conflicts.

Comment on lines +194 to +207
release = Version(self.version).major
unsupported_options = []
if release < 14:
unsupported_options.append('with_curl')
if release < 15:
unsupported_options.append('with_zstd')
unsupported_options.append('with_httplib')
# TODO fix with_zstd and llvm 17
if release >= 17:
self.output.warning(f"with_zstd will fail in llvm 17 with conan 2")
unsupported_options.append('with_zstd')
for opt in unsupported_options:
self.output.warning(f"{opt} is unsupported in llvm {release}")
self.options.rm_safe(opt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably better to delete options based off of the version and any settings in the config_options method instead. This way, the options won't be available when consumers try to set them in the first place. You'll want to put that above the configure method to reflect the evaluation order.

    def config_options(self):
        if self.settings.os == "Windows":
            self.options.rm_safe("fPIC")  # FPIC MANAGEMENT (KB-H007)
        release = Version(self.version).major
        unsupported_options = []
        if release < 14:
            unsupported_options.append('with_curl')
        if release < 15:
            unsupported_options.append('with_zstd')
            unsupported_options.append('with_httplib')
        # TODO fix with_zstd and llvm 17
        if release >= 17:
            unsupported_options.append('with_zstd')
        for opt in unsupported_options:
            self.options.rm_safe(opt)

cmake = self._cmake_configure()
cmake.build()

def _cmake_configure(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be migrated to use the toolchain in the generate method instead of the CMake configure method.

self.tool_requires("cmake/[>=3.21.3 <4.0.0]")
self.tool_requires("ninja/[>=1.10.0 <2.0.0]")

def generate(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need VirtualBuildEnv here.

@jusito
Copy link
Author

jusito commented Oct 15, 2023

@jwillikers I will test the configs locally first before adding your suggestions, some parts are really what I was looking for, thanks a lot!

@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ✔️

All green in build 9 (10984d3c14c1d8946457daf6e896580c8944e90f):

  • llvm/17.0.2:
    All packages built successfully! (All logs)

  • llvm/16.0.6:
    All packages built successfully! (All logs)

  • llvm/14.0.6:
    All packages built successfully! (All logs)

  • llvm/13.0.1:
    All packages built successfully! (All logs)

  • llvm/15.0.7:
    All packages built successfully! (All logs)


Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

See details:

Failure in build 9 (10984d3c14c1d8946457daf6e896580c8944e90f):

  • llvm/17.0.2:
    CI failed to create some packages (All logs)

    Logs for packageID c91a80ea6aa72820c0bd2ebd0161d11819e57875:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=msvc
    compiler.cppstd=17
    compiler.runtime=dynamic
    compiler.runtime_type=Release
    compiler.version=193
    os=Windows
    [options]
    */*:shared=False
    
    [...]
    llvm/17.0.2 (test package): CMakeToolchain generated: ..\..\..\CMakeUserPresets.json
    llvm/17.0.2 (test package): Generating aggregated env files
    llvm/17.0.2 (test package): Generated aggregated env files: ['conanbuild.bat', 'conanrun.bat']
    
    ======== Testing the package: Building ========
    llvm/17.0.2 (test package): Calling build()
    llvm/17.0.2 (test package): Running CMake.configure()
    llvm/17.0.2 (test package): RUN: cmake -G "Visual Studio 17 2022" -DCMAKE_TOOLCHAIN_FILE="C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package/build/msvc-193-x86_64-17-release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_CXX_STANDARD="17" -Dllvm_build_llvm_dylib="False" "C:\J2\w\prod-v2\bsr\cci-85ed5e90\recipes\llvm\all\test_package"
    -- Using Conan toolchain: C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package/build/msvc-193-x86_64-17-release/generators/conan_toolchain.cmake
    -- Conan toolchain: C++ Standard 17 with extensions OFF
    -- The CXX compiler identification is MSVC 19.36.32532.0
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Conan: Component target declared 'LLVMSupport'
    -- Conan: Target declared 'llvm::llvm'
    -- Conan: Target declared 'ZLIB::ZLIB'
    -- Conan: Component 'LLVMSupport' found in package 'llvm'
    -- Configuring done (4.3s)
    -- Generating done (0.0s)
    CMake Warning:
      Manually-specified variables were not used by the project:
    
        CMAKE_POLICY_DEFAULT_CMP0091
    
    
    -- Build files have been written to: C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package/build/msvc-193-x86_64-17-release
    
    llvm/17.0.2 (test package): Running CMake.build()
    llvm/17.0.2 (test package): RUN: cmake --build "C:\J2\w\prod-v2\bsr\cci-85ed5e90\recipes\llvm\all\test_package\build\msvc-193-x86_64-17-release" --config Release
    MSBuild version 17.6.3+07e294721 for .NET Framework
    
      1>Checking Build System
      Building Custom Rule C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package/CMakeLists.txt
      test_package.cpp
      test_package.vcxproj -> C:\J2\w\prod-v2\bsr\cci-85ed5e90\recipes\llvm\all\test_package\build\msvc-193-x86_64-17-release\Release\test_package.exe
      Building Custom Rule C:/J2/w/prod-v2/bsr/cci-85ed5e90/recipes/llvm/all/test_package/CMakeLists.txt
    
    
    ======== Testing the package: Executing test ========
    llvm/17.0.2 (test package): Running test()
    llvm/17.0.2 (test package): RUN: ./test_package
    '.' is not recognized as an internal or external command,
    operable program or batch file.
    
    ERROR: llvm/17.0.2 (test package): Error in test() method, line 55
    	self.run("./test_package")
    	ConanException: Error 1 while executing
    
  • llvm/13.0.1:
    All packages built successfully! (All logs)

  • llvm/16.0.6:
    All packages built successfully! (All logs)

  • llvm/14.0.6:
    Didn't run or was cancelled before finishing

  • llvm/15.0.7:
    Didn't run or was cancelled before finishing


Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.

@jwillikers jwillikers mentioned this pull request Oct 19, 2023
17 tasks
# creating job pools with current free memory
'ram_per_compile_job': '2000',
'ram_per_link_job': '14000',
'conan_center_index_limits': True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of an option, you can use a conf variable to control this, like user.c3i.skip:llvm.
You can get the value of this variable via self.conf.get.

self.conf.get('user.c3i.skip:llvm', default=False, check_type=bool)

Then in global.conf or on the CLI, it can be defined.

conan ... --conf user.c3i.skip:llvm=True

@jwillikers
Copy link
Contributor

Why is LLVM_TARGETS_TO_BUILD set to all by default? Could this default instead to just the target architecture specified in the host profile?

Also, this recipe would be helpful for #20528, which requires components from llvm, one of which is libclc. I'm attempting to use this recipe as a dependency, but libclc isn't produced by the recipe when I enable the with_project_libclc option.
Something like the following should help with packaging libclc for reuse.

    def package(self):
        cmake = self._cmake_configure()
        cmake.install()
        if self.options.get_safe("with_project_libclc"):
            copy(self, "*.bc", os.path.join(self.package_folder, "share", "clc"), os.path.join(self.package_folder, "lib", "clc"))
    def package_info(self):
        self.cpp_info.components["libclc"].set_property("pkg_config_name", "libclc")
        self.cpp_info.components["libclc"].libdirs = [os.path.join(self.package_folder, "lib", "clc")]

@jwillikers
Copy link
Contributor

jwillikers commented Oct 20, 2023

Ah, I see now. It appears that libclc should probably be placed in a separate package. It depends on SPIRV-LLVM-Translator which depends on both SPIRV-tools and LLVM. Also, I had to build it directly from its separate source directory locally to build it at all. Apparently, the source code for SPIRV-LLVM-Translator can also be be included in the projects or tools directory to have it built as part of LLVM, but it has a different license.

Also, with LLVM 17, I noticed that the flang and mlir projects are absent from the list of projects.

@jusito
Copy link
Author

jusito commented Oct 23, 2023

Why is LLVM_TARGETS_TO_BUILD set to all by default? Could this default instead to just the target architecture specified in the host profile?

I stumbled upon this too, but didn't thought about taking host profile which sounds reasonable.

Also, with LLVM 17, I noticed that the flang and mlir projects are absent from the list of projects.

Ok, also a good idea for your config_options suggestion, to remove it there. For this recipe and changed config I have already a workflow to check for definition/requirement changes, didn't expect to miss such change, thanks!

[...] but libclc isn't produced by the recipe when I enable the with_project_libclc option. [...] Ah, I see now. It appears that libclc should probably be placed in a separate package. [...]

So this option shouldn't be part of this recipe.

regarding:

self.conf.get('user.c3i.skip:llvm', default=False, check_type=bool)

Where did you find this? During my search even looked through related issues and never found a note on this, really handy if it works.

@jwillikers
Copy link
Contributor

That was recommended here.

@jacobfriedman
Copy link

Happy to see this moving along. Thank you for the hard work @jusito

@jacobfriedman
Copy link

Building LLVM 15 with a direct_deploy, clang requires libatomic. I wonder if COMPILER_RT_BUILD_STANDALONE_LIBATOMIC will help if COMPILER_RT is needed.

@jusito
Copy link
Author

jusito commented Nov 17, 2023

@jacobfriedman
Just to clarify for replication, for a llvm 15 build:

  • LLVM_ENABLE_PROJECTS should contain clang and compiler-rt
  • What do mean by direct deploy, conan create ?
  • there is a dependency missing which may be fixed with COMPILER_RT_BUILD_STANDALONE_LIBATOMIC, which adds clang_rt.atomic as dependency to compiler_rt
    correct?

@sykhro
Copy link
Contributor

sykhro commented Nov 17, 2023

@jacobfriedman Just to clarify for replication, for a llvm 15 build:

  • LLVM_ENABLE_PROJECTS should contain clang and compiler-rt

compiler-rt should be in LLVM_ENABLE_RUNTIMES

  • there is a dependency missing which may be fixed with COMPILER_RT_BUILD_STANDALONE_LIBATOMIC, which adds clang_rt.atomic as dependency to compiler_rt

compiler-rt already includes libatomic. the COMPILER_RT_BUILD_STANDALONE_LIBATOMIC flag builds clang_rt.atomic separately, so that you can link to it with -latomic like you would do with gcc (which uses GNU's libatomic.a)

@jacobfriedman
Copy link

jacobfriedman commented Nov 17, 2023 via email

Copy link
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Feb 25, 2024
@planetmarshall
Copy link
Contributor

Why is LLVM_TARGETS_TO_BUILD set to all by default? Could this default instead to just the target architecture specified in the host profile?

Those are the targets for which LLVM can generate code. If you remove targets, you will be removing the ability of LLVM to generate code for those architectures. It's not really connected to the conan host or build profiles - it would be like compiling Clang only with the ability to output X86 code.

For example, someone could use LLVM as a library to, say, write a program that compiles CUDA to X86 assembly - but that program could itself be compiled on an X86 host targeting ARM64.

Copy link
Contributor

github-actions bot commented Apr 5, 2024

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Apr 5, 2024
@jwillikers
Copy link
Contributor

Please don't close github-actions bot.

@github-actions github-actions bot removed the stale label Apr 19, 2024
@jcar87
Copy link
Contributor

jcar87 commented Apr 30, 2024

We are currently not prioritising packages that add toolchains (rather than libraries or pure executables), as we are prioritising other issues in our backlog.

LLVM itself has a myriad of components, and I suspect different users would have different needs. We are opening this issue to track this: #23834 - actual uses cases with examples will help the team gain better insights at how a potential llvm recipe may be used, whether or not it is worth packaging different LLVM components in separate recipes, and so on. Any information of this would be appreciated in that issue so that we can move forward, thanks!

@jcar87 jcar87 closed this Apr 30, 2024
@jacobfriedman
Copy link

Why would you close this and open another issue to justify this project?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet