Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package LLVM core #292

Closed
lucienboillod opened this issue May 28, 2018 · 37 comments
Closed

Package LLVM core #292

lucienboillod opened this issue May 28, 2018 · 37 comments
Assignees
Labels
feedback Waiting for feedback from the original issue author. Gets closed after several weeks if no feedback packaging Plans to create a package within Bincrafters

Comments

@lucienboillod
Copy link
Member

LLVM is a collection of several projects (LLVMCore, Clang, libcxx ...). After trying to build everything at once, we figured out that was really too huge and we decided to package the llvm core at first step, and then others sub-projects as submodule (like we do for Qt or Boost).

@uilianries
Copy link
Member

I think @Manu343726 is working on this, but on his gitlab. I read some posts on Twitter about LLVM + Conan.

@lucienboillod
Copy link
Member Author

lucienboillod commented May 28, 2018

@uilianries Thanks for the information, I will ask him then.

I already packaged llvm+clang+libcxx on my own github https://github.com/lucienboillod/conan-llvm was too long for the CI but it was working nice locally.

Few days ago I created the llvmcore repository also https://github.com/lucienboillod/conan-llvmcore
It seems to pass, but I had to ask 90 min for both appveyor and travis CI.

@lucienboillod
Copy link
Member Author

@uilianries Ok I saw the repo you talked about, it is a fork of @polysquare old repository, it seems to fetch from no-official url source. But it could be nice to have feedback from them 😊

@lucienboillod lucienboillod added the packaging Plans to create a package within Bincrafters label May 28, 2018
@Manu343726
Copy link

@uilianries thanks for pinging. I'm working on automating the packaging of different llvm components and versions, https://gitlab.com/Manu343726/clang-conan-packages

The master branch has a pipeline that builds and deploys version 3.8.0 to my conan repo. To support following versions, I had to change the recipes so sources are fetched from the right repository (The old @polysquare repository has 3.8.0 sources only). All recipes have been modernized to support latest conan versions. Also, I managed to get the recipes working on Windows with VS2017, the repository already includes binary packages with release and debug VS2017 builds. Due to space limitations of my bintray account, no debug packages are included for linux.

All the refactoring work needed to support more versions and share code between recipes is being done here https://gitlab.com/Manu343726/clang-conan-packages/tree/common_recipe_scripts

Once I get 3.8.0 to 6.0.0 working and the recipes sharing common code (There's a lot of redundant and/or unused code in the current recipes inherited from the original repo) I will request for admission in conan-center. ETA 1-2 months

@Manu343726
Copy link

About the sources not coming from official LLVM repository, I would not like to have users of the recipes to rely on LLVM releases server (It would not be the first time that server is down bc of people fetching releases for their CI from there). So my build pipeline will include a step to fetch sources and deploy them to bintray, so all conan users rely entirely on jfrog infrastructure (Easier to report and get things fixed quickly :) ).

@lucienboillod
Copy link
Member Author

@Manu343726 Nice work! Do you plan to package also others llvm's sub-projects ? (openMP, polly, lldb, licxxabi ...). Imo the idea would be to have a llvm package on top of every llvm projects and every sub-projects as submodules (so the packages are the smallest possible) like with boost.
The way you are doing looks really promising, if you need any help do not hesitate.

Btw you can ask bintray to add more space on your account so you can also include debug and such, they are really receptives, tell me if you need any contact I already have an open thread with them juste about uploading size of LLVM in bintray.

@SSE4 @solvingj wdyt ? Should we stop the current work on llvm packaging and assist @Manu343726 on his work ?

@SSE4
Copy link
Member

SSE4 commented May 29, 2018

@lucienboillod in general yes, I'd like to avoid resource duplication as much as possible, so let's assist @Manu343726 and build other sub-projects on top of his llvm-core package.

@Manu343726
Copy link

Manu343726 commented May 30, 2018

@lucienboillod my idea was to package the components needed by libclang, to then make a libclang meta-package on top of the clang one. But I will be glad to add other LLVM modules to the pipeline as well. The pipeline is auto-generated from a declarative YAML file, similar to what conan-package-tools does.

The template also generates jobs that package python packages into conan packages, autogenerating the recipes. For a python package directory like:

Package/
    module.py
    othermodule.py
    __init__.py

the job generates the recipes needed to make that python modules available as a conan python package. Both features are implemented by two python scripts in the repository. There's also a job to mirror the LLVM sources in a bintray repo.

So adding more LLVM modules to the pipeline would be as easy as adding more entries to the packages list of the yaml file, with the corresponding recipe directories in the sourcetree.

LLVM modules are cmake-modularized, all module dependencies are requested through standard find() config files, so as long as you set the right cmake module paths the build of each module can depend directly on existing already-built modules (That's why the build pipeline is written in that order, each stage assumes previous stages have successfully submitted dependencies to the conan repository).

@Manu343726
Copy link

BTW it should be possible to check for recipe changes in the CI, so that jobs that build unchanged packages are skipped (That would help a lot with my electricity bills :P).

Also, the CI generator script only implements the .gitlab.ci file, but it should be easy to implement other services such as appveyor and travisci as well.

@Manu343726
Copy link

Hi Manu343726, 

Your Bintray account is currently rate-limited due to overuse. 
Customer Support should contact you at the earliest to remove the limits from your account. You may also contact them directly at support@jfrog.com. 

I definitely need support guys xD. Ping @danimtb

@Manu343726
Copy link

I've finaly got help from the jfrog side, my limits have been increased but my account cannot cope with all the packages in any way. So I've removed all binaries from my account and fired a reduced build. Once the package is submitted to conan-center we could increase the number of package variants built by the pipeline.

@lucienboillod
Copy link
Member Author

Hello @Manu343726 , how is going the project ? 😄 need any help ?

@Manu343726
Copy link

Sorry for the late response

I've managed to have the four recipes working sharing code with a conan python package (For things like install, sources download, etc). After testing the package with one of my projects I noticed I'm not exporting the llvm libraries as part of cpp_info.libs. If I use CMake relying on the standard LLVMConfig scripts everything works.

Now the question is how I should package each llvm library so that its dependencies are correctly handled by conan. LLVM is a monolithic monster much like boost.

@Croydon
Copy link
Member

Croydon commented Aug 11, 2018

What is the current state of this? I'm specifically interested in a recipe for libclang

@Manu343726
Copy link

Hi people,

I'm back working on this, currently in the last stage to get a PoC: A recipe for a library (cppast) that depends on libclang.

The main issue with this package are the different LLVM modules, which I'm exporting as part of cpp_info.libs (Which means any package depending on LLVM links with all llvm libraries. The same happens with clang and its modules).
Also, I think there's an issue with the llvm package library path, because the clang binary compiled by the clang package cannot find the LLVM libraries. Here's the output of ldd clang:

$ ldd /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/clang
	linux-vdso.so.1 (0x00007ffcc93c0000)
	libLLVMX86CodeGen.so.6 => not found
	libLLVMX86AsmParser.so.6 => not found
	libLLVMX86Desc.so.6 => not found
	libLLVMX86Info.so.6 => not found
	libLLVMAnalysis.so.6 => not found
	libLLVMCodeGen.so.6 => not found
	libLLVMCore.so.6 => not found
	libLLVMipo.so.6 => not found
	libLLVMInstCombine.so.6 => not found
	libLLVMInstrumentation.so.6 => not found
	libLLVMMC.so.6 => not found
	libLLVMMCParser.so.6 => not found
	libLLVMObjCARCOpts.so.6 => not found
	libLLVMOption.so.6 => not found
	libLLVMScalarOpts.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMTransformUtils.so.6 => not found
	libLLVMVectorize.so.6 => not found
	libclangBasic.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/libclangBasic.so.6 (0x00007ff73dbed000)
	libclangCodeGen.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/libclangCodeGen.so.6 (0x00007ff73d5eb000)
	libclangDriver.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/libclangDriver.so.6 (0x00007ff73d251000)
	libclangFrontend.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/libclangFrontend.so.6 (0x00007ff73cf00000)
	libclangFrontendTool.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/libclangFrontendTool.so.6 (0x00007ff73ccfa000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ff73cb6b000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007ff73c9a7000)
	libLLVMCore.so.6 => not found
	libLLVMMC.so.6 => not found
	libLLVMSupport.so.6 => not found
	libclangAST.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangAST.so.6 (0x00007ff73c44f000)
	libclangAnalysis.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangAnalysis.so.6 (0x00007ff73c1ac000)
	libclangLex.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangLex.so.6 (0x00007ff73bee4000)
	libLLVMAnalysis.so.6 => not found
	libLLVMBitReader.so.6 => not found
	libLLVMBitWriter.so.6 => not found
	libLLVMCore.so.6 => not found
	libLLVMCoroutines.so.6 => not found
	libLLVMCoverage.so.6 => not found
	libLLVMipo.so.6 => not found
	libLLVMIRReader.so.6 => not found
	libLLVMInstCombine.so.6 => not found
	libLLVMInstrumentation.so.6 => not found
	libLLVMLTO.so.6 => not found
	libLLVMLinker.so.6 => not found
	libLLVMMC.so.6 => not found
	libLLVMObjCARCOpts.so.6 => not found
	libLLVMObject.so.6 => not found
	libLLVMPasses.so.6 => not found
	libLLVMProfileData.so.6 => not found
	libLLVMScalarOpts.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMTarget.so.6 => not found
	libLLVMTransformUtils.so.6 => not found
	libLLVMOption.so.6 => not found
	libLLVMSupport.so.6 => not found
	libclangEdit.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangEdit.so.6 (0x00007ff73bccd000)
	libclangParse.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangParse.so.6 (0x00007ff73b9e1000)
	libclangSema.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangSema.so.6 (0x00007ff73b0d4000)
	libclangSerialization.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangSerialization.so.6 (0x00007ff73ad50000)
	libLLVMBitReader.so.6 => not found
	libLLVMOption.so.6 => not found
	libLLVMProfileData.so.6 => not found
	libLLVMSupport.so.6 => not found
	libclangRewriteFrontend.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangRewriteFrontend.so.6 (0x00007ff73aad9000)
	libclangARCMigrate.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangARCMigrate.so.6 (0x00007ff73a629000)
	libclangStaticAnalyzerFrontend.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/libclangStaticAnalyzerFrontend.so.6 (0x00007ff73a3ec000)
	libLLVMOption.so.6 => not found
	libLLVMSupport.so.6 => not found
	libm.so.6 => /usr/lib/libm.so.6 (0x00007ff73a265000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff73e337000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff73a24b000)
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMMC.so.6 => not found
	libLLVMMCParser.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMBitReader.so.6 => not found
	libLLVMSupport.so.6 => not found
	libclangRewrite.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/../lib/libclangRewrite.so.6 (0x00007ff73a035000)
	libLLVMSupport.so.6 => not found
	libclangStaticAnalyzerCheckers.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/../lib/libclangStaticAnalyzerCheckers.so.6 (0x00007ff739ae3000)
	libLLVMSupport.so.6 => not found
	libclangStaticAnalyzerCore.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/../lib/libclangStaticAnalyzerCore.so.6 (0x00007ff739725000)
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libclangASTMatchers.so.6 => /home/manu343726/.conan/data/clang/6.0.0/Manu343726/testing/package/a6a012ba8289bad40cf4bd917ff36cd45fe77dab/bin/../lib/../lib/../lib/../lib/libclangASTMatchers.so.6 (0x00007ff7394b9000)
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found
	libLLVMSupport.so.6 => not found

As you can see, clang modules are found by the loader, but LLVM's are not.

@Manu343726
Copy link

Found the problem: The clang executable RPATHs were filled with full paths of my CI builder, which means I cannot use precompiled packages. If I build from sources everything works fine.

@Croydon
Copy link
Member

Croydon commented Oct 21, 2018

If you got some stable Conan recipes we would happily taking them in to Bincrafters and give you push rights to the new repository. Are some dependencies already working?

It's kinda hard to get an overview as you have several conanfiles per repository and have some custom build helpers. Also are you GitHub repositories outdated and you continued working on Gitlab exclusively?

@Croydon

This comment has been minimized.

@Manu343726
Copy link

Manu343726 commented Oct 21, 2018

If you got some stable Conan recipes we would happily taking them in to Bincrafters and give you push rights to the new repository. Are some dependencies already working?

The situation is as follows:

  • The recipes use a custom python package for common scripts, which should be updated to the new python requires infrastructure (This was done before that feature was announced).
  • All llvm/clang recipes are hosted in the same repo so I can write and fire a full CI pipeline to test and upload all the packages.
  • Yes, I'm doing everything in GitLab, but I would consider switching if that fits you better. Even more considering that Github is working on its own pipeline and TravisCI supports all major platforms now (There are no public GitlabCI runners for windows, but that's not a problem since I always fire up my own machines to run the clang pipeline, which is very intensive for the public runners).
  • Last time I checked the packages also worked for Windows, but I would like to run much more tests before submitting.
  • Packages still expose all LLVM/Clang libraries as a monolith, which is a problem since consumer applications end up linking to all the libs. I would love to discuss ideas about this point.
  • The RPATHs thing. I'm sure there's already a standard Conan solution to this problem, but I don't know what that solution is.

TL;DR: I would happily transfer control to the bincrafters community and write llvm/clang distribution packages from my work following standard practices (conan-package-tools, python requires, travis ci, etc).

@Manu343726
Copy link

Manu343726 commented Oct 22, 2018

FYI gitlab public runners are much more reliable now, the full pipeline worked wo issues https://gitlab.com/Manu343726/clang-conan-packages/pipelines/33749222

@Manu343726
Copy link

Some bad news: Recent versions of compiler-rt (I'm building 6.0.0) cannot be built standalone with MSVC due to a bug in the cmake scripts https://bugs.llvm.org/show_bug.cgi?id=38586

@Manu343726
Copy link

Ok people, some news about the llvm/clang/libclang packages. I've managed to get a setup working, by packaging llvm, compiler-rt, and clang as "component" packages (Full builds of llvm, compiler-rt, and clang projects respectively). Then one meta-package is defined for each module in llvm, compiler-rt, and clang. Those meta-packages package libraries and headers corresponding to the module only (copied from the corresponding "component" package folder, which is required as a build_requires), with inter-module dependencies specified as requires between the different meta-packages.

I have uploaded the recipes here https://bintray.com/manu343726/conan-packages/, there's a libclang meta-package that I'm using for testing purposes that you should be able to require and link statically and dynamically. There are pre-built packages for linux gcc 8.1 stdlibc++11

@Manu343726
Copy link

I've only meta-packaged modules required by libclang, some common modules such as libtooling are missing.

@uilianries

This comment has been minimized.

@Croydon
Copy link
Member

Croydon commented Jan 8, 2019

I haven't fully understood yet how you structure the recipes, but would you say you can split them into several smaller repositories as soon as you start to use python_requires (you have written about this earlier)? It's quite uncommon to have a huge monolithic repository for several Conan recipes, but then again I don't understand your structure yet so I can't really say what would be reasonable...

@Manu343726
Copy link

Manu343726 commented Jan 21, 2019

I'm currently working on a CI stage for testing tinyrefl package, and since tinyrefl depends on libclang I've followed the ugly path of pushing the compiled llvm/clang packages from that jobs once they're compiled. But I would like to setup a proper pipeline for my clang-conan-packages repo with conan-package-tools. Do you have any experience using conan-package-tools with gitlab-ci?

@uilianries
Copy link
Member

Do you have any experience using conan-package-tools with gitlab-ci?

I have! Usually I use this template: https://gitlab.com/snippets/1799856

The build.py referred there looks like:

from cpt.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds()
    builder.run()

But you can change anything.

@Manu343726
Copy link

FYI I finally wrote the CI pipeline using my tool. See https://gitlab.com/Manu343726/clang-conan-packages/pipelines/44984999

@Manu343726
Copy link

I'm dropping Clang 4.0.0 and 5.0.0 from the packaging pipeline due to some unresolved dependencies of the libclang static package (It's weird because the dynamic versions are working fine). I don't want to delay the release of tinyrefl more and since I'm not interested in Clang < 6.0.1 I'm not motivated enough to keep fighting with this issue. Anyway feel free to PR to try to fix the problem.

@jgsogo
Copy link
Member

jgsogo commented Feb 1, 2019

image

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 😞 🎉 🎉

The Mother of all CIs!

@Croydon
Copy link
Member

Croydon commented Mar 29, 2019

@Manu343726 I believe you have done an excellent job here. The only problem is now how we do integrate this into Bincrafters. This is such a huge monster that it is hard to overview and to tackle

Some basic questions to begin with:

  1. To we want to maintain this in a monolithic repository like it is currently? Similar to Qt
  2. Do we want to split it into many repositories? Comparable to Boost

Obviously both has advantages and disadvantages.

I guess at least for the beginning we should keep the monolithic repository and start to adapt it to Bincrafters/GitHub/Azure Pipelines (Travis can't handle that many jobs)

@Manu343726 Are you still interested to get this adopted by Bincrafters? If so, are you fine with putting your scripts/recipes under the MIT license?

@Croydon Croydon self-assigned this Mar 29, 2019
@Manu343726
Copy link

Manu343726 commented Apr 1, 2019

To we want to maintain this in a monolithic repository like it is currently?

I think this is the best option since there's a lot of meta-packages with dependencies between them.

Bincrafters/GitHub/Azure Pipelines (Travis can't handle that many jobs)

And also port the packaging scripts to conan-package-tools.

are you fine with putting your scripts/recipes under the MIT license?

I thought they were already. Let me check.

@Croydon
Copy link
Member

Croydon commented Apr 7, 2019

@Manu343726 It seems like some sub-directories do have a LICENSE.md, but that doesn't apply to everything than. If you could add one to the root directory that would be perfect for confirmation 😄

@Croydon
Copy link
Member

Croydon commented May 20, 2019

@Manu343726 Ping 😄

@Croydon Croydon added the feedback Waiting for feedback from the original issue author. Gets closed after several weeks if no feedback label Jun 11, 2019
@no-response
Copy link

no-response bot commented Aug 10, 2019

This issue has been automatically closed because there has been no response to our request for more information or feedback from the original author. With only the information that is currently in the issue, we don't have enough information to take further action. Please reach out if you have or find the answers we need so that we can investigate further.

@no-response no-response bot closed this as completed Aug 10, 2019
@Croydon
Copy link
Member

Croydon commented Aug 10, 2019

@Manu343726 If at any point in the future you could add a LICENSE file to the root of your repository, that would be awesome. Otherwise we can't build upon our work.

We would also need a new champion moving such a gigantic project forward.

Closing for now.

Wishlist: conan-io/wishlist#197

@blockspacer
Copy link

blockspacer commented May 28, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feedback Waiting for feedback from the original issue author. Gets closed after several weeks if no feedback packaging Plans to create a package within Bincrafters
Projects
None yet
Development

No branches or pull requests

7 participants