-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Package LLVM core #292
Comments
I think @Manu343726 is working on this, but on his gitlab. I read some posts on Twitter about LLVM + Conan. |
@uilianries Thanks for the information, I will ask him then. I already packaged llvm+clang+libcxx on my own github https://github.com/lucienboillod/conan-llvm was too long for the CI but it was working nice locally. Few days ago I created the llvmcore repository also https://github.com/lucienboillod/conan-llvmcore |
@uilianries Ok I saw the repo you talked about, it is a fork of @polysquare old repository, it seems to fetch from no-official url source. But it could be nice to have feedback from them 😊 |
@uilianries thanks for pinging. I'm working on automating the packaging of different llvm components and versions, https://gitlab.com/Manu343726/clang-conan-packages The master branch has a pipeline that builds and deploys version 3.8.0 to my conan repo. To support following versions, I had to change the recipes so sources are fetched from the right repository (The old @polysquare repository has 3.8.0 sources only). All recipes have been modernized to support latest conan versions. Also, I managed to get the recipes working on Windows with VS2017, the repository already includes binary packages with release and debug VS2017 builds. Due to space limitations of my bintray account, no debug packages are included for linux. All the refactoring work needed to support more versions and share code between recipes is being done here https://gitlab.com/Manu343726/clang-conan-packages/tree/common_recipe_scripts Once I get 3.8.0 to 6.0.0 working and the recipes sharing common code (There's a lot of redundant and/or unused code in the current recipes inherited from the original repo) I will request for admission in conan-center. ETA 1-2 months |
About the sources not coming from official LLVM repository, I would not like to have users of the recipes to rely on LLVM releases server (It would not be the first time that server is down bc of people fetching releases for their CI from there). So my build pipeline will include a step to fetch sources and deploy them to bintray, so all conan users rely entirely on jfrog infrastructure (Easier to report and get things fixed quickly :) ). |
@Manu343726 Nice work! Do you plan to package also others llvm's sub-projects ? (openMP, polly, lldb, licxxabi ...). Imo the idea would be to have a llvm package on top of every llvm projects and every sub-projects as submodules (so the packages are the smallest possible) like with boost. Btw you can ask bintray to add more space on your account so you can also include debug and such, they are really receptives, tell me if you need any contact I already have an open thread with them juste about uploading size of LLVM in bintray. @SSE4 @solvingj wdyt ? Should we stop the current work on llvm packaging and assist @Manu343726 on his work ? |
@lucienboillod in general yes, I'd like to avoid resource duplication as much as possible, so let's assist @Manu343726 and build other sub-projects on top of his llvm-core package. |
@lucienboillod my idea was to package the components needed by libclang, to then make a libclang meta-package on top of the clang one. But I will be glad to add other LLVM modules to the pipeline as well. The pipeline is auto-generated from a declarative YAML file, similar to what conan-package-tools does. The template also generates jobs that package python packages into conan packages, autogenerating the recipes. For a python package directory like:
the job generates the recipes needed to make that python modules available as a conan python package. Both features are implemented by two python scripts in the repository. There's also a job to mirror the LLVM sources in a bintray repo. So adding more LLVM modules to the pipeline would be as easy as adding more entries to the LLVM modules are cmake-modularized, all module dependencies are requested through standard |
BTW it should be possible to check for recipe changes in the CI, so that jobs that build unchanged packages are skipped (That would help a lot with my electricity bills :P). Also, the CI generator script only implements the .gitlab.ci file, but it should be easy to implement other services such as appveyor and travisci as well. |
I definitely need support guys xD. Ping @danimtb |
I've finaly got help from the jfrog side, my limits have been increased but my account cannot cope with all the packages in any way. So I've removed all binaries from my account and fired a reduced build. Once the package is submitted to conan-center we could increase the number of package variants built by the pipeline. |
Hello @Manu343726 , how is going the project ? 😄 need any help ? |
Sorry for the late response I've managed to have the four recipes working sharing code with a conan python package (For things like install, sources download, etc). After testing the package with one of my projects I noticed I'm not exporting the llvm libraries as part of cpp_info.libs. If I use CMake relying on the standard LLVMConfig scripts everything works. Now the question is how I should package each llvm library so that its dependencies are correctly handled by conan. LLVM is a monolithic monster much like boost. |
What is the current state of this? I'm specifically interested in a recipe for libclang |
Hi people, I'm back working on this, currently in the last stage to get a PoC: A recipe for a library (cppast) that depends on libclang. The main issue with this package are the different LLVM modules, which I'm exporting as part of
As you can see, clang modules are found by the loader, but LLVM's are not. |
Found the problem: The clang executable RPATHs were filled with full paths of my CI builder, which means I cannot use precompiled packages. If I build from sources everything works fine. |
If you got some stable Conan recipes we would happily taking them in to Bincrafters and give you push rights to the new repository. Are some dependencies already working? It's kinda hard to get an overview as you have several conanfiles per repository and have some custom build helpers. Also are you GitHub repositories outdated and you continued working on Gitlab exclusively? |
This comment has been minimized.
This comment has been minimized.
The situation is as follows:
TL;DR: I would happily transfer control to the bincrafters community and write llvm/clang distribution packages from my work following standard practices (conan-package-tools, python requires, travis ci, etc). |
FYI gitlab public runners are much more reliable now, the full pipeline worked wo issues https://gitlab.com/Manu343726/clang-conan-packages/pipelines/33749222 |
Some bad news: Recent versions of compiler-rt (I'm building 6.0.0) cannot be built standalone with MSVC due to a bug in the cmake scripts https://bugs.llvm.org/show_bug.cgi?id=38586 |
Ok people, some news about the llvm/clang/libclang packages. I've managed to get a setup working, by packaging llvm, compiler-rt, and clang as "component" packages (Full builds of llvm, compiler-rt, and clang projects respectively). Then one meta-package is defined for each module in llvm, compiler-rt, and clang. Those meta-packages package libraries and headers corresponding to the module only (copied from the corresponding "component" package folder, which is required as a I have uploaded the recipes here https://bintray.com/manu343726/conan-packages/, there's a libclang meta-package that I'm using for testing purposes that you should be able to require and link statically and dynamically. There are pre-built packages for linux gcc 8.1 stdlibc++11 |
I've only meta-packaged modules required by libclang, some common modules such as libtooling are missing. |
This comment has been minimized.
This comment has been minimized.
I haven't fully understood yet how you structure the recipes, but would you say you can split them into several smaller repositories as soon as you start to use python_requires (you have written about this earlier)? It's quite uncommon to have a huge monolithic repository for several Conan recipes, but then again I don't understand your structure yet so I can't really say what would be reasonable... |
I'm currently working on a CI stage for testing tinyrefl package, and since tinyrefl depends on libclang I've followed the ugly path of pushing the compiled llvm/clang packages from that jobs once they're compiled. But I would like to setup a proper pipeline for my clang-conan-packages repo with conan-package-tools. Do you have any experience using conan-package-tools with gitlab-ci? |
I have! Usually I use this template: https://gitlab.com/snippets/1799856 The build.py referred there looks like:
But you can change anything. |
FYI I finally wrote the CI pipeline using my tool. See https://gitlab.com/Manu343726/clang-conan-packages/pipelines/44984999 |
I'm dropping Clang 4.0.0 and 5.0.0 from the packaging pipeline due to some unresolved dependencies of the libclang static package (It's weird because the dynamic versions are working fine). I don't want to delay the release of tinyrefl more and since I'm not interested in Clang < 6.0.1 I'm not motivated enough to keep fighting with this issue. Anyway feel free to PR to try to fix the problem. |
@Manu343726 I believe you have done an excellent job here. The only problem is now how we do integrate this into Bincrafters. This is such a huge monster that it is hard to overview and to tackle Some basic questions to begin with:
Obviously both has advantages and disadvantages. I guess at least for the beginning we should keep the monolithic repository and start to adapt it to Bincrafters/GitHub/Azure Pipelines (Travis can't handle that many jobs) @Manu343726 Are you still interested to get this adopted by Bincrafters? If so, are you fine with putting your scripts/recipes under the MIT license? |
I think this is the best option since there's a lot of meta-packages with dependencies between them.
And also port the packaging scripts to conan-package-tools.
I thought they were already. Let me check. |
@Manu343726 It seems like some sub-directories do have a |
@Manu343726 Ping 😄 |
This issue has been automatically closed because there has been no response to our request for more information or feedback from the original author. With only the information that is currently in the issue, we don't have enough information to take further action. Please reach out if you have or find the answers we need so that we can investigate further. |
@Manu343726 If at any point in the future you could add a LICENSE file to the root of your repository, that would be awesome. Otherwise we can't build upon our work. We would also need a new champion moving such a gigantic project forward. Closing for now. Wishlist: conan-io/wishlist#197 |
@Croydon |
LLVM is a collection of several projects (LLVMCore, Clang, libcxx ...). After trying to build everything at once, we figured out that was really too huge and we decided to package the llvm core at first step, and then others sub-projects as submodule (like we do for Qt or Boost).
The text was updated successfully, but these errors were encountered: