llvm.org: reduce installed size

[jonchang]

This may help shrink binary sizes by avoiding static linking.

[what-the-diff]

PR Summary

• Enhancement to the Build Process
  A new flag, -DLLVM_LINK_LLVM_DYLIB=ON, has been added to the build section in our configuration file (package.yml). This modification will allow the building process to incorporate LLVM's dynamic libraries, ensuring our application is always on par with latest features and improvements in LLVM.

[jonchang]

On darwin+aarch64 it looks like this significantly decreases the size of the unpacked artifact:

before: 3.1G
after: 766M

Will dig into the Linux failures, which I believe can be addressed by tweaking some options.

[mxcl]

I tested locally and linux passes if we remove the other linux specific ARGS.

$ du -hd1 prefixes/linux/llvm.org/v16/
169M	prefixes/linux/llvm.org/v16//bin
 44K	prefixes/linux/llvm.org/v16//libexec
 46M	prefixes/linux/llvm.org/v16//include
683M	prefixes/linux/llvm.org/v16//lib
180K	prefixes/linux/llvm.org/v16//share
899M	prefixes/linux/llvm.org/v16/

[jonchang]

Nice work! (I still haven't gotten tea set up on my Linux box yet.) The only thing left to do is to consider whether this change is desirable. My understanding is that this will lead to a small performance penalty for compiling with LLVM, which is maybe not such a big deal but at the margins (e.g., compiling Firefox or Chrome) would be an issue.

[mxcl]

(I still haven't gotten tea set up on my Linux box yet.

pkg -L build llvm.org in your pantry clone runs it via docker in case you weren't aware of this feature.

[mxcl]

if we remove the other linux specific ARGS.

this was ofc just a test to see if it helped, those ARGS were added by @jhheider for a reason presumably.

[mxcl]

-- Linker detection: unknown
CMake Error at /tmp/eb4c19d9/llvm/cmake/modules/CheckProblematicConfigurations.cmake:14 (if):
  if given arguments:

    "STREQUAL" "MSVC"

  Unknown arguments specified
Call Stack (most recent call first):
  /tmp/eb4c19d9/llvm/cmake/modules/HandleLLVMOptions.cmake:10 (include)
-- Configuring incomplete, errors occurred!
  CMakeLists.txt:157 (include)

I got this once too, but only once. Worked a second time after cleaning. So… that's weird.

edit: no I'm wrong it only builds if we remove the if from the top of the build script thus removing compiler-rt from the runtime list. And I assume we need that for some reason.

[jonchang]

I suspect that some of the build opts conflict between building llvm/clang/lldb and compiler-rt. I'm testing something out now to spin off compiler-rt into compiler-rt.llvm.org (like we already do with openmp.llvm.org).

My preference would actually be to try to split these packages up fully (such that we have an lldb.llvm.org and clang.llvm.org, also should help reduce build sizes) but since there's a fair number of things that already depend on llvm.org I'll hold off on that.

[jhheider]

1. complier-rt was asked for here
2. splitting it all up seems like a great idea
3. you can easily keep things that depend on llvm.org working by making llvm.org depend on the things you split out (i'm sure you know this; this is the meta-package idea to help ease transition)

[jonchang]

all good points! forgot about meta-packages / package groups, will give it a shot

[jonchang]

Currently building/testing clang and lld but here's the current plan:

• llvm.org/core will contain the "core" LLVM utilities and libraries
• clang.llvm.org has clang, which depends on compiler-rt.llvm.org and llvm.org/core
• lld.llvm.org has lld
• The llvm.org metapackage will point to clang, lld, compiler-rt, and core
• Additional LLVM subprojects and fixing old dependencies on llvm.org to the appropriate subprojects

[jhheider]

Hm. That sounds amazing. We might have a version-locking issue. @mxcl we could handle clang.llvm.org: ${{version}} moustaches in dependencies: but that'd just ruin the resolution algo. Thoughts there?

[jonchang]

We could just have everything that currently depends on llvm.org to instead depend on llvm.org/core plus clang.llvm.org (I doubt anything actually used lld). That way there would be nothing in pantry that directly depended upon llvm.org and thus avoiding problems with dependency resolution.

[jhheider]

That's likely the right move. We could, in tealib make llvm.org an alias for the other two to ease adoption. and not trigger a rebuild of 98% of the pantry which I need to remember to cancel before i hate everything.

which might mean renaming the old package and keeping it on hand, in case it's needed. @ABevier would hate us for that, and rightly. this change is going to require precision.

[mxcl]

seems like a very well thought through organization! great work. Will this pkg become llvm.org/core?

[jonchang]

Yes, currently testing to see if all the pieces work well together on darwin+linux. Didn't want to push yet to avoid tying up CI for something that might not work

[jonchang]

Okay, figured clang out, it was a bug in llvm's build system. Testing now...

[jonchang]

This is getting pretty close to done, the package yamls are close to their final state. I have tested this on macOS arm64, and am in the process of doing the same for linux intel and arm. I need to come up with a better test for compiler-rt and package some other llvm subprojects (and take a look at openmp since that's already packaged separately)

[mxcl]

time to support no src no build pkgs probs! what you think @jhheider

[jhheider]

Meta packages are a thing. We could probably use them just in resolution. If they don't have a build: key we can assume they're just dependency bundles.

The pantry workflow just needs to yeet them through.