brew install llvm --with-python --with-lldb --with-clang fails

[xu-cheng]

https://gist.github.com/xu-cheng/41ca3c4ad95a72351e39

cc @DomT4, @tamird

[xu-cheng]

The real error:

/tmp/llvm20150511-61002-1y7lnwn/llvm-3.6.0.src/tools/lldb/tools/debugserver/source/MacOSX/dbgnub-mig.defs:13:10: fatal error: 'DNBConfig.h' file not found
#include "DNBConfig.h"
         ^
1 error generated.

And some related info: http://sushihangover.github.io/building-lldb-on-mac-os-x/

[DomT4]

Ugh. LLVM 😭.

You found about the one clear reference that I did as well. The debugserver xcodeproject inside the directory it builds from contains this block:

/* Begin PBXShellScriptBuildPhase section */
        26CE05C7115C36870022F371 /* ShellScript */ = {
            isa = PBXShellScriptBuildPhase;
            buildActionMask = 2147483647;
            files = (
            );
            inputPaths = (
                "$(SRCROOT)/scripts/dbgnub-config.pl",
            );
            outputPaths = (
                "$(DERIVED_FILE_DIR)/DNBConfig.h",
            );
            runOnlyForDeploymentPostprocessing = 0;
            shellPath = /bin/sh;
            shellScript = "perl -x scripts/dbgnub-config.pl\n";
        };

Maybe that's not getting run, or maybe it's being placed in the wrong location? If I run the project inside xcode I get lots of 'DNBConfig.h' file not found message as well, but could be unrelated. Presume this problem doesn't exist on the autotooled latest LLVM in Versions?

[tamird]

Does this repro with just brew install llvm --with-clang --with-lldb?

[tamird]

This actually failed for me in a whole different way:

cd /tmp/llvm20150511-65141-166fd89/bin && CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate codesign --force --sign lldb_codesign debugserver
lldb_codesign: no identity found
make[2]: *** [bin/debugserver] Error 1
make[1]: *** [tools/lldb/tools/debugserver/source/MacOSX/CMakeFiles/debugserver.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

[tamird]

I'm pretty sure this is not a cmake regression - the code signing thing might be a 3.6 regression though.

[mjbshaw]

Should homebrew roll back the llvm formula to 3.6.0 until it's fixed upstream? Or perhaps error out if --with-clang --with-lldb is specified (until it's fixed)?

Or should users be expected to not attempt to upgrade llvm?

The main pain here is waiting until the end of building llvm to hit the error from the build system. Burning that much time and CPU kinda sucks from a UX perspective, so a way to soften the blow until things are fixed would be nice.

[DomT4]

Would personally like to remove the lldb option from the CMake build for now, but has been some pushback on that.

[DomT4]

Gonna ping @Homebrew/owners @bfontaine for some LLVM discussions. Let's make a decision about whether or not we're happy keeping the broken option around in the main Homebrew/homebrew formula.

[tdsmith]

I'm missing some context. What discussion would you like to have? Is it correct that the build fails if and only if both --with-lldb and --with-clang are both specified? (In that case, it makes sense to me to odie in install if both options are specified, as a stopgap.) Why remove --with-lldb completely?

Are the failure conditions identical for 3.6.0 and 3.6.1 or does 3.6.1 introduce new failure conditions?

[DomT4]

A general discussion I guess about what we do with lldb in LLVM - It's an extremely popular add-on to the LLVM formulae, but since we switched over to the CMake build (Which solved other problems) the lldb element has been consistently refusing to compile.

are both specified?

Building --with-lldb requires --with-clang and the formula enforces that.

Are the failure conditions identical for 3.6.0 and 3.6.1 or does 3.6.1 introduce new failure conditions?

Identical. It's the codesign failure. We hit some DNBConfig issues with 3.6.0 which you can see further up this issue, but I haven't seen that reproduced on 3.6.1 yet.

[xu-cheng]

I'm inclined to fix the bug(reporting to upstream) rather than simple killing it. Becuase killing it doesn't solve anything. People who need lldb will continue to file issue to complain its missing. And given the popularity of lldb it wouldn't be just a few.

[mjbshaw]

Are the failure conditions identical for 3.6.0 and 3.6.1 or does 3.6.1 introduce new failure conditions?

Identical. It's the codesign failure. We hit some DNBConfig issues with 3.6.0 which you can see further up this issue, but I haven't seen that reproduced on 3.6.1 yet.

I was able to successfully build 3.6.0 with --with-lld --with-clang --with-lldb, but I can't build 3.6.1, so I'm not sure things are 100% identical. That's partially one reason why I separately filed #40359.

[greymouser]

I wanted to bring the following up, since from this issue and related issues it seems that it's understood this is a code signing issue, but I'm not sure everyone understands the why of it.

It's well documented in LLDB that lldb needs to be code-signed. This isn't some peculiarity of LLDB -- GDB has to do this as well to work on Darwin. Darwin is the peculiar aspect to all this.

The mach kernel has a task_for_pid() call that basically allows one process to access another. The access policy for this call is enforced on Darwin/MacOSX by the taskgated daemon -- man taskgated. As defined in /System/Library/LaunchDaemons/com.apple.taskgated.plist, taskgated on my Yosemite machine is launched only with -s. This means that when task_for_pid gets called, this service will be checked pretty much always.

So, it's not weird at all that LLDB requires code-signing to operate correctly as accessing another process is it's job. What is weird and frustrating is that:

• The LLDB build process requires the code signing identity to be called "lldb_codesign" precisely, and
• With security (and probably certtool to create the cert), there probably can and should be an assistive script to help people create this identity.

After giving up on 3.6.1 out of brew last night -- dying at 98% is killer, btw -- I built an LLVM release from svn/git. LLVM, as I learned, is both over-documented and poorly documented thanks to documentation atrophy. However, the LLDB docs (above) pretty much spelled out what I needed to do with respect to code-signing.

After creating an svn/git build, out of curiosity, I ran the brew install again and it just worked (as it had access to the lldb_codesign identity I had created).

I also came across the CODESIGN_TOOLS make var (e.g make CODESIGN_TOOLS={0,1}) while trying to debug this. While promising, that seems to handle signing for all tools optionally, whereas lldb is deciding to always need signing (not sure, as I was only curious about debugserver).

So, while it's ugly, this could probably be solved with a prebuild/preinstall type feature that checks for the existence of (and maybe validity of) the lldb_codesign identity. Brew does have post install procedures that users are expected to manually take care of, so I suppose this wouldn't be beyond the pale.

If someone is compiling LLVM with options, there's a good chance that an error/warning message with some documentation and asking them to retry would be greatly appreciated over a long running build that dies at 98%.

I hope this information is helpful. Cheers!

[tamird]

@greymouser nice writeup, thank you. Unfortunately, there's nothing we can do about the property of dying at 98%, and I'm totally with you that it sucks, but that's how the LLVM build system works; the codesign check happens lazily.

I'm going to look into CODESIGN_TOOLS and the possibility of following the script programmatically.

[tdsmith]

Unfortunately, there's nothing we can do about the property of dying at 98%

Homebrew can refuse to start the build if we can detect the failure condition in advance, like @DomT4 is suggesting.

[tamird]

Right, but then homebrew's detection of the failure condition has to track logic in LLVM's build system.

[greymouser]

@tamird I may have conflated the situation earlier mentioning CODESIGN_TOOLS.

I believe CODESIGN_TOOLS (and the corresponding TOOL_CODESIGN_IDENTITY setting it uses) is intended for distributors (e.g. Apple) to sign all binaries created by the build. This logic is in ./Makefile.rules ~L1421.

However, this is not the case (i.e. separate from) for the debugserver target in the lldb tool directory. In ./tools/lldb/tools/debugserver/source/MacOSX/CMakeLists.txt ~L55, the codesigning is set to happen whenever the debugserver target gets built with the lldb_codesign identity -- i.e always. (And as noted earlier, that's a Darwin quirk required for its security scheme, and not something that can be turned off.)

While the actual name of the identity "lldb_codesign" might change, the test and check is easy enough ... something like the following from the build root should return enough info to stop the build early enough, if the user does not have a valid lldb_codesign cert in place:

build $ ident_name=$(make -pn debugserver |ruby -pe 'ident="none"; while gets ; next unless $_ =~ /^.*--sign\s{1}([[:word:]]+)\s+/; ident=$1; end; puts ident')
build $ ident_id=$(security find-certificate -c $ident_name -Z |head -n 1| cut -d' ' -f3)
build $ has_lldb_codesign=$(security -q find-identity -v -p codesigning |grep -q $ident_id && echo yes || echo no)
build $ echo $has_lldb_codesign 
yes

[tamird]

@greymouser open a PR?

[DomT4]

We need to do something about this soon. It is horrible UX to allow the build to get to ~90% and have it fail at that point, especially given the length of an LLVM --with-clang --with-lldb build. If there's no better logic available, I'll push something like:

diff --git a/Library/Formula/llvm.rb b/Library/Formula/llvm.rb
index cad8897..78e7856 100644
--- a/Library/Formula/llvm.rb
+++ b/Library/Formula/llvm.rb
@@ -69,6 +69,7 @@ class Llvm < Formula
   option "with-rtti", "Build with C++ RTTI"
   option "with-python", "Build Python bindings against Homebrew Python"
   option "without-assertions", "Speeds up LLVM, but provides less debug information"
+  option "with-confirmed-codesign", "I have the necessary codesigning identity for LLDB"

   deprecated_option "rtti" => "with-rtti"
   deprecated_option "disable-assertions" => "without-assertions"
@@ -91,6 +92,15 @@ class Llvm < Formula
     # Apple's libstdc++ is too old to build LLVM
     ENV.libcxx if ENV.compiler == :clang

+    if build.with?("lldb") && build.without?("confirmed-codesign")
+      raise <<-EOS.undent
+        lldb_codesign identity must be available to build with LLDB.
+        See: https://llvm.org/svn/llvm-project/lldb/trunk/docs/code-signing.txt
+
+        Please confirm that it is by passing `--with-confirmed-codesign` to brew.
+      EOS
+    end

For discussion tomorrow. Welcome less hacky workarounds - Whether here or upstream.

[tamird]

That change looks good to me. There is no sane programmatic way to create certificates, the clicky process seems to be the only way.