Issues with Ubuntu build from source instructions Bionic 18.04

[willfindlay]

Hey all. I'm normally an Arch user, but I've been trying to set up CI for one of my projects, so I'm forced to use Ubuntu 18.04 here.

For some reason when building bcc from source I keep getting these LLVM-related linker errors, despite having the dependencies installed and despite cmake appearing to locate the dependencies properly. Am I missing something? Thanks.

All relevant output is below.

Edit: By the way, I've tried with other versions of LLVM as well, up to and including 10.

Setup script:

# Install dependencies
sudo apt install -y gcc g++ bison build-essential cmake flex git libedit-dev clang-9 llvm-9 libllvm9 llvm-9-dev libclang-9-dev python zlib1g-dev libelf-dev libfl-dev

# Install bcc
git clone https://github.com/iovisor/bcc.git /tmp/bcc
mkdir /tmp/bcc/build
pushd /tmp/bcc/build 
cmake -DPYTHON_CMD=python3 .. 
make -j $(nproc)
sudo make install
pushd src/python
make -j $(nproc)
sudo make install 
popd 
popd

Relevant cmake output:

semaphore@semaphore-vm:/tmp/bcc/build$ cmake ..
-- Latest recognized Git tag is v0.17.0
-- Git HEAD is 2c61168a59e6590bf5f540a96045bb9d74771df4
-- Revision is 0.17.0-2c61168a
-- Found LLVM: /usr/lib/llvm-9/include 9.0.0
-- Could NOT find LuaJIT (missing: LUAJIT_LIBRARIES LUAJIT_INCLUDE_DIR)
CMake Warning at tests/python/CMakeLists.txt:6 (message):
  Recommended test program 'arping' not found


CMake Warning at tests/python/CMakeLists.txt:10 (message):
  Recommended test program 'netperf' not found


CMake Warning at tests/python/CMakeLists.txt:16 (message):
  Recommended test program 'iperf' or 'iperf3' not found


-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/bcc/build

Linker errors:

semaphore@semaphore-vm:/tmp/bcc/build$ make
[  6%] Built target bpf-shared
[ 11%] Built target bpf-static
[ 13%] Built target bcc-loader-static
[ 15%] Built target clang_frontend
[ 16%] Built target api-static
[ 17%] Built target usdt-static
[ 21%] Built target b_frontend
[ 27%] Built target bcc-shared
[ 34%] Built target bcc-static
[ 34%] Built target bcc_py_python3
[ 35%] Built target bps
[ 36%] Linking CXX executable LLCStat
../../src/cc/libbcc.a(bpf_module_rw_engine.cc.o): In function `ebpf::finish_sscanf(llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>&, std::vector<llvm::Value*, std::allocator<llvm::Value*> >*, std::string*, std::map<std::string, llvm::Value*, std::less<std::string>, std::allocator<std::pair<std::string const, llvm::Value*> > > const&, bool)':
bpf_module_rw_engine.cc:(.text+0x77d): undefined reference to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)'
bpf_module_rw_engine.cc:(.text+0x970): undefined reference to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)'
bpf_module_rw_engine.cc:(.text+0xbb3): undefined reference to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)'
bpf_module_rw_engine.cc:(.text+0xd9e): undefined reference to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)'
bpf_module_rw_engine.cc:(.text+0xf70): undefined reference to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)'
../../src/cc/libbcc.a(bpf_module_rw_engine.cc.o):bpf_module_rw_engine.cc:(.text+0x10c5): more undefined references to `llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)' follow
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::genParamIndirectAssign(clang::FunctionDecl*, std::string&, char const**)':
b_frontend_action.cc:(.text+0x163c): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::VisitBinaryOperator(clang::BinaryOperator*)':
b_frontend_action.cc:(.text+0x2f9e): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::genParamDirectAssign(clang::FunctionDecl*, std::string&, char const**)':
b_frontend_action.cc:(.text+0x785a): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::VisitFunctionDecl(clang::FunctionDecl*)':
b_frontend_action.cc:(.text+0x886e): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BFrontendAction::EndSourceFileAction()':
b_frontend_action.cc:(.text+0x9664): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o):b_frontend_action.cc:(.text+0x9a8c): more undefined references to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const' follow
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::VisitCallExpr(clang::CallExpr*)':
b_frontend_action.cc:(.text+0xc3a3): undefined reference to `clang::QualType::getAsString() const'
b_frontend_action.cc:(.text+0xc3af): undefined reference to `clang::DeclarationName::getAsString() const'
b_frontend_action.cc:(.text+0xc819): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0xd18a): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0xd371): undefined reference to `clang::QualType::getAsString() const'
b_frontend_action.cc:(.text+0xd37d): undefined reference to `clang::DeclarationName::getAsString() const'
b_frontend_action.cc:(.text+0xd5f5): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0xd951): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0xdb3d): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0xdcb1): undefined reference to `clang::QualType::getAsString() const'
b_frontend_action.cc:(.text+0xdcbd): undefined reference to `clang::DeclarationName::getAsString() const'
b_frontend_action.cc:(.text+0xdedf): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::ProbeVisitor::VisitUnaryOperator(clang::UnaryOperator*)':
b_frontend_action.cc:(.text+0x114fa): undefined reference to `clang::QualType::getAsString() const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::ProbeVisitor::VisitArraySubscriptExpr(clang::ArraySubscriptExpr*)':
b_frontend_action.cc:(.text+0x11918): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0x11966): undefined reference to `clang::QualType::getAsString() const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::ProbeVisitor::VisitMemberExpr(clang::MemberExpr*)':
b_frontend_action.cc:(.text+0x14986): undefined reference to `clang::Rewriter::getRewrittenText(clang::CharSourceRange) const'
b_frontend_action.cc:(.text+0x149b7): undefined reference to `clang::QualType::getAsString() const'
b_frontend_action.cc:(.text+0x149ee): undefined reference to `clang::QualType::getAsString() const'
../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeConsumer::HandleTranslationUnit(clang::ASTContext&)':
b_frontend_action.cc:(.text+0x176d1): undefined reference to `clang::QualType::getAsString() const'
../../src/cc/libbcc.a(bpf_module.cc.o): In function `ebpf::BPFModule::dump_ir(llvm::Module&)':
bpf_module.cc:(.text+0x434f): undefined reference to `llvm::createPrintModulePass(llvm::raw_ostream&, std::string const&, bool)'
../../src/cc/libbcc.a(bpf_module.cc.o): In function `ebpf::BPFModule::run_pass_manager(llvm::Module&)':
bpf_module.cc:(.text+0x4482): undefined reference to `llvm::createPrintModulePass(llvm::raw_ostream&, std::string const&, bool)'
../../src/cc/libbcc.a(bpf_module.cc.o): In function `llvm::RTDyldMemoryManager::getSymbolAddress(std::string const&)':
bpf_module.cc:(.text._ZN4llvm19RTDyldMemoryManager16getSymbolAddressERKSs[_ZN4llvm19RTDyldMemoryManager16getSymbolAddressERKSs]+0x4): undefined reference to `llvm::RTDyldMemoryManager::getSymbolAddressInProcess(std::string const&)'
../../src/cc/libbcc.a(bpf_module.cc.o):(.data.rel.ro._ZTVN4ebpf15MyMemoryManagerE[_ZTVN4ebpf15MyMemoryManagerE]+0x50): undefined reference to `llvm::SectionMemoryManager::finalizeMemory(std::string*)'
../../src/cc/libbcc.a(bpf_module.cc.o):(.data.rel.ro._ZTVN4ebpf15MyMemoryManagerE[_ZTVN4ebpf15MyMemoryManagerE]+0x90): undefined reference to `llvm::RTDyldMemoryManager::getPointerToNamedFunction(std::string const&, bool)'
../../src/cc/libbcc.a(bcc_debug.cc.o): In function `ebpf::SourceDebugger::dump()':
bcc_debug.cc:(.text+0x94f): undefined reference to `llvm::TargetRegistry::lookupTarget(std::string const&, std::string&)'
../../src/cc/libbcc.a(json_map_decl_visitor.cc.o): In function `ebpf::BMapDeclVisitor::genJSONForField(clang::FieldDecl*)':
json_map_decl_visitor.cc:(.text+0xd14): undefined reference to `llvm::APInt::toString(unsigned int, bool) const'
../../src/cc/frontends/clang/libclang_frontend.a(tp_frontend_action.cc.o): In function `ebpf::TracepointTypeVisitor::VisitFunctionDecl(clang::FunctionDecl*)':
tp_frontend_action.cc:(.text+0x6d5): undefined reference to `clang::QualType::getAsString() const'
collect2: error: ld returned 1 exit status
examples/cpp/CMakeFiles/LLCStat.dir/build.make:165: recipe for target 'examples/cpp/LLCStat' failed
make[2]: *** [examples/cpp/LLCStat] Error 1
CMakeFiles/Makefile2:795: recipe for target 'examples/cpp/CMakeFiles/LLCStat.dir/all' failed
make[1]: *** [examples/cpp/CMakeFiles/LLCStat.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

[yonghong-song]

Just tried with a fresh ubuntu 18.04 (kernel 4.15.0-122-generic) and follow the above script, and I did not see any issues.

[willfindlay]

This is very weird indeed... I suspect there may perhaps be some weird default configuration in the Semaphore VM that is causing trouble. Thanks for looking into it @yonghong-song, I'll keep investigating on my end.

[willfindlay]

Closing this for now, as I've not had time to investigate and instead moved to a different CI service