Using LLVM (Clang++) to compile PALISADE for BPF target

[OluAgunloye]

@jackcmay

Hello,

This is a continuation of the issues I'm experiencing with PALISADE:

`PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0. Program arguments: /usr/bin/clang -target bpf -DMATHBACKEND=2 -DPARALLEL -I/root/palisade-development/third-party/include -I/root/palisade-development/third-party/cereal/include -I/root/palisade-development/third-party/google-test/googletest -I/root/palisade-development/third-party/google-test/googletest/include -I/root/palisade-development/src/core/include -I/root/palisade-development/build/src/core -I/root/palisade-development/src/core/lib -I/usr/include/c++/10 -I/root/include/x86_64-linux-gnu/c++/10 -D__x86_64__ -D__LP64__ -I/root/include/x86_64-linux-gnu -Wall -Werror -O3 -DPALISADE_VERSION=1.11.4 -Wno-unused-private-field -Wno-shift-op-parentheses -O3 -DNDEBUG -fPIC -std=gnu++11 -o CMakeFiles/coreobj.dir/lib/encoding/coefpackedencoding.cpp.o -c /root/palisade-development/src/core/lib/encoding/coefpackedencoding.cpp

1. parser at end of file
2. Code generation
3. Running pass 'Function Pass Manager' on module '/root/palisade-development/src/core/lib/encoding/coefpackedencoding.cpp'.
4. Running pass 'Live Variable Analysis' on function '@_ZN8lbcrypto18CoefPackedEncoding6EncodeEv'
   #0 0x00007f68cfa98ae3 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xbd9ae3)
   [LLD][BPF] Enable BPF shared object creation #1 0x00007f68cfa96df0 llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xbd7df0)
   Fix Github Actions #2 0x00007f68cf9e7e76 (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xb28e76)
   [BPF] Disable llvm tests incompatible with Solana BPF backend #3 0x00007f68d730a1f0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x141f0)
   [BPF] Allow selectively disable compiler builtins for BPF target #4 0x00007f68cfd43e6c llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xe84e6c)
   [BPF] Map signed division operation to corresponding intrinsic function #5 0x00007f68cfd443cf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xe853cf)
   [SOL][BPF] Adjust BPF tests #6 0x00007f68cfda551e llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xee651e)
   [SOL][BPF] Allow misaligned loads #7 0x00007f68cfbc058d llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd0158d)
   [SOL][BPF] Add BPF compiler-rt builtins #8 0x00007f68cfbc5f73 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd06f73)
   [SOL][BPF] Enable Solana BPF extensions as subtarget feature #9 0x00007f68cfbc0bdf llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd01bdf)
   Fixes required for Solang #10 0x00007f68d5bf26f6 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_deletellvm::raw_pwrite_stream >) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x15446f6)
   [SOL][BPF] Remove debug printf #11 0x00007f68d5e8b8af (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x17dd8af)
   [SOL][BPF] Disable debug info when solana feature flag is set #12 0x00007f68d4fe6b64 clang::ParseAST(clang::Sema&, bool, bool) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x938b64)
   Upgrade to rust 1.52.1 #13 0x00007f68d657fd48 clang::FrontendAction::Execute() (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1ed1d48)
   [SOL] Set max stores per mem func depending on the target features #14 0x00007f68d650d7d1 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1e5f7d1)
   Add type attributes to LLVM C API #15 0x00007f68d65e2d62 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1f34d62)
   Using LLVM (Clang++) to compile PALISADE for BPF target #16 0x00000000004127e2 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/usr/bin/clang+0x4127e2)
   [SOL] Prevent breaking ISelDAG connectivity on replacing loads by con… #17 0x0000000000410b5e (/usr/bin/clang+0x410b5e)
   [SOL] Revert to R_BPF_64_32 until support for R_BPF_64_ABS32 added #18 0x00007f68d62296c2 (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b7b6c2)
   [SOL] Add sbf-solana-solana target triplet #19 0x00007f68cf9e7c4d llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xb28c4d)
   [SOL] Turn on solana feature for SBF target by default #20 0x00007f68d6228eb9 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optionalllvm::StringRef >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b7aeb9)
   [SOL] Register SBF asm parser #21 0x00007f68d61fe46f clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b5046f)
   [SOL] Add missing SBF conditions to match BPFEL target #22 0x00007f68d61fe827 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b50827)
   [SOL] add support for (pseudo) atomics to SBF #23 0x00007f68d621320c clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b6520c)
   [SOL] disable llvm.bpf.load.* intrinsics on SBF #24 0x0000000000410434 main (/usr/bin/clang+0x410434)
   [SOL] re-enable debug info on SBF #25 0x00007f68ce977565 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x28565)
   Dynamic stack frame size support #26 0x000000000040dd1e _start (/usr/bin/clang+0x40dd1e)
   clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
   Ubuntu clang version 12.0.0-3ubuntu1~21.04.2
   Target: bpf
   Thread model: posix
   InstalledDir: /usr/bin
   clang: note: diagnostic msg:

---

​
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/coefpackedencoding-e3e0eb.cpp
clang: note: diagnostic msg: /tmp/coefpackedencoding-e3e0eb.sh
clang: note: diagnostic msg:
​`

[OluAgunloye]

@jackcmay This seems almost exclusively a Clang++ issue too. Is there a g++ to BPF target for compiling LLVM?

[OluAgunloye]

Stuck here, @jackcmay . Streams seem to be a large issue (all types).

In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/complex:242: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/sstream:184: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/istream:163: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/ostream:140: /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:750:26: error: use of undeclared identifier 'strtoll_l' long long __ll = strtoll_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:790:35: error: use of undeclared identifier 'strtoull_l' unsigned long long __ll = strtoull_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:819:12: error: use of undeclared identifier 'strtof_l' return strtof_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:825:12: error: use of undeclared identifier 'strtod_l' return strtod_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:831:12: error: use of undeclared identifier 'strtold_l' return strtold_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:1105:9: error: use of undeclared identifier 'sscanf_l' if (__libcpp_sscanf_l(__buf.c_str(), _LIBCPP_GET_C_LOCALE, "%p", &__v) != 1) ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/__bsd_locale_defaults.h:34:61: note: expanded from macro '__libcpp_sscanf_l' #define __libcpp_sscanf_l(...) sscanf_l(__VA_ARGS__)

[animber-coder]

@jackcmay can clang compile any std c++ project with solana bpf-tools on host target x86_64-linux-gnu?
where can we get bpf stdc++ headers and libraries?

[animber-coder]

@jackcmay can solana-labs:solana-rustc branch solve this problem?

[jackcmay]

We don't provide any bpf compiled standard libs for C/C++ compilation yet. In order to support that the libraries would need to be modified to remove all system calls as well as probably a bunch of other changes to get it to build. We have been hesitant to pick a standard library since the C space is so fragmented. Picking glibc might be ok. @cpp-guru do you have a recommendation? @dmakarov thoughts on us releasing a bpf/solana compatible stdlib?

[dmakarov]

We may try compiling newlib for BPF configuring syscalls so that they don't conflict with our runtime restrictions. This will not help building PALISADE, however.

As an experiment I can probably try compiling llvm-project/libcxx for BPF, but I'm afraid it's going to be a substantial struggle to have it built. I guess if we want C/C++ toolchain to be useful for on-chain program development, we need to provide the standard libraries, eventually.

[animber-coder]

@jackcmay "Picking glibc might be ok." - yes. However, I'd like Solana dev team to provide standard C++ libraries as well. Why don't we use C++ libraries for Solana on-chain program? Many C++ libraries(for example, Palisade) will be waiting to be easily implemented on chain.

[OluAgunloye]

@jackcmay @cpp-guru @dmakarov
Can we work together on this?

Would like to help get this done for obvious personal reasons, but also because it's about time this is done anyway. Would be willing to put up some funds to make it work.

[jackcmay]

@OluAgunloye Thanks for the interest! @dmakarov and I will sync up and get back to you.

[dmakarov]

@OluAgunloye as first step we're going to provide newlib (an implementation of C standard library) binaries compiled to BPF. Do you have any concerns about newlib?

[OluAgunloye]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile with its ability to resolve sys calls almost regardless of underlying system. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

[dmakarov]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile its sys call compatibility. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

We'll see if we can adapt llvm-project/libcxx (or part of it) to be buildable for BPF.

[OluAgunloye]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile its sys call compatibility. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

We'll see if we can adapt llvm-project/libcxx (or part of it) to be buildable for BPF.

Hello! Just thought I'd stop by to see how things are going. Does newlib compile easily enough to BPF?

[dmakarov]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile its sys call compatibility. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

We'll see if we can adapt llvm-project/libcxx (or part of it) to be buildable for BPF.

Hello! Just thought I'd stop by to see how things are going. Does newlib compile easily enough to BPF?

No yet, but I'm working on it. It'll take some time before we can package newlib binaries built for BPF in our tools release.

[animber-coder]

@dmakarov @jackcmay @OluAgunloye

Hello all of you.

1. I've compiled Palisade project with clang++ provided by Solana BPF tools and faced stack dump.
   Can I compile it after building llvm libcxx for BPF? Or is this due to clang++ itself?
2. What do you think about uclibc++ as a standard c++ library for bpf?
   I think Solana platform can be regarded as a embedded system and llvm libcxx is not suitable for BPF.
3. Can we work together on newlib for BPF?

[dmakarov]

@dmakarov @jackcmay @OluAgunloye

Hello all of you.

1. I've compiled Palisade project with clang++ provided by Solana BPF tools and faced stack dump.
   Can I compile it after building llvm libcxx for BPF? Or is this due to clang++ itself?
2. What do you think about uclibc++ as a standard c++ library for bpf?
   I think Solana platform can be regarded as a embedded system and llvm libcxx is not suitable for BPF.
3. Can we work together on newlib for BPF?

1. I think you need a C++ STL library compiled to BPF, and clang++ itself should not cause problems.
2. Yes we can use uclibc++ instead of llvm-project libcxx. Could you validate that uclibc++ is sufficient to build PALISIDE on some other platform for which uclibc++ is already known to be supported?
3. Absolutely, you're welcome to submit PRs.

[animber-coder]

@dmakarov @jackcmay @OluAgunloye
Hello all of you.

1. I've compiled Palisade project with clang++ provided by Solana BPF tools and faced stack dump.
   Can I compile it after building llvm libcxx for BPF? Or is this due to clang++ itself?
2. What do you think about uclibc++ as a standard c++ library for bpf?
   I think Solana platform can be regarded as a embedded system and llvm libcxx is not suitable for BPF.
3. Can we work together on newlib for BPF?

1. I think you need a C++ STL library compiled to BPF, and clang++ itself should not cause problems.
2. Yes we can use uclibc++ instead of llvm-project libcxx. Could you validate that uclibc++ is sufficient to build PALISIDE on some other platform for which uclibc++ is already known to be supported?
3. Absolutely, you're welcome to submit PRs.

1. Perfect.
2. I tried uclibc++ for bpf, but failed due to lack of some c++ features like std::move, std::forward.
   I'd like to ask you what you recommend either libcxx or uclibc++ for bpf?

[dmakarov]

2. I tried uclibc++ for bpf, but failed due to lack of some c++ features like std::move, std::forward.
   I'd like to ask you what you recommend either libcxx or uclibc++ for bpf?

My thinking was that you need to build PALISADE so it would be good to check whether uclibc++ has enough C++ STL functionality to build PALISADE on top of it. This could be validated if you build uclibc++ on a known supported platform (not BPF), and then build PALISADE using uclibc++. If that doesn't work, there's no point porting uclibc++ to BPF because uclibc++ targeted to BPF will not help to build PALISADE. I personally don't have a preference -- whichever is enough to get started and easier to port to BPF, that one we should start with.

[animber-coder]

That makes me sense, but I'd like you to consider uclibc++ is much lighter than libcxx. uclibc++ is designed for embedded system so personally I'd like to recommend uclibc++ for Solana program.
By the way, could you provide stdc header files like stdlib.h, stdint.h for bpf?

1. I tried uclibc++ for bpf, but failed due to lack of some c++ features like std::move, std::forward.
   I'd like to ask you what you recommend either libcxx or uclibc++ for bpf?

My thinking was that you need to build PALISADE so it would be good to check whether uclibc++ has enough C++ STL functionality to build PALISADE on top of it. This could be validated if you build uclibc++ on a known supported platform (not BPF), and then build PALISADE using uclibc++. If that doesn't work, there's no point porting uclibc++ to BPF because uclibc++ targeted to BPF will not help to build PALISADE. I personally don't have a preference -- whichever is enough to get started and easier to port to BPF, that one we should start with.

That makes me sense, but I'd like you to consider uclibc++ is much lighter than libcxx. uclibc++ is designed for embedded system so personally I'd like to recommend uclibc++ for Solana program.
By the way, could you provide stdc header files like stdlib.h, stdint.h, wchar.h for bpf?

[dmakarov]

I agree that uclibc++ is lighter and may be more suitable for our purposes. We just need to make sure the library has what we need. I'm not very familiar with uclibc++, skimmed through README and FAQ -- it's an alpha quality, although stable, has not all of C++ STL etc.

The headers that you mentioned are part of the newlib package. What do you mean by providing them for BPF? When we add newlib to our toolchain distribution package, the header files will be included in the package. I'm not sure at the moment that there need to be BPF specific changes in these header files.

To give you some heads up, I was able to build most of libc part of the newlib source tree for BPF, with few exceptions, namely anything with varargs (not supported by our compiler) and the setjump code. So what I'm working on now is configuring the build to exclude anything that is not supported for BPF from building.

[animber-coder]

I agree that uclibc++ is lighter and may be more suitable for our purposes. We just need to make sure the library has what we need. I'm not very familiar with uclibc++, skimmed through README and FAQ -- it's an alpha quality, although stable, has not all of C++ STL etc.

The headers that you mentioned are part of the newlib package. What do you mean by providing them for BPF? When we add newlib to our toolchain distribution package, the header files will be included in the package. I'm not sure at the moment that there need to be BPF specific changes in these header files.

To give you some heads up, I was able to build most of libc part of the newlib source tree for BPF, with few exceptions, namely anything with varargs (not supported by our compiler) and the setjump code. So what I'm working on now is configuring the build to exclude anything that is not supported for BPF from building.

No more questions. You are doing excellent work!

[animber-coder]

@dmakarov

Can I ask if you finished configuring the build to exclude anything that is not supported for BPF from building?

[dmakarov]

@dmakarov

Can I ask if you finished configuring the build to exclude anything that is not supported for BPF from building?

Last week I was almost entirely out of the office and didn’t have a chance to work on this. I will finish it this week. I’m sorry for the delay.

[dmakarov]

Both libc and libm of newlib now fully compile for BPF (excluding various printf and scanf functions), but I currently see this warning on my system

libc.a the table of contents is empty (no object file members in the library define global symbols)

So linking with libc leaves out some symbols undefined. I'll see how to fix this soon.

[animber-coder]

@dmakarov

I hope this issue is resolved and look forward to working with you soon.
Can I ask if it is possible in 2-3 days?

[dmakarov]

I think I resolved the issue with the table of contents and built the newlib libraries correctly. I'll run some tests, and if everything works I should be able to release the toolchain with the newlib binaries and header files in the next two or three days.

[dmakarov]

I released a version of bpf-tools that contains newlib headers and BPF compiled library binaries. You can download the package from here https://github.com/solana-labs/bpf-tools/releases/tag/v1.17

You would need to set up your build system to search for header files in your_installation_path/bpf-tools/llvm/include directory and for libraries in your_installation_path/bpf-tools/llvm/lib. For example, if you install in usual toolchain location, such as ~/.cache/solana/v1.17/bpf-tools, the paths to headers and libraries would be ~/.cache/solana/v1.17/bpf-tools/llvm/include and ~/.cache/solana/v1.17/bpf-tools/llvm/lib correspondingly.

Later I'll update solana SDK makefile to add the paths to headers and libraries.

[OluAgunloye]

@dmakarov Thanks, will try to build and let you know.

[animber-coder]

@dmakarov

Hello,

We've tried to build PALISADE using bpf-tools v1.8.
After some modifications, we could successfully build libcxx and libcxxabi of llvm-project, and palisade-release.
We added simple PALISADE functions into helloworld project and built, tried to deploy.
An error occurred during deployment:

Error: ELF error: ELF error: .bss section not supported

The problem is that palisade-release has a lot of classes and structures that contain static member variables and these are located in .bss section.

...
[16] .bss NOBITS 0000000000785d50 785d49 0004b0 00 WA 0 0 8
[17] .bss._ZN6cereal6detail12StaticObjectINS0_18PolymorphicCastersEE8instanceE NOBITS 0000000000786200 785d49 000008 00 WA 0 0 8
[18] .bss._ZGVN6cereal6detail12StaticObjectINS0_18PolymorphicCastersEE8instanceE NOBITS 0000000000786208 785d49 000008 00 WA 0 0 8
[19] .bss._ZN6cereal6detail12StaticObjectINS0_8VersionsEE8instanceE NOBITS 0000000000786210 785d49 000008 00 WA 0 0 8
[20] .bss._ZGVN6cereal6detail12StaticObjectINS0_8VersionsEE8instanceE NOBITS 0000000000786218 785d49 000008 00 WA 0 0 8
[21] .bss._ZN8lbcrypto26BinaryUniformGeneratorImplIN9bigintnat12NativeVectorINS1_14NativeIntegerTImEEEEE14m_distributionE NOBITS 0000000000786220 785d49 000008 00 WA 0 0 8
...

Can you let us know how we solve these issues?
Thank you.

[dmakarov]

Currently, there's no way around this, because our VM doesn't accept BPF programs with static data, and it looks like eliminating the static data members is not feasible in your case. We'll discuss this issue and I'll update you what may be a way forward.

[dmakarov]

@cpp-guru For now we're not going to allow on-chain programs to have static variables. This means, the only option for you is to try to change the library code so that it does not use static data members.

In fact, I'm preparing a new release of bpf-tools with an updated build of newlib. I configured newlib to use only re-entrant versions of C standard library functions, so that no global variable (such as errno) is ever used. In this configuration the users of the library should call, for example, atoi_r instead of atoi, and so for every function that has a re-entrant variant. This also may make it more difficult to build libcxx for BPF target, if some of its functions call non-reentrant variants of the C library functions.

[dmakarov]

We'll track the support for C++ standard library in a separate issue solana-labs/solana#21930
Closing this issue as it can't be resolved as is at the moment.