Build failure on Arch Linux

[dannasessha]

Trying to build from zcash master branch, I am experiencing a build failure.

To reproduce...

1. checkout zcash/master branch
2. ./zcutil/clean.sh
3. ./zcutil/build.sh -j $(nproc)

Expecting...
successful build with stdout including:
finished release [optimized] target(s) in ...

Observed behaviour...
build failure, stdout ending with:

checking whether the C compiler works... no
configure: error: in `.../zcash/depends/work/build/x86_64-pc-linux-gnu/libevent/2.1.12-86b28079bae':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** [funcs.mk:262: .../zcash/depends/work/build/x86_64-pc-linux-gnu/libevent/2.1.12-86b28079bae/./.stamp_configured] Error 77
make: Leaving directory '.../zcash/depends'

The above error was from an attempt to build a specific commit, but I tried to investigate more using git bisect. The "breaking change" for me was between 81764889c Merge pull request #6071 from str4d/llvm-14 (breaks) and df6e05173 Merge pull request #6083 from zcash/leveldb-1.22 (works) on master.

Specifically it told me...
12c36670f (refs/bisect/bad) depends: Update Clang / libcxx to LLVM 14.0.6

12c36670febc195a4cac92d54e5591b6a33e3f60 is the first bad commit
commit 12c36670febc195a4cac92d54e5591b6a33e3f60
Author: Jack Grigg <jack@z.cash>
Date:   Tue Jul 12 12:43:32 2022 +0000

    depends: Update Clang / libcxx to LLVM 14.0.6

    In practice we are using 14.0.0 in most cases, as the LLVM Project have
    not published Ubuntu binaries for any point release after 14.0.0 (which
    we are using here).

 contrib/devtools/update-clang-hashes.sh | 49 +++++++++++++++++++++++++++++++++
 depends/packages/libcxx.mk              | 32 ++++++++++++---------
 depends/packages/native_clang.mk        | 24 ++++++++++------
 qa/zcash/postponed-updates.txt          | 16 +----------
 4 files changed, 85 insertions(+), 36 deletions(-)
 create mode 100755 contrib/devtools/update-clang-hashes.sh

Local machine running Arch Linux
Linux A 5.19.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 17 Aug 2022 13:48:51 +0000 x86_64 GNU/Linux
Plenty of CPU, RAM, disk space.
I've built zcash successfully many times on Arch.

gcc --version
gcc (GCC) 12.1.1 20220730
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ld -v
GNU ld (GNU Binutils) 2.39

as --version
GNU assembler (GNU Binutils) 2.39
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `x86_64-pc-linux-gnu'.

[str4d]

CI builds and tests on Arch, and AFAICT that is succeeding as recently as today. So the next step here would be to try and spot any differences between the Arch setup here and in CI.

[mdr0id]

@dannasessha If you have docker you can sanity check the build against our CI image:
docker pull electriccoinco/zcashd-worker-archlinuxv2

I do recall some annoying issues on certain versions of ARCH, but this one in CI is pinned. If you still hit issues please let us know and I will try to reproduce on a newer version of ARCH.

[mdr0id]

There is also some annoying link issues, see our docker file:

ARG FROMBASEOS
ARG FROMBASEOS_BUILD_TAG
FROM $FROMBASEOS:$FROMBASEOS_BUILD_TAG

RUN pacman -Syyu --noconfirm \
    && pacman -S --noconfirm \
      base-devel \
      git \
      python3 \
      python-pip \
      ncurses \
      wget

RUN sudo link /lib/libtinfo.so.6 /lib/libtinfo.so.5

[AloeareV]

@dannasessha If you have docker you can sanity check the build against our CI image: docker pull electriccoinco/zcashd-worker-archlinuxv2

I do recall some annoying issues on certain versions of ARCH, but this one in CI is pinned. If you still hit issues please let us know and I will try to reproduce on a newer version of ARCH.

I encountered this same bug on latest master (6c392dbb0b43d6988283e40ce423242833df2d30). Same hardware as above, versions:

Linux 5.19.13-arch1-1 x86_64
gcc (GCC) 12.2.0
GNU ld (GNU Binutils) 2.39.0
GNU assembler (GNU Binutils) 2.39.0

I was able to build fine on the docker image (electriccoinco/zcashd-worker-archlinuxv2).

[mdr0id]

@AloeareV Nice! We are also working to update the arch image mentioned above to something more current. Hope to provide news on this soon.

[autotunafish]

There is a difference in the build versions: 5.19.13 and 5.19.2 though I'm not sure if it might affect the outcome.

[autotunafish]

I see the new docker image and I was wondering if there's any actual documentation for running zcashd on Arch?

[mdr0id]

Not at the moment but we are trying to find the newer version that will actually build zcash.
Below is the basic steps on an arch rig:

RUN pacman -Sy archlinux-keyring --noconfirm && pacman-key --init && pacman-key --populate archlinux \
    && pacman -Syyu --noconfirm --ignore filesystem \
    && pacman -S --noconfirm --ignore filesystem \
      base-devel \
      git \
      jq \
      python3 \
      python-pip \
      ncurses \
      wget

RUN sudo link /lib/libtinfo.so.6 /lib/libtinfo.so.5

The issues start around libtinfo linking and maybe some newer bugs.

[autotunafish]

Ok, I'll try that at some point. The Docker appears to be running though.

[str4d]

We think that this may be the same issue as #6193.

[autotunafish]

Building from source on Alter Linux, which is an Arch flavor that supports apt, fails with a similar instance of not being able to build the C. I'll rebuild it and grab that specific error later today hopefully.
OS source: https://alter.fascode.net/

Edit: Just a FYI but for some reason, and maybe it's just my system running this Alter distro, running pacman at all it seems breaks the keyring badly!

[zancas]

I've reproduced this build error.

[autotunafish]

Sorry I got distracted with zebra, it consistently build-fails with a rocksdb error 'cannot find stdarg.h'. But I think it's also not linked correctly, fills up the root dir (or something). The zebra docker fired right up and I'm gonna see how it does for a while and then come back.

[chappjc]

Observed behaviour... build failure, stdout ending with:

checking whether the C compiler works... no
configure: error: in `.../zcash/depends/work/build/x86_64-pc-linux-gnu/libevent/2.1.12-86b28079bae':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** [funcs.mk:262: .../zcash/depends/work/build/x86_64-pc-linux-gnu/libevent/2.1.12-86b28079bae/./.stamp_configured] Error 77
make: Leaving directory '.../zcash/depends'

Same here on the v0.5.3 tag, and when I do look at libevent's config.log for details, it points to a zlib linking issue:

configure:4325: checking whether the C compiler works
configure:4347: clang -target x86_64-pc-linux-gnu -B/.../zcash/depends/x86_64-pc-linux-gnu/native/bin -m64 -pipe -O3 -I/.../zcash/depends/x86_64-pc-linux-gnu/include -fuse-ld=lld  -L/.../zcash/depends/x86_64-pc-linux-gnu/lib conftest.c  >&5
ld.lld: error: /usr/lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/crti.o: contains a compressed section, but zlib is not available
ld.lld: error: /usr/lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/crti.o: contains a compressed section, but zlib is not available
ld.lld: error: /usr/lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/crtn.o: contains a compressed section, but zlib is not available
ld.lld: error: /usr/lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/crtn.o: contains a compressed section, but zlib is not available
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
configure:4351: $? = 1
configure:4391: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "libevent"
| #define PACKAGE_TARNAME "libevent"
| #define PACKAGE_VERSION "2.1.12-stable"
| #define PACKAGE_STRING "libevent 2.1.12-stable"
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define PACKAGE "libevent"
| #define VERSION "2.1.12-stable"
| #define NUMERIC_VERSION 0x02010c00
| /* end confdefs.h.  */
| 
| int
| main (void)
| {
| 
|   ;
|   return 0;
| }

A clue? rust-lang/rust#97452

[chappjc]

ldd reports that ./depends/x86_64-pc-linux-gnu/native/bin/ld.lld is not linked with libz/zlib support.
On the other hand, the system package's llvm-link is (ldd /usr/bin/llvm-link).
Seems like the issue is that the native clang downloaded by the depends package does not support linking with zlib compression.

The native clang downloaded by v5.2.0 however does have libz:
libz.so.1 => /usr/lib/libz.so.1 (0x00007f1329c98000)
The version downloaded by v5.3.0 lacks it.

Related: ziglang/zig-bootstrap#57

[defuse]

I'm running into the same error as @chappjc, and I think their analysis is correct. The clang 14.0.0 changelog mentions a zlib-related change so maybe that's a clue:

The legacy -gz=zlib-gnu and -Wa,--compress-debug-sections=zlib-gnu have been removed.

[defuse]

My Arch Linux happens to have v14.0.6 of ld.lld so I was able to work around it by hacking the depends system to use my system's copy:

$ git diff
diff --git a/depends/packages/native_clang.mk b/depends/packages/native_clang.mk
index 37a3c2c31..3a8f8aca1 100644
--- a/depends/packages/native_clang.mk
+++ b/depends/packages/native_clang.mk
@@ -51,6 +51,7 @@ define $(package)_stage_cmds
   cp -P bin/lld-link $($(package)_staging_prefix_dir)/bin && \
   cp -P bin/llvm-ranlib $($(package)_staging_prefix_dir)/bin && \
   cp -P bin/llvm-strip $($(package)_staging_prefix_dir)/bin && \
+  cp /usr/bin/ld.lld $($(package)_staging_prefix_dir)/bin && \
   (test ! -f include/x86_64-unknown-linux-gnu/c++/v1/__config_site || \
    cp include/x86_64-unknown-linux-gnu/c++/v1/__config_site include/c++/v1/__config_site) && \
   mv include/ $($(package)_staging_prefix_dir) && \

This is not a good idea unless your system uses the same version as v5.3.0 does (14.0.6):

$ /usr/bin/ld.lld --version
LLD 14.0.6 (compatible with GNU linkers)

[kpcyrd]

I was able to build 5.3.0 on Arch Linux like this: https://github.com/archlinux/svntogit-community/blob/b93b617866305f8c6d649f511ff0a7f0cb5d953e/trunk/PKGBUILD

This doesn't use ./zcutil/build.sh though and instead tries to build directly with system tools. :)

[dannasessha]

I was able to build 5.3.0 on Arch Linux like this: https://github.com/archlinux/svntogit-community/blob/b93b617866305f8c6d649f511ff0a7f0cb5d953e/trunk/PKGBUILD

This doesn't use ./zcutil/build.sh though and instead tries to build directly with system tools. :)

This is very interesting!

However, I do notice that your script uses oracle.com as a source. Is this true also in the "mainstream" zcash build.sh method?

[kpcyrd]

Yes:

zcash/depends/packages/bdb.mk

Line 3 in b67ffbb

$(package)_download_path=https://download.oracle.com/berkeley-db

zcash/COPYING

Lines 32 to 35 in b67ffbb

	Although almost all of the Zcash code is licensed under "permissive" open source
	licenses, users and distributors should note that when built using the default
	build options, Zcash depends on Oracle Berkeley DB 6.2.x, which is licensed
	under the GNU Affero General Public License.

Berkeley DB was bought by oracle at some point, Linux distributions usually only ship 5.3 but I don't know the reason for this, given that 6.2 is AGPL and could be distributed.