bug: nix build fails when using RLN support

[richard-ramos]

In #730 RLN support is included by default in go-waku. However, it seems that nix does not like it because nix build fails with the following error:

error: builder for '/nix/store/d9rma0sayfm9pl8042810rqa7mggdh8c-go-waku.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking source archive /nix/store/8pn5ks6ylg54ng9ax9ppfks7ifqpgshw-source
       > source root is source
       > patching sources
       > configuring
       > building
       > Building subPackage ./cmd/waku
       > # github.com/waku-org/go-waku/cmd/waku
       > /nix/store/r0x0agq0vwn0p6z99vkkvn8l8a8idzsb-go-1.19.8/share/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
       > /nix/store/ybw485608d7f1yv1v071j2052q64mvla-binutils-2.40/bin/ld: cannot find -lrln: No such file or directory
       > collect2: error: ld returned 1 exit status
       For full logs, run 'nix-store -l /nix/store/d9rma0sayfm9pl8042810rqa7mggdh8c-go-waku.drv'.

It seems that it has trouble finding librln.a which is included as part of go-zerokit-rln-x86_64 and go-zerokit-rln-apple dependencies. This error does not happen when using make build

In the meantime I had to add tags = [ "gowaku_no_rln" ]; to buildGo119Module so go-waku compiles.

@jakubgs, any idea what could be happening here?

[jakubgs]

Here's the command that actually fails:

+ /nix/store/ybw485608d7f1yv1v071j2052q64mvla-binutils-2.40/bin/ld -z relro -z now -plugin /nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/libexec/gcc/x86_64-unknown-linux-gnu/12.2.0/liblto_plugin.so -plugin-opt=/nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/libexec/gcc/x86_64-unknown-linux-gnu/12.2.0/lto-wrapper -plugin-opt=-fresolution=/build/ccCX0aIG.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --eh-frame-hdr -m elf_x86_64 -export-dynamic -dynamic-linker /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 -o $WORK/b001/exe/a.out /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/crt1.o /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/crti.o /nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/crtbegin.o -L/build/source/vendor/github.com/waku-org/go-zerokit-rln-x86_64/rln/../libs/x86_64-unknown-linux-gnu -L/nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib -L/nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0 -L/nix/store/vaxgqbm6h3zvhpgk9xqqk97hqjp9gzvs-gcc-12.2.0-lib/x86_64-unknown-linux-gnu/lib -L/nix/store/vaxgqbm6h3zvhpgk9xqqk97hqjp9gzvs-gcc-12.2.0-lib/lib -L/nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib -L/nix/store/vaxgqbm6h3zvhpgk9xqqk97hqjp9gzvs-gcc-12.2.0-lib/lib -L/nix/store/b5mkki8bysygaf7a01jpj88zrxibkrv4-gcc-wrapper-12.2.0/bin -L/nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0 -L/nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64 -L/nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/../../.. -dynamic-linker=/nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/ld-linux-x86-64.so.2 /build/go-link-2076656440/go.o /build/go-link-2076656440/000000.o /build/go-link-2076656440/000001.o /build/go-link-2076656440/000002.o /build/go-link-2076656440/000003.o /build/go-link-2076656440/000004.o /build/go-link-2076656440/000005.o /build/go-link-2076656440/000006.o /build/go-link-2076656440/000007.o /build/go-link-2076656440/000008.o /build/go-link-2076656440/000009.o /build/go-link-2076656440/000010.o /build/go-link-2076656440/000011.o /build/go-link-2076656440/000012.o /build/go-link-2076656440/000013.o /build/go-link-2076656440/000014.o /build/go-link-2076656440/000015.o /build/go-link-2076656440/000016.o /build/go-link-2076656440/000017.o /build/go-link-2076656440/000018.o /build/go-link-2076656440/000019.o /build/go-link-2076656440/000020.o /build/go-link-2076656440/000021.o /build/go-link-2076656440/000022.o /build/go-link-2076656440/000023.o /build/go-link-2076656440/000024.o /build/go-link-2076656440/000025.o /build/go-link-2076656440/000026.o /build/go-link-2076656440/000027.o /build/go-link-2076656440/000028.o /build/go-link-2076656440/000029.o /build/go-link-2076656440/000030.o /build/go-link-2076656440/000031.o /build/go-link-2076656440/000032.o /build/go-link-2076656440/000033.o /build/go-link-2076656440/000034.o /build/go-link-2076656440/000035.o /build/go-link-2076656440/000036.o /build/go-link-2076656440/000037.o /build/go-link-2076656440/000038.o /build/go-link-2076656440/000039.o /build/go-link-2076656440/000040.o /build/go-link-2076656440/000041.o /build/go-link-2076656440/000042.o /build/go-link-2076656440/000043.o /build/go-link-2076656440/000044.o /build/go-link-2076656440/000045.o -lpthread -ldl -lrln -ldl -lm -rpath /nix/store/iv7x6a4qwbpdqycp1b0qxw2vnrr4px7n-go-waku/lib64 -rpath /nix/store/iv7x6a4qwbpdqycp1b0qxw2vnrr4px7n-go-waku/lib -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /nix/store/dj5anrpwcn0p6h3xnqyr5mz30fc7l825-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/crtend.o /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/crtn.o -rpath /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib -rpath /nix/store/vaxgqbm6h3zvhpgk9xqqk97hqjp9gzvs-gcc-12.2.0-lib/lib
/nix/store/ybw485608d7f1yv1v071j2052q64mvla-binutils-2.40/bin/ld: cannot find -lrln: No such file or directory

Got it by adding NIX_DEBUG = 8 to buildGo119Module arguments. The key part is here:

... -lpthread -ldl -lrln -ldl -lm -rpath ...

It appears we are passing -lrln to the dynamic linker, but why? Where is that coming from?

[jakubgs]

When I run the build outside of nix derivation but in a nix shell I can see this is set in CGO_LDFLAGS.

[jakubgs@caspair:~/work/go-waku]$ rm -fr ~/.cache/go*

[jakubgs@caspair:~/work/go-waku]$ make build > build.log 2>&1

[jakubgs@caspair:~/work/go-waku]$ grep lrln build.log 
TERM='dumb' CGO_LDFLAGS='"-g" "-O2" "-lrln" "-ldl" "-lm" "-L/home/jakubgs/go/pkg/mod/github.com/waku-org/go-zerokit-rln-x86_64@v0.0.0-20230905182930-2b11e72ef866/rln/../libs/x86_64-unknown-linux-gnu"' /nix/store/r0x0agq0vwn0p6z99vkkvn8l8a8idzsb-go-1.19.8/share/go/pkg/tool/linux_amd64/cgo -objdir $WORK/b511/ -importpath github.com/waku-org/go-zerokit-rln-x86_64/rln -- -I $WORK/b511/ -g -O2 ./link.go ./wrapper.go
TERM='dumb' gcc -I /home/jakubgs/go/pkg/mod/github.com/waku-org/go-zerokit-rln-x86_64@v0.0.0-20230905182930-2b11e72ef866/rln -fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=$WORK/b511=/tmp/go-build -gno-record-gcc-switches -o $WORK/b511/_cgo_.o $WORK/b511/_cgo_main.o $WORK/b511/_x001.o $WORK/b511/_x002.o $WORK/b511/_x003.o -g -O2 -lrln -ldl -lm -L/home/jakubgs/go/pkg/mod/github.com/waku-org/go-zerokit-rln-x86_64@v0.0.0-20230905182930-2b11e72ef866/rln/../libs/x86_64-unknown-linux-gnu

So it appears to be also affecting the working build. Next thing to check is:

• How is -lrln ending up in CGO_LDFLAGS?
• Does it need to be in CGO_LDFLAGS for build to work?
• Can we remove it from CGO_LDFLAGS in Nix derivation?

[richard-ramos]

How is -lrln ending up in CGO_LDFLAGS?

This is being done here and it will pick up the libraries from https://github.com/waku-org/go-zerokit-rln-x86_64/tree/master/libs (there's similar code in go-zerokit-rln-apple and go-zerokit-rln-arm)

Does it need to be in CGO_LDFLAGS for build to work?

Compilation would fail with a lot of undefined reference to MISSING FUNCTION_NAME_GOES_HERE

[richard-ramos]

So it appears to be also affecting the working build.

Uh? I'm not getting errors when doing make build

[jakubgs]

I worded it badly. I mean the presence of -lrln in CGO_LDFLAGS also exists in non-nix builds.

[jakubgs]

One key thing to keep in mind that this error is caused by flag formatting, and not actual missing files:

.../ld: cannot find -lrln: No such file or directory

It's not saying rln doesn't exist, it's saying -lrln doesn't exist, which means it's thinking that's a file.

[jakubgs]

Foe example, I wonder if we need a space in that LDFLAGS line:

#cgo LDFLAGS:-lrln -ldl -lm

https://github.com/waku-org/go-zerokit-rln-x86_64/blob/2b11e72ef8665777102a3fc7c4bf15742260176b/rln/link.go#L4C1-L4C1

Because if we look at this example:

// #cgo LDFLAGS: -lpng

https://pkg.go.dev/cmd/cgo

It looks like there's a space before the -l flag. Not sure, just an idea.

[richard-ramos]

I tried in #743 which uses waku-org/go-zerokit-rln-x86_64@6057b97 (containing an space before the -l) and ended up with the same cannot find -lrln: No such file or directory error.

[richard-ramos]

In the link you shared for c-go, it says the following:

When the cgo directives are parsed, any occurrence of the string ${SRCDIR} will be replaced by the absolute path to the directory containing the source file. This allows pre-compiled static libraries to be included in the package directory and linked properly. For example if package foo is in the directory /go/src/foo:
// #cgo LDFLAGS: -L${SRCDIR}/libs -lfoo
Will be expanded to:
// #cgo LDFLAGS: -L/go/src/foo/libs -lfoo

I'll try this tomorrow