Errors cross compiling go-yara

[Xumeiquer]

I am trying to cross compile go-yara for x86 and x64 and I am getting some errors.
The tool set for cross compiling is xgo.

I can cross compile correctly for darwin or windows, but I am unable to do it for linux.

x64

/usr/local/lib/libyara.a(math.o): In function `string_entropy':
math.c:(.text+0xecc): undefined reference to `log2'
/usr/local/lib/libyara.a(math.o): In function `data_entropy':
math.c:(.text+0x1150): undefined reference to `log2'
collect2: error: ld returned 1 exit status

x86

/usr/bin/ld: skipping incompatible /usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: skipping incompatible //usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: cannot find -lyara
collect2: error: ld returned 1 exit status

I am not sure that the errors came either from go-yara or yara itself.

If you need more datils, please ask them.

[hillu]

The log2 function needs libm, so I suppose that the compiler switch -lm is missing there. For x86 it seems that the linker is only finding libyara.a for the wrong architecture, presumably your /usr/local/lib/libyara.a is an x64 library.

I have never tried to work with xgo, can you provide the command line you used?

[Xumeiquer]

Hi,
Here is more detail about the cross compilation process. This pice of code id for x64.

I cut some bits just for shorten it a little bit.

Compiling & Installing Yara

You can see while configuring the Yara sources log2 is availabe.

[...]
Compiling for linux/amd64...
Bootstaping dependency yara-3.7.1 for x86_64-linux...
[...]
checking whether to check for GCC pthread/shared inconsistencies... yes
checking whether -pthread is sufficient with -shared... yes
checking for isnan in -lm... yes
checking for log2 in -lm... yes
[...]
Building dependency yara-3.7.1 for x86_64-linux...
[...]
Making install in libyara
[...]
Libraries have been installed in:
   /usr/local/lib`

Compiling the project with go-yara

Do you have any idea why this is happening?

I put the entire log.

WORK=/tmp/go-build672133109
mkdir -p $WORK/github.com/xxx/common/command/_obj/
mkdir -p $WORK/github.com/xxx/common/
cd /ext-go/1/src/github.com/xxx/common/command
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/xxx/common/command.a -trimpath $WORK -p github.com/xxx/common/command -complete -buildid 6de232fd460c689487cd553d492a5b981f9a6a4e -D _/ext-go/1/src/github.com/xxx/common/command -I $WORK -pack ./command.go
mkdir -p $WORK/github.com/xxx/common/errors/_obj/
cd /ext-go/1/src/github.com/xxx/common/errors
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/xxx/common/errors.a -trimpath $WORK -p github.com/xxx/common/errors -complete -buildid 0b88032c86f59d76b47652396d7a97a9baba41d6 -D _/ext-go/1/src/github.com/xxx/common/errors -I $WORK -pack ./error.go
mkdir -p $WORK/github.com/hillu/go-yara/_obj/
mkdir -p $WORK/github.com/hillu/
cd /ext-go/1/src/github.com/hillu/go-yara
pkg-config --cflags yara
mkdir -p $WORK/github.com/go-ini/ini/_obj/
mkdir -p $WORK/github.com/go-ini/
cd /ext-go/1/src/github.com/go-ini/ini
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/go-ini/ini.a -trimpath $WORK -p github.com/go-ini/ini -complete -buildid 474f2dbaf2ddd368243b39907845dc5143ab9520 -D _/ext-go/1/src/github.com/go-ini/ini -I $WORK -pack ./error.go ./file.go ./ini.go ./key.go ./parser.go ./section.go ./struct.go
cd /ext-go/1/src/github.com/hillu/go-yara
pkg-config --libs yara
CGO_LDFLAGS="-g" "-O2" "-L/usr/local/lib" "-lyara" /usr/local/go/pkg/tool/linux_amd64/cgo -objdir $WORK/github.com/hillu/go-yara/_obj/ -importpath github.com/hillu/go-yara -- -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 cgo.go compiler.go compiler_yara37.go error.go error_yara34.go error_yara35.go error_yara36.go error_yara37.go rule.go rule_yara37.go rules.go rules_yara34.go stream.go
mkdir -p $WORK/github.com/sirupsen/logrus/_obj/
mkdir -p $WORK/github.com/sirupsen/
cd /ext-go/1/src/github.com/sirupsen/logrus
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/sirupsen/logrus.a -trimpath $WORK -p github.com/sirupsen/logrus -complete -buildid 69e4640f7e47be37721fc4a6cace76ae47f6b1d5 -D _/ext-go/1/src/github.com/sirupsen/logrus -I $WORK -pack ./alt_exit.go ./doc.go ./entry.go ./exported.go ./formatter.go ./hooks.go ./json_formatter.go ./logger.go ./logrus.go ./terminal_linux.go ./terminal_notwindows.go ./text_formatter.go ./writer.go
cd $WORK
gcc -fdebug-prefix-map=a=b -c trivial.c
gcc -gno-record-gcc-switches -c trivial.c
cd /ext-go/1/src/github.com/hillu/go-yara
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/_cgo_export.o -c $WORK/github.com/hillu/go-yara/_obj/_cgo_export.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/cgo.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/cgo.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/compiler.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/compiler.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/compiler_yara37.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/compiler_yara37.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/error.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/error.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/error_yara34.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/error_yara34.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/error_yara35.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/error_yara35.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/error_yara36.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/error_yara36.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/error_yara37.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/error_yara37.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/rule.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/rule.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/rule_yara37.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/rule_yara37.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/rules.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/rules.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/rules_yara34.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/rules_yara34.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/stream.cgo2.o -c $WORK/github.com/hillu/go-yara/_obj/stream.cgo2.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/rules-callback.o -c ./rules-callback.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I/usr/local/include -I $WORK/github.com/hillu/go-yara/_obj/ -g -O2 -o $WORK/github.com/hillu/go-yara/_obj/_cgo_main.o -c $WORK/github.com/hillu/go-yara/_obj/_cgo_main.c
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/github.com/hillu/go-yara/_obj/_cgo_.o $WORK/github.com/hillu/go-yara/_obj/_cgo_main.o $WORK/github.com/hillu/go-yara/_obj/_cgo_export.o $WORK/github.com/hillu/go-yara/_obj/cgo.cgo2.o $WORK/github.com/hillu/go-yara/_obj/compiler.cgo2.o $WORK/github.com/hillu/go-yara/_obj/compiler_yara37.cgo2.o $WORK/github.com/hillu/go-yara/_obj/error.cgo2.o $WORK/github.com/hillu/go-yara/_obj/error_yara34.cgo2.o $WORK/github.com/hillu/go-yara/_obj/error_yara35.cgo2.o $WORK/github.com/hillu/go-yara/_obj/error_yara36.cgo2.o $WORK/github.com/hillu/go-yara/_obj/error_yara37.cgo2.o $WORK/github.com/hillu/go-yara/_obj/rule.cgo2.o $WORK/github.com/hillu/go-yara/_obj/rule_yara37.cgo2.o $WORK/github.com/hillu/go-yara/_obj/rules.cgo2.o $WORK/github.com/hillu/go-yara/_obj/rules_yara34.cgo2.o $WORK/github.com/hillu/go-yara/_obj/stream.cgo2.o $WORK/github.com/hillu/go-yara/_obj/rules-callback.o -g -O2 -L/usr/local/lib -lyara
# github.com/hillu/go-yara
/usr/local/lib/libyara.a(math.o): In function `string_entropy':
math.c:(.text+0xecc): undefined reference to `log2'
/usr/local/lib/libyara.a(math.o): In function `data_entropy':
math.c:(.text+0x1150): undefined reference to `log2'
collect2: error: ld returned 1 exit status
2018/02/20 19:08:52 Failed to cross compile package: exit status 2.
make: *** [linux-x64] Error 1

[hillu]

@Xumeiquer Please tell me what exact command you used to build your program so I can try to reproduce it.

One thing that strikes me as odd is that even though you seem to have built a custom copy of yara, libyara that has been installed to /usr/local/ is used by the build system.

[Xumeiquer]

Hi,

As I already said I am using xgo to cross compile my project. xgo uses docker with several compilers for several architectures. The project xgo allows to compile C-like dependencies as well and the way it does is by running ./configure, make and, make install. As you may know, Yara uses a bootstrap command before calling ./configure so I made a a fix for using xgo with Yara in https://github.com/Xumeiquer/xgo and especially in https://github.com/Xumeiquer/xgo/blob/master/docker/base/build_deps.sh.

What I suggest if you try to reproduce is to download the karalabe/xgo-latest and them modify the build_deps.sh file and generate a new image from the running container.

It may be a little bit tedious but it is a way to cross compile projects easily.

After all of this, the command line I use to build my project is:

DOCKER_IMAGE := "x-go-yara" ## Modified version of karalabe/xgo-latest
DEPS := "https://github.com/VirusTotal/yara/archive/3.7.1.tar.gz"

xgo -x -image $(DOCKER_IMAGE) -dest build -deps $(DEPS) -targets="linux/386" .

The same way to cross compile my project for Linux 64bit I use the following command:

DOCKER_IMAGE := "x-go-yara" ## Modified version of karalabe/xgo-latest
DEPS := "https://github.com/VirusTotal/yara/archive/3.7.1.tar.gz"

xgo -x -image $(DOCKER_IMAGE) -dest build -deps $(DEPS) -targets="linux/amd64" .

Just in case you ask, I can cross compile without any issues for Windows and Darwin.

[hillu]

A thing that seems odd to me is that you are patching specific configure options into the setup script; did --depsargs not work for you?

Could you try if adding -lm to the libs.private entry of yara.pc works for you?

[Xumeiquer]

Hi,
Using --depsargs doesn't work because what yara build process needs is to run the bootstrap first.

I also tried to modify the yara.pc, but unfortunately it doesn't work either.

What I have done in terms to "debug" this is running the build process manually step by step. I have discover that exporting the CGO_CFLAGS and CGO_LDFLAGS and adding -lm to CGO_LDFLAGS go-yara compiles well.

# export CGO_CFLAGS="-I${YARA_SRC}/libyara/include"
# export CGO_LDFLAGS="-L${YARA_SRC}/libyara/.libs -lyara -lm"
# go build -tags yara3.7

Do you have any clue about if it would be possible to add -lm as -ldflag or -extldflags basically without exporting the CGO_ variables?

[hillu]

Using --depsargs doesn't work because what yara build process needs is to run the bootstrap first.

Ah, this is probably why I roll my own scripts / Makefiles for cross-building instead of relying on tools like xgo. :-)

I also tried to modify the yara.pc, but unfortunately it doesn't work either.

Ah, I forgot to mention it: You'd have to pass the "yara_static" build tag, so the Go toolchain calls "pkg-config --static yara".

[hillu]

Assuming that the issue has been fixed or become irrelevant at some point in the last 5 years, I'm closing it.