ld: library not found for -lhwloc

[indexroot]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Run

brew install go bzr jq pkg-config rustup hwloc
git clone https://github.com/filecoin-project/lotus.git
cd lotus/
export CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__"
export CGO_CFLAGS="-D__BLST_PORTABLE__"
make clean && make all

2. See error

Expected behavior

+ exit
rm -f lotus
go build  -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.6de62411c -extldflags=-L/usr/local/opt/openssl@1.1/lib" -o lotus ./cmd/lotus
# github.com/filecoin-project/filecoin-ffi/generated
ld: library not found for -lhwloc
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [lotus] Error 2

Screenshots

Version (run lotus version):
latest

Additional context
~/.zshrc
Add

export LDFLAGS="-L/usr/local/Cellar/hwloc/2.4.0/lib"
export CPPFLAGS="-I/usr/local/Cellar/hwloc/2.4.0/include"

[NewMai]

try to install libhwloc-dev with command:

sudo apt install libhwloc-dev -y

[jleni]

It seems that a mac is being used (brew) not ubuntu/linux

[lonnylot]

@indexroot Have you solved this yet? I'm experiencing the same issue. Here is where I am so far:

hwloc is found if you first:

export LIBRARY_PATH=/opt/homebrew/lib

However, then I receive this: ld: warning: ignoring file /Users/lonnykapelushnik/Development/lotus/extern/filecoin-ffi/libfilcrypto.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

[jleni]

I have the same problem.
There seems to be a problem when using new M1-based Macs.

[travisperson]

By default lotus build process downloads the Filecoin FFI build from Github. You will need to build it yourself here.

You can do so by setting the environment var FFI_BUILD_FROM_SOURCE=1 prior to running the build. You may also need/want to set RUSTFLAGS="-C target-cpu=native".

[lonnylot]

I wanted to add a message I sent in Slack last night so everyone knows where I left off:

I then tried getting it to build libfilcrypto from source via:

export FFI_BUILD_FROM_SOURCE=1
brew install rust
ln -s /opt/homebrew/bin/rustup-init /opt/homebrew/bin/rustup

Remove toolchain from cargo command in rust/scripts/build-release.sh because nightly-2020-10-05 does not exist for aarch64-apple-darwin

Which lead me to my current issue, which is sha2-asm doesn't work on the M1: RustCrypto/asm-hashes#28

I'm a bit busy today, but will try to add RUSTFLAGS="-C target-cpu=native" either later tonight or tomorrow night (GMT-5) and then, if it doesn't work, I'll continue down the rabbit hole.

[lonnylot]

Just a quick update: RUSTFLAGS="-C target-cpu=native" didn't change anything. Same issue as before.

[cryptonemo]

I wouldn't expect it to work out of the box. At the very least, FFI does some CPU flag detection and I believe is limited to x86_64 and aarch at the moment. I don't expect those changes will be very difficult, but something to look at if you're trying this route before it's officially tested/supported.

[jennijuju]

Issue can be tracked here

[jennijuju]

related #5979

[jennijuju]

filecoin-project/filecoin-ffi#174

[jennijuju]

please use filecoin-project/filecoin-ffi#174 and have LIBRARY_PATH=/opt/homebrew/lib

[jennijuju]

git clone https://github.com/filecoin-project/lotus
cd lotus
git submodule update --init --recursive
export GOARCH=arm64
export CGO_ENABLED=1
export LIBRARY_PATH=/opt/homebrew/lib
cd extern/filecoin-ffi
git fetch -a
git checkout m1-portable
make clean
make
cd ../../
make lotus

[sanderpick]

FYI, looks like m1-portable branch was deleted. I guess the changes are now in master?

[cryptonemo]

rust-fil-proofs v7.0.1 contains the proper m1 support (not just a portable/compat mode). I believe building lotus v1.9.0-rc3 or later will have the support out of the box.

[sanderpick]

Cool, thanks!

[JustinGaoF]

@cryptonemo Maybe I ask a stupid question. How can I use the rust-fil-proofs v7.0.1 while build the lotus? I tried to use master and release/v1.9.0 branch of the lotus with master branch of filecoin-ffi, still met the issue src/lotus/extern/filecoin-ffi/libfilcrypto.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

[pefish]

Is there any progress?

[cryptonemo]

Did you try: export LIBRARY_PATH=/opt/homebrew/lib? The other option is to disable multicore sdr (see README: https://github.com/filecoin-project/filecoin-ffi)

[LexLuthr]

I was running into same error on M1 Mac with latest release when I was installing devnet. The following steps helped me.

1. brew install go bzr jq pkg-config rustup hwloc. - Please uninstall rust as it would clash with rustup in case you have installed
2. export LIBRARY_PATH=/opt/homebrew/lib
3. export FFI_BUILD_FROM_SOURCE=1
4. rustup-init (This should install cargo)
5. Follow the next steps as per the docs.