Cannot build on wasm32-unknown-unknown target

[hamza1311]

I tried to add syntect with onig dependency on wasm target but it failed to compile. Looking into it, it turned out to a problem with rust-onig.

Steps to reproduce

• Create a new cargo binary crate
• Add onig = "6" dependency
• Build it for wasm target

Cargo.toml

[package]
name = "onig-wasm-test"
version = "0.1.0"
authors = ["Hamza <muhammadhamza1311@gmail.com>"]
edition = "2018"

[dependencies]
onig = "6"

Outout of a clean (first) build

$ cargo build --target=wasm32-unknown-unknown

    Updating crates.io index
   Compiling libc v0.2.79
   Compiling memchr v2.3.3
   Compiling bitflags v1.2.1
   Compiling version_check v0.9.2
   Compiling glob v0.3.0
   Compiling cfg-if v0.1.10
   Compiling libloading v0.6.3
   Compiling proc-macro2 v1.0.24
   Compiling log v0.4.11
   Compiling lazy_static v1.4.0
   Compiling unicode-width v0.1.8
   Compiling unicode-xid v0.2.1
   Compiling regex-syntax v0.6.18
   Compiling quick-error v1.2.3
   Compiling strsim v0.8.0
   Compiling termcolor v1.1.0
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling bindgen v0.55.1
   Compiling lazycell v1.3.0
   Compiling rustc-hash v1.1.0
   Compiling shlex v0.1.1
   Compiling peeking_take_while v0.1.2
   Compiling pkg-config v0.3.18
   Compiling cc v1.0.60
   Compiling thread_local v1.0.1
   Compiling textwrap v0.11.0
   Compiling humantime v1.3.0
   Compiling nom v5.1.2
   Compiling clang-sys v1.0.1
   Compiling aho-corasick v0.7.13
   Compiling quote v1.0.7
   Compiling atty v0.2.14
   Compiling which v3.1.1
   Compiling clap v2.33.3
   Compiling regex v1.3.9
   Compiling cexpr v0.4.0
   Compiling env_logger v0.7.1
   Compiling onig_sys v69.5.1
The following warnings were emitted during compilation:

warning: In file included from oniguruma/src/regexec.c:36:
warning: oniguruma/src/regint.h:100:10: fatal error: 'stdlib.h' file not found
warning: #include <stdlib.h>
warning:          ^~~~~~~~~~
warning: 1 error generated.

error: failed to run custom build command for `onig_sys v69.5.1`

Caused by:
  process didn't exit successfully: `/home/hamza/Downloads/code/rust/onig-wasm-test/target/debug/build/onig_sys-0b840d93c9b4ffd9/build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_wasm32-unknown-unknown = None
  CC_wasm32_unknown_unknown = None
  TARGET_CC = None
  CC = None
  CFLAGS_wasm32-unknown-unknown = None
  CFLAGS_wasm32_unknown_unknown = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "/home/hamza/Downloads/code/rust/onig-wasm-test/target/wasm32-unknown-unknown/debug/build/onig_sys-07c76955b2fa4d55/out" "-I" "oniguruma/src" "-DONIG_DISABLE_DIRECT_THREADING=1" "-DONIG_EXTERN=__attribute__((visibility(\"default\")))" "-o" "/home/hamza/Downloads/code/rust/onig-wasm-test/target/wasm32-unknown-unknown/debug/build/onig_sys-07c76955b2fa4d55/out/oniguruma/src/regexec.o" "-c" "oniguruma/src/regexec.c"
  cargo:warning=In file included from oniguruma/src/regexec.c:36:
  cargo:warning=oniguruma/src/regint.h:100:10: fatal error: 'stdlib.h' file not found
  cargo:warning=#include <stdlib.h>
  cargo:warning=         ^~~~~~~~~~
  cargo:warning=1 error generated.
  exit code: 1

  --- stderr


  error occurred: Command "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "/home/hamza/Downloads/code/rust/onig-wasm-test/target/wasm32-unknown-unknown/debug/build/onig_sys-07c76955b2fa4d55/out" "-I" "oniguruma/src" "-DONIG_DISABLE_DIRECT_THREADING=1" "-DONIG_EXTERN=__attribute__((visibility(\"default\")))" "-o" "/home/hamza/Downloads/code/rust/onig-wasm-test/target/wasm32-unknown-unknown/debug/build/onig_sys-07c76955b2fa4d55/out/oniguruma/src/regexec.o" "-c" "oniguruma/src/regexec.c" with args "clang" did not execute successfully (status code exit code: 1).

Enviromment

OS: Fedora release 32 (Thirty Two) x86_64; Kernel : 5.8.10-200.fc32.x86_64 (if that matters)
Clang Version:

$ clang --version
clang version 10.0.1 (Fedora 10.0.1-2.fc32)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

rustc version: rustc 1.46.0 (04488afe3 2020-08-24)
cargo version: cargo 1.46.0 (149022b1d 2020-07-17)

[iwillspeak]

I’ve never built Onig for WASM. The Artichoke peeps have though. This is their reference: https://github.com/artichoke/artichoke/blob/414b7a2fe387c480d1385efb0f63963153754a3e/artichoke-backend/Cargo.toml#L31

[iwillspeak]

I wonder if the default features could be the difference. Disabling that will mean we don’t try to generate an up to date set of bindings and fall back on our built in set. I guess we have one that works for WASM if Artichoke has it working.

[lopopolo]

I guess we have one that works for WASM if Artichoke has it working.

Artichoke does not build with wasm32-unknown-unknown -- it builds with wasm32-unknown-emscripten. In general, when building C sources as part of a Rust lib or binary, the wasm32-unknown-unknown will not work. There is no C standard library in wasm32-unknown-unknown.

This upstream issue is relevant for one reason why this doesn't work: rustwasm/team#291.

[lopopolo]

The Artichoke peeps have though.

@iwillspeak next time, please feel free to @ me if there's onig-Artichoke questions I can help with 😄

[shybyte]

On Ubtuntu 18.04, using "wasm-pack", I had got the same fatal error: 'stdlib.h' file not found problem, mentioned in the original issue description.
I have solved it by

export CPATH=/usr/include

However, now I get the error message

 rust-lld: error: duplicate symbol: OnigEncodingASCII
          >>> defined in /home/marco/workspace/private/wasm-game-of-life-2/target/wasm32-unknown-unknown/debug/deps/libonig_sys-538ff2b6c9a16eb8.rlib(regexec.o)
          >>> defined in /home/marco/workspace/private/wasm-game-of-life-2/target/wasm32-unknown-unknown/debug/deps/libonig_sys-538ff2b6c9a16eb8.rlib(regerror.o)

Is this the kind of error that will vanish when I somehow configure wasm-pack to create a wasm32-unknown-emscripten target output?
Have you seen this error before?

(I have now solved my original problem by using a different regexp crate, but maybe this comment is somehow helpful, so I leave it here).

[MolotovCherry]

Got this error when wanting to use onig with wasm32-unknown-unknown. Then I remembered when I was working with emscripten once, that Rust doesn't really allow you to compile C libraries unless it's with emscripten. Except the emscripten target is not maintained well..

[iwillspeak]

Doing some more digging on this, it seems that the underlying c library, Oniguruma, doesn't build cleanly on either wasm32-unknown-unknown, or wasm32-unknown-emscripten. Not sure what we can do better about this from an Onig point of view. If anyone has more experience with these toolchains and wants to open a PR with suggested improvements it would be more than welcome.

EDIT Scratch that. Onig does build on Emscripten if you have the rust target wasm32-unknown-emscripten installed and use

cargo build --no-default-features --target=wasm32-unknown-emscripten --features=generate

I'm not really sure what we can do about building for -unknown-unknown though. The build fails there because it can't find c libraries:

running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "/home/will/Repositories/rust-onig/target/wasm32-unknown-unknown/debug/build/onig_sys-d79b79668a7ad64f/out" "-I" "oniguruma/src" "-DONIG_DISABLE_DIRECT_THREADING=1" "-DONIG_EXTERN=__attribute__((visibility(\"default\")))" "-o" "/home/will/Repositories/rust-onig/target/wasm32-unknown-unknown/debug/build/onig_sys-d79b79668a7ad64f/out/oniguruma/src/regexec.o" "-c" "oniguruma/src/regexec.c"
  cargo:warning=In file included from oniguruma/src/regexec.c:36:
  cargo:warning=oniguruma/src/regint.h:109:10: fatal error: 'stdlib.h' file not found
  cargo:warning=#include <stdlib.h>
  cargo:warning=         ^~~~~~~~~~
  cargo:warning=1 error generated.

Still open to any PRs if people want to contribute, but closing this issue for now.

[MolotovCherry]

Last I knew, building C libraries is only possible with emscripten unfortunately.

Edit: rustwasm/team#291 and rustwasm/team#291 (comment)