Skip to content

"recursive" Dependency Causes "section too large" Error When Compiling for wasm #13513

New issue
New issue
Closed
Listed in
#13815
#16418
Closed
"recursive" Dependency Causes "section too large" Error When Compiling for wasm#13513
Listed in
#13815
#16418
Labels
bugSomething isn't working
@berkaysynnada

Description

@berkaysynnada
berkaysynnada
opened on Nov 21, 2024 · edited by berkaysynnada
Contributor

Describe the bug

I am encountering a problem when trying to compile wasmtest with the following command:

RUSTFLAGS="-C link-arg=--max-memory=4294967296" cargo build --release --target wasm32-unknown-unknown

The build fails with the following error:

error: failed to build archive at `.../target/wasm32-unknown-unknown/release/deps/libpsm-cc498cff4b514c30.rlib`: LLVM error: section too large

The following warnings were emitted during compilation:

warning: psm@0.1.24: warning: ...

I suspect the issue is related to the psm dependency, and it is coming with recursive crate, which was introduced in #13310. I am unsure if this error can be resolved without removing or replacing this dependency.

To Reproduce

Go in datafusion/wasmtest and run cargo build --target wasm32-unknown-unknown

Expected behavior

It needs to compile

Additional context

I tried again after removing recursive dep, and it works successfully

Activity

berkaysynnada
added
bugSomething isn't working
on Nov 21, 2024
berkaysynnada
changed the title [-]recursive Dependency Causes "section too large" Error When Compiling for wasm32-unknown-unknown[/-] [+]"recursive" Dependency Causes "section too large" Error When Compiling for wasm32-unknown-unknown[/+] on Nov 21, 2024
berkaysynnada
changed the title [-]"recursive" Dependency Causes "section too large" Error When Compiling for wasm32-unknown-unknown[/-] [+]"recursive" Dependency Causes "section too large" Error When Compiling for wasm[/+] on Nov 21, 2024
Eason0729

Eason0729 commented on Nov 26, 2024

@Eason0729
Eason0729
on Nov 26, 2024
Contributor

I have try to reproduce the bug by either build wasmtest or build stacker with wasm32-unknown-unknown, but was unable to reproduce the issue.

Can you provide more information? (cargo and rustc version...)

blaginin

blaginin commented on Nov 26, 2024

@blaginin
blaginin
on Nov 26, 2024
Contributor

This obviously doesn't resolve the issue, but rustwasm/wasm-pack#1381 (comment) fixed it for me

berkaysynnada

berkaysynnada commented on Nov 29, 2024

@berkaysynnada
berkaysynnada
on Nov 29, 2024
ContributorAuthor

I have applied the @blaginin suggestion:

berkaysahin@Berkays-MacBook-Pro wasmtest % clang --version                                        
Homebrew clang version 19.1.4
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/19.1.4/bin
Configuration file: /opt/homebrew/etc/clang/arm64-apple-darwin23.cfg

@Eason0729 these are my cargo and rust versions

berkaysahin@Berkays-MacBook-Pro wasmtest % cargo --version                                        
cargo 1.82.0 (8f40fc59f 2024-08-21)
berkaysahin@Berkays-MacBook-Pro wasmtest % rustc --version                                        
rustc 1.82.0 (f6e511eec 2024-10-15)

When I build with: cargo build --target wasm32-unknown-unknown ,
it successfully compiles and generates the binary.

However, in the release mode: cargo build --release --target wasm32-unknown-unknown,

Compiling psm v0.1.24
   Compiling arrow-ipc v53.3.0
   Compiling arrow-json v53.3.0
   Compiling arrow-csv v53.3.0
   Compiling stacker v0.1.17
   Compiling recursive v0.1.1
error: failed to build archive at `/Users/berkaysahin/Desktop/datafusion-upstream/target/wasm32-unknown-unknown/release/deps/libpsm-cc498cff4b514c30.rlib`: LLVM error: section too large

The following warnings were emitted during compilation:

warning: psm@0.1.24: warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: /Users/berkaysahin/Desktop/datafusion-upstream/target/wasm32-unknown-unknown/release/build/psm-d273ebae732f9287/out/libpsm_s.a the table of contents is empty (no object file members in the library define global symbols)

error: could not compile `psm` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

I don't know why this still refers to Xcode path.

Since wasm-pack build goes with release flag, we need to solve this (assuming I am not the only one)

Eason0729

Eason0729 commented on Dec 1, 2024

@Eason0729
Eason0729
on Dec 1, 2024
Contributor

It compiles under 1.82.0 and 1.83.0(didn't use LLVM 15), I think it's LLVM version related.

In addition, I may need some help on how to use specific version of LLVM?

root@73b0a356b106:/stacker/psm# echo $PATH
/usr/lib/llvm-15/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

root@73b0a356b106:/stacker/psm# clang --version
Debian clang version 15.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-15/bin

root@73b0a356b106:/stacker/psm# rustc --version --verbose 
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: x86_64-unknown-linux-gnu
release: 1.83.0
LLVM version: 19.1.1
blaginin
mentioned this on Dec 2, 2024
berkaysynnada
mentioned this on Dec 14, 2024
peter-toth

peter-toth commented on Dec 14, 2024

@peter-toth
peter-toth
on Dec 14, 2024
Contributor

@berkaysynnada, is this sill an issue? rustwasm/wasm-pack#1381 (comment) seems to work for me too.

% export PATH=/opt/homebrew/Cellar/llvm/19.1.4/bin/:$PATH

% clang --version
Homebrew clang version 19.1.4
Target: arm64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/19.1.4/bin
Configuration file: /opt/homebrew/etc/clang/arm64-apple-darwin23.cfg

% rustc --version --verbose
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: aarch64-apple-darwin
release: 1.83.0
LLVM version: 19.1.1

% cargo clean
...

% cargo build --release --target wasm32-unknown-unknown
...
   Compiling datafusion-wasmtest v43.0.0 (/Users/ptoth/git/apache/datafusion/datafusion/wasmtest)
    Finished `release` profile [optimized] target(s) in 1m 36s
berkaysynnada

berkaysynnada commented on Dec 14, 2024

@berkaysynnada
berkaysynnada
on Dec 14, 2024
ContributorAuthor

That worked for me now (and passed the tests). Perhaps I should have tried with cargo clean 😞

berkaysynnada
closed this as completedon Dec 14, 2024
alamb
mentioned this on Dec 23, 2024
robtaylor

robtaylor commented on Jan 22, 2025

@robtaylor
robtaylor
on Jan 22, 2025

I'm still getting this issue. Mac M4 with Sequoia 15.2.

% rustc --version --verbose
rustc 1.84.0 (9fc6b4312 2025-01-07)
binary: rustc
commit-hash: 9fc6b43126469e3858e2fe86cafb4f0fd5068869
commit-date: 2025-01-07
host: aarch64-apple-darwin
release: 1.84.0
LLVM version: 19.1.5

error: failed to build archive at `[cut]/wasm32-unknown-unknown/release/deps/libpsm-5686a1fe6f354cae.rlib`: LLVM error: section too large

warning: psm@0.1.24: warning: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: [cut]wasm32-unknown-unknown/release/build/psm-2da2e2ac984ce6da/out/libpsm_s.a the table of contents is empty (no object file members in the library define global symbols)
error: could not compile `psm` (lib) due to 1 previous error

2 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @robtaylor@peter-toth@blaginin@Eason0729@berkaysynnada

      Issue actions
        "recursive" Dependency Causes "section too large" Error When Compiling for wasm · Issue #13513 · apache/datafusion