Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clang++ shipped with the wasi-sdk errors with LLVM ERROR: out of memory - clang++ shipped with emscripten is ok #326

Closed
yowl opened this issue May 6, 2023 · 25 comments · Fixed by #417

Comments

@yowl
Copy link

yowl commented May 6, 2023

Using the clang++ shipped with wask-sdk, which reports as 16.0.0:

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe --version
clang version 16.0.0
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin

it fails on windows with some input (a 25MB .bc file) with

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc -o obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: e:\\github\\wasi-sdk\\bin\\clang++.exe -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\\GitHub\\wasi-sdk\\share\\wasi-sysroot -c -g3 obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.bc -o obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.o -pthread --sysroot=e:\\github\\wasi-sdk/share/wasi-sysroot -target wasm32-wasi-threads
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc'.
3.      Running pass 'WebAssembly Instruction Selection' on function '@S_P_CoreLib_System_Globalization_UmAlQuraCalendar__InitDateMapping'
Exception Code: 0xC000001D
#0 0x0088fb73
#1 0x75b9e332
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 16.0.0
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

With emscripten's clang++ (also 16.0.0, but maybe a different commit):

E:\tmp\consolelocal>%EMSDK%\upstream\bin\clang++.exe --version
clang version 16.0.0 (https://github.com/llvm/llvm-project 8b587113b746f31b63fd6473083df78cef30a72e)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: E:\GitHub\emsdk\upstream\bin

This passes:

E:\tmp\consolelocal>%EMSDK%\upstream\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc -o obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

The LLVM for the function that is failing is attached, but it might not be that relevant, maybe its just run out of memory in general, this function appears 74% down the input. Also attached the complete .bc input.

S_P_CoreLib_System_Globalization_UmAlQuraCalendar__InitDateMapping.txt
consoleclrjit.zip

@sbc100
Copy link
Member

sbc100 commented May 8, 2023

There are several reasons such a difference between the two compiler binaries might exist. Firstly, the emscripten binaries are build against LLVM tip-of-tree so they are built from very different set of sources. Secondly they are built with LTO which would effect all kind of things, including memory usage.

How much memory does the machine your running on have? Does the issue still occur on machines with a lot more memory? (i.e. is this just a run of mill OOM, or is it some kind of bug?)

@yowl
Copy link
Author

yowl commented May 8, 2023

64GB, when I run it the memory graph in Task Manager shows just a small spike( not a great indicator, but something):
image

@yowl
Copy link
Author

yowl commented May 8, 2023

I don't have anything with more than 64GB, unfortunately.

@sbc100
Copy link
Member

sbc100 commented May 8, 2023

How much memory does the emscripten version clang use?

I guess you could try building wasi-sdk with clang tip-of-tree to see if there is perhaps a bugfix?

@yowl
Copy link
Author

yowl commented May 8, 2023

With emscripten it barely moves
image

I can try clang tip-of-tree with wasi-sdk, or when do you think v21 will be out?

@sbc100
Copy link
Member

sbc100 commented May 9, 2023

wasi-sdk doesn't use tip-of-tree llvm, so the next major llvm update would be when llvm 17 is released (which could be a while).

@yowl
Copy link
Author

yowl commented May 13, 2023

If llvm 16 accepted a bug fix (assuming that's what it is), would wasi-sdk take it?

@sunfishcode
Copy link
Member

Yes, if a fix is backported to the release/16.x branch, wasi-sdk can update to it.

@yowl
Copy link
Author

yowl commented May 14, 2023

Thanks, it looks like upstream is fine. There is clearly a difference between clang++ from the one from Wasi-SDK and https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.0/LLVM-16.0.0-win64.exe. Here is the output from Wasi-SDK's clang++ and LLVM's clang++ . Are there any options I could pass to LLVM's version to make it behave like the one from Wasi-SDK?

E:\tmp\consolelocal>"c:\Program Files\LLVM\bin\clang++.exe" "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc" -o "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc" -o "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: e:\\github\\wasi-sdk\\bin\\clang++.exe obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.bc -o obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.o -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=e:\\github\\wasi-sdk/share/wasi-sysroot -pthread -target wasm32-wasi-threads -c -g3
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc'.
3.      Running pass 'WebAssembly Instruction Selection' on function '@S_P_CoreLib_System_Reflection_Runtime_MethodInfos_RuntimeConstructedGenericMethodInfo__get_ReflectedType'
Exception Code: 0xC000001D
0x0084FB73 <unknown module>
0x7649E332 <unknown module>
0x008323A1 <unknown module>
0x008344C0 <unknown module>
0x010F2CA9 <unknown module>
0x010F1ADD <unknown module>
0x010DB8F9 <unknown module>
0x7732BAFE <unknown module>
0x010DB8AA <unknown module>
0x010EF651 <unknown module>
0x009DA3AB <unknown module>
0x00695AD4 <unknown module>
0x00698B5B <unknown module>
0x00E18697 <unknown module>
0x01043B12 <unknown module>
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 16.0.0
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

@abrown
Copy link
Collaborator

abrown commented Aug 22, 2023

@sunfishcode and I were discussing this: before we decide what to do here we need to understand better what is the difference in behavior between this LLVMv16 Clang and the wasi-sdk Clang. One suggestion would be to compile with -S with both versions and see if there is any difference in that output?

@yowl
Copy link
Author

yowl commented Aug 23, 2023

I could try -S on a simple C program, I can't do it on this input as the SDK clang++ fails with LLVM ERROR: out of memory

@yowl
Copy link
Author

yowl commented Aug 23, 2023

Only thing I've noticed that is different is for WASI SDK the default target is different:

clang -cc1 version 16.0.0 based upon LLVM 16.0.0 default target wasm32-wasi

And the LLVM download:

clang -cc1 version 16.0.0 based upon LLVM 16.0.0 default target x86_64-pc-windows-msvc

Probably not interesting though?

@yowl
Copy link
Author

yowl commented Sep 7, 2023

Small update, clang++ from the wasi-sdk for Linux is also fine:

scott@Matumbo:~/clang-repro$ ../wasi-sdk-20.0/bin/clang++ consoleclrjit.bc -o consoleclrjit.o -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

So problem seems isolated to the Windows build.

@yowl
Copy link
Author

yowl commented Jan 12, 2024

Tried with sdk 21 and problem persists:

C:\tmp>%WASI_SDK_PATH%\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=c:\GitHub\wasi-sdk21\share\wasi-sysroot -c -g3 consoleclrjit.bc -o consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: c:\\github\\wasi-sdk21\\bin\\clang++.exe -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=c:\\GitHub\\wasi-sdk21\\share\\wasi-sysroot -c -g3 consoleclrjit.bc -o consoleclrjit.o -pthread --sysroot=c:\\github\\wasi-sdk21/share/wasi-sysroot -target wasm32-wasi-threads
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'consoleclrjit.bc'.
3.      Running pass 'WebAssembly Explicit Locals' on function '@S_P_TypeLoader_Internal_TypeSystem_MethodSignature__Equals_0'
Exception Code: 0xC000001D
 #0 0x002f4c15
 #1 0x76dce332
 #2 0x002d6d34
 #3 0x002d8de4
 #4 0x00b15963
 #5 0x00b15cd8
 #6 0x00b15c42
 #7 0x00b15bd4
 #8 0x004496fa
 #9 0x028a7400
#10 0x027f328c
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 17.0.6
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: c:/github/wasi-sdk21/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

@yowl
Copy link
Author

yowl commented Mar 9, 2024

More info, the x86 version of clang++ fails as above, the x64 version passes. Is there a reason the SDK uses the 32 bit version, do LLVM even support that properly ?

@yowl
Copy link
Author

yowl commented Mar 9, 2024

Wasi-SDK includes the 32 bit version.

@abrown
Copy link
Collaborator

abrown commented Mar 12, 2024

I checked and this is not the case:

$ file wasi-sdk-21.0/bin/clang-17
wasi-sdk-21.0/bin/clang-17: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=2b3af36d4a17f9cb2a3c33a3d530eece25498ba6, stripped

At least not on Linux, but now I see that your examples are all Windows-based. Can you confirm via some shell output that in fact those binaries are all 32-bit on Windows? Seems a bit weird but maybe there's some Makefile setting that needs to be tweaked (?).

@yowl
Copy link
Author

yowl commented Mar 12, 2024

Sure:

C:\github\runtimelab>dumpbin /headers %WASI_SDK_PATH%\bin\clang.exe
Microsoft (R) COFF/PE Dumper Version 14.38.33135.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file c:\github\wasi-sdk21\bin\clang.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               9 number of sections
        657CFC0A time date stamp Fri Dec 15 20:23:22 2023
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

@abrown
Copy link
Collaborator

abrown commented Mar 13, 2024

Ok, I think this makes things more clear. What might be happening here is that the script that downloads the release artifacts is first extracting the x64 version and then overwriting this with the x86 version :

ls -1 *.zip | xargs -n1 unzip -q -o -d release

Maybe a fix for this is to do something similar to the exclusion added to skip one of the Ubuntu versions:

if [ "${NAME}" = "dist-ubuntu-latest" ]; then
continue
fi

One way to check whether all of this is in fact the case would be to retrieve the x64 binary from this workflow: https://github.com/WebAssembly/wasi-sdk/actions/runs/7228245836. That workflow is the one that was used for creating the wasi-sdk-21 release in which I am guessing we used the x86 version instead. Can you see if the clang in the dist-windows-latest-x64 zip file fixes the issue you were seeing?

@yowl
Copy link
Author

yowl commented Mar 13, 2024

Yes, I confirm that clang fixes the issue

C:\tmp>"C:\Users\scott\Downloads\wasi-sdk-21.0+m-mingw.tarx\wasi-sdk-21.0+m\bin\clang.exe" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 c:\tmp\consoleclrjit.bc -o c:\tmp\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

C:\tmp>

@dicej
Copy link
Contributor

dicej commented May 8, 2024

What's left to do here? Do we need to make a 22.1 release that includes the x64 build of clang etc. instead of the x86 build?

@abrown
Copy link
Collaborator

abrown commented May 8, 2024

This fell off my radar: the issue felt resolved in my mind although in fact no one ever modified download-workflow-artifacts.sh. @dicej, are you interested in resolving this? I think the first thing to do is to modify that script so that future releases use the right binary. After that, yeah, we could draft a new release or probably even modify the release artifacts for wasi-sdk-22 (?).

@dicej
Copy link
Contributor

dicej commented May 8, 2024

Yeah, I'll put it on my to-do list and open a PR today or tomorrow.

dicej added a commit to dicej/wasi-sdk that referenced this issue May 9, 2024
This helps ensure the identically-named 32-bit tarfile doesn't overwrite the
64-bit one when we extract everything into a single directory.

I'll follow this up by making an updated wasi-sdk-22.0.m-mingw.tar.gz and adding
it to the wasi-sdk-22 release.

Fixes WebAssembly#326

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej
Copy link
Contributor

dicej commented May 9, 2024

I've addressed this for future releases in #417, and @alexcrichton has kindly added https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-22/wasi-sdk-22.0.m-mingw64.tar.gz to the wasi-sdk-22 release. Note the mingw64 part of the name in that URL -- we gave it a different name to avoid confusion since we're doing this retroactively. For future releases there will just be a single, 64-bit Windows tarball.

abrown pushed a commit that referenced this issue May 9, 2024
This helps ensure the identically-named 32-bit tarfile doesn't overwrite the
64-bit one when we extract everything into a single directory.

I'll follow this up by making an updated wasi-sdk-22.0.m-mingw.tar.gz and adding
it to the wasi-sdk-22 release.

Fixes #326

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@abrown
Copy link
Collaborator

abrown commented May 9, 2024

@dicej, thanks for fixing this!

jsturtevant added a commit to bytecodealliance/componentize-dotnet that referenced this issue Jun 6, 2024
Remove emscripten dependency and  use correct wasi-sdk url for windows: WebAssembly/wasi-sdk#326
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants