New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to compile a simple Swift file (maybe not possible?) #2427

Open
geelen opened this Issue Jun 14, 2014 · 83 comments

Comments

Projects
None yet
@geelen

geelen commented Jun 14, 2014

Hi team, playing around with Emscripten in earnest for the first time, not my usual area of work so apologies if this is clearly never going to work. What I'd like to do is compile a simple Swift program to JS:

println("Hello, world!")

So far what I've tried is:

xcrun swift hello_world.swift -emit-bc -o hello_world.bc
emcc hello_world.bc

but that fails because it can't find certain symbols (looks like the standard library isn't present):

Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Drilling a bit further, and looking at how xcrun swift -v hello_world.swift actually works, I'm able to use this linking command to compile the bitcode to an executable using normal ld.

cp /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libswift_stdlib_core.dylib .
/usr/bin/ld hello_world.bc \
              -lSystem -arch x86_64 \
              -L . -rpath . \
              -o hello_world
./hello_world
=> Hello, world!

So, it seems like libswift_stdlib_core.dylib is the only dependency for hello_world.bc to be properly linked to an executable.

But I'm stuck - is there some equivalent between the -L and -rpath flags on ld that I should be passing to emcc? Or is a dylib like that not possible to be used in emscripten? The final command I tried was:

> emcc libswift_stdlib_core.dylib hello_world.bc                                 ~/src/experiments/swift.js • 2.1.0p0
WARNING  root: emcc: cannot find library "swift_stdlib_core"
Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Hopefully something's possible from here - Swift is a neat high-level language with a nice type system and no garbage collector, which makes me think it's a good fit for compiling to JS.

@juj

This comment has been minimized.

Collaborator

juj commented Jun 14, 2014

Unfortunately it is not possible to link .dylib files to Emscripten. The reason for that is that the .dylibs already contain native machine code for x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form again. What one would have to do is implement the standard library for Swift and compile that in. The unrecognized struct value errors sounds like something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions with your own implementation?

@geelen

This comment has been minimized.

geelen commented Jun 14, 2014

Interesting, I suspected as much, but had hoped dylibs might have llvm
bitcode in there.

I'll take a look to see how much of the Swift standard library source is
available to compile from, but I fear it won't be enough.

Appreciate the response, thanks!

On Saturday, June 14, 2014, juj notifications@github.com wrote:

Unfortunately it is not possible to link .dylib files to Emscripten. The
reason for that is that the .dylibs already contain native machine code for
x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form
again. What one would have to do is implement the standard library for
Swift and compile that in. The unrecognized struct value errors sounds like
something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions
with your own implementation?


Reply to this email directly or view it on GitHub
#2427 (comment).

@iongion

This comment has been minimized.

iongion commented Oct 5, 2014

What if we implement our own standard lib, according to http://practicalswift.com/2014/06/14/the-swift-standard-library-list-of-built-in-functions there are just 74 functions

@JoshCheek

This comment has been minimized.

JoshCheek commented Oct 20, 2014

If the println command was omitted, would this work? (it didn't for me, I still got the same result, but I'm very ignorant) Then one could create a library and just call it from js without it depending on anything.

Also, an update, the commands in the first post seem deprecated, here is what I tried:

Versions

$ xcrun swiftc --version
Swift version 1.0 (swift-600.0.45.3.2)
Target: x86_64-apple-darwin13.2.0

$ emcc --version
emcc (Emscripten GCC-like replacement) 1.25.0 ()
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The file to compile:

$ cat f.swift
var message = 1

The IR

$ xcrun swiftc -emit-ir f.swift
; ModuleID = '-'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

%Si = type <{ i64 }>

@_Tv1f7messageSi = global %Si zeroinitializer, align 8

define internal void @top_level_code() {
entry:
  store i64 1, i64* getelementptr inbounds (%Si* @_Tv1f7messageSi, i32 0, i32 0), align 8
  ret void
}

define i32 @main(i32 %argc, i8** %argv) {
entry:
  %0 = call i8* @_TFSsa6C_ARGCVSs5Int32()
  %1 = bitcast i8* %0 to i32*
  store i32 %argc, i32* %1
  %2 = call i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()
  %3 = bitcast i8* %2 to i8***
  store i8** %argv, i8*** %3
  call void @top_level_code()
  ret i32 0
}

declare i8* @_TFSsa6C_ARGCVSs5Int32()

declare i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()

!llvm.module.flags = !{!0, !1, !2, !5, !6, !7, !8}

!0 = metadata !{i32 2, metadata !"Dwarf Version", i32 3}
!1 = metadata !{i32 1, metadata !"Debug Info Version", i32 1}
!2 = metadata !{i32 6, metadata !"Linker Options", metadata !3}
!3 = metadata !{metadata !4}
!4 = metadata !{metadata !"-lswiftCore"}
!5 = metadata !{i32 1, metadata !"Objective-C Version", i32 2}
!6 = metadata !{i32 1, metadata !"Objective-C Image Info Version", i32 0}
!7 = metadata !{i32 1, metadata !"Objective-C Image Info Section", metadata !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!8 = metadata !{i32 4, metadata !"Objective-C Garbage Collection", i32 0}

The SIL (IDK if it's useful, but can't hurt):

$ xcrun swiftc -emit-sil f.swift
sil_stage canonical

import Builtin
import Swift
import SwiftShims

var message: Int

// top_level_code
sil private @top_level_code : $@thin () -> () {
bb0:
  %0 = global_addr #message : $*Int               // user: %3
  %1 = integer_literal $Builtin.Word, 1           // user: %2
  %2 = struct $Int (%1 : $Builtin.Word)           // user: %3
  store %2 to %0 : $*Int                          // id: %3
  %4 = tuple ()                                   // user: %5
  return %4 : $()                                 // id: %5
}

// Swift.Int._convertFromBuiltinIntegerLiteral (Swift.Int.Type)(Builtin.Int2048) -> Swift.Int
sil public_external [transparent] @_TFSi33_convertFromBuiltinIntegerLiteralfMSiFBi2048_Si : $@thin (Builtin.Int2048, @thin Int.Type) -> Int {
bb0(%0 : $Builtin.Int2048, %1 : $@thin Int.Type):
  %2 = builtin_function_ref "s_to_s_checked_trunc_Int2048_Word" : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %3
  %3 = apply %2(%0) : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %4
  %4 = tuple_extract %3 : $(Builtin.Word, Builtin.Int1), 0 // user: %5
  %5 = struct $Int (%4 : $Builtin.Word)           // user: %6
  return %5 : $Int                                // id: %6
}

Trying to compile with emcc:

$ xcrun swiftc -emit-bc f.swift -o f.bc
$ emcc f.bc
WARNING: Linking two modules of different data layouts: '/Users/josh/.emscripten_cache/libc.bc' is 'e-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-p:32:32:32-v128:32:128-n32-S128' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'
WARNING: Linking two modules of different target triples: /Users/josh/.emscripten_cache/libc.bc' is 'asmjs-unknown-emscripten' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'x86_64-apple-darwin13.2.0'
Unknown specifier in datalayout string
UNREACHABLE executed at /Users/clb/emscripten-fastcomp/lib/IR/DataLayout.cpp:300!
0  opt                      0x00000001086d04ae llvm::sys::PrintStackTrace(__sFILE*) + 46
1  opt                      0x00000001086d07bb PrintStackTraceSignalHandler(void*) + 27
2  opt                      0x00000001086d0b4c SignalHandler(int) + 412
3  libsystem_platform.dylib 0x00007fff8b0e35aa _sigtramp + 26
4  libsystem_platform.dylib 0x00007fff6492d380 _sigtramp + 3649347056
5  opt                      0x00000001086d07eb raise + 27
6  opt                      0x00000001086d08a2 abort + 18
7  opt                      0x000000010865a7a6 llvm::llvm_unreachable_internal(char const*, char const*, unsigned int) + 198
8  opt                      0x0000000108416b74 llvm::DataLayout::parseSpecifier(llvm::StringRef) + 2804
9  opt                      0x0000000108415c57 llvm::DataLayout::init(llvm::StringRef) + 471
10 opt                      0x000000010749b47e llvm::DataLayout::DataLayout(llvm::StringRef) + 158
11 opt                      0x0000000107482ba5 llvm::DataLayout::DataLayout(llvm::StringRef) + 37
12 opt                      0x000000010747943c main + 3756
13 libdyld.dylib            0x00007fff8a2865fd start + 1
Stack dump:
0.  Program arguments: /Users/josh/code/emsdk_portable/clang/e1.25.0_64bit/opt /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc -strip-debug -internalize -internalize-public-api-list=main,malloc,free -globaldce -pnacl-abi-simplify-preopt -pnacl-abi-simplify-postopt -enable-emscripten-cxx-exceptions -o /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc.opt.bc
Traceback (most recent call last):
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/emcc", line 1224, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/tools/shared.py", line 1357, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:
@JoshCheek

This comment has been minimized.

JoshCheek commented Oct 20, 2014

If anyone can look at the above and give me an idea of what specifically it's missing, ie "it's trying to link stdlib.swift" and a link to somewhere that someone documents how they solved a similar problem, then I'll put a few hours into trying to get it. If there's any reasonable progress in that time, I'll report it here and probably keep at it.

Right now I'm just so ignorant that I don't even know what what the problem is or what to look for in terms of resolving it. But I have like 50 options I'm considering for my project, including things like Elm which I've already put a few days into playing with, so I need some guidance to not wander in circles for a months.

@kripken

This comment has been minimized.

Owner

kripken commented Oct 20, 2014

The targets need to match. As the IR shows, the swift frontend emits

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

Emscripten uses its own triple (asmjs-unknown-emscripten). You might be able to force the frontend to use it? If not, you might be able to just edit the LLVM IR file in text format to fix it (create one using emcc, to see the right values for those two lines). But, this is risky, as the triple affects stuff that is baked into the output in various places...

@JoshCheek

This comment has been minimized.

JoshCheek commented Oct 21, 2014

So, I'm ignorant enough that I don't know how to interpret "the targets need to match" (I don't know what "target" means, or what a "triple" is -- kinda sounds like a fixed-length set of memory, like a "tuple" in a functional language). So it's unclear to me if it's something that I fucked up (e.g. did I compile it wrong, or with my own version of some binary, since emscripten uses its own set of binaries -- e.g. python2 was not in the PATH, and I couldn't find it in the emsdk_portable, so I just made a bash script named python2 that then invoked python and forwarded the args. Seemed reasonable since I had Python 2.7.5 installed, which the readme alleged was compatible https://github.com/kripken/emscripten/blob/06961a0ef6e3d8d92d5e36ff904262fefec62bec/tests/python/readme.md, and xcrun is some binary that I don't understand, which is just sort of generally available, but didn't come from emscripten) or is it some issue caused by Swift being overly zealous about an OSX environment, or is it some flag to find to tell it to chill the fuck out and forget about being execuatble and just compile as if some other program will load it and deal with all the integration stuff? I mean, you'll notice all I do in this script is declare a variable and assign it the value 1, so what is it trying to dynamically load? And does that have anything to do with the "targets" not matching?

I'll create the file using emcc and diff them and spend an hour poring over the differences, confused and trying and failing repeatedly, to figure out what I need to do, but I don't know how. The bc format si the only one I've found so far that emcc will even consider trying to compile.

The reality is that I have no particularly good model for figuring out how this stuff works. I once spent 8 hours getting a C program to compile and load a library I was interested in, that's about the extent of my experience. Code that operates at this level operates under a model that I am simply ignorant of, so while we're in this domain, it's reasonable to me like I've never programmed before.

I'm actually pretty good at guessing and cursing repeatedly trying and failing and guessing and cursing again, but I need some sort of model or context to iterate upon. And there's a sufficiently large amount of information available out there, that I need someone who's familiar with this domain to clue me in, and imply to me that this isn't a giant waste of my time (b/c, lets be honest, I could just suck it up and write my shit in JavaScript).

@kripken

This comment has been minimized.

Owner

kripken commented Oct 21, 2014

A target is what clang/LLVM is emitting code for. When you build normally, on OSX your target triple is something like darwin-intel-apple or some other combo that contains the information that you are building for intel hardware, to run on a darwin kernel on an apple userspace. This affects codegen in various ways, and the LLVM IR is not portable because of it.

Emscripten has its own target, and emcc tells clang and LLVM to use it, -triple=asmjs-unknown-emscripten or something like that.

@JoshCheek

This comment has been minimized.

JoshCheek commented Oct 22, 2014

Hmm. Guess I thought LLVM was the target.

Anyway, asmjs-unknown-emscripten looks correct:

$ ag asmjs-unknown-emscripten | wc -l
     115

-triple isn't the correct flag, apparently, and it tells me the wrong help file:

$ xcrun swiftc -Xllvm "-triple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '-triple=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-spiller=asmjs-unknown-emscripten'?

$ swift '(LLVM option parsing)' -help
fish: Unknown command 'swift'

$ xcrun swift '(LLVM option parsing)' -help
<unknown>:0: error: no such file or directory: '(LLVM option parsing)'

$ xcrun swift 'LLVM option parsing' -help
<unknown>:0: error: no such file or directory: 'LLVM option parsing'

$ xcrun swift 'option parsing' -help
<unknown>:0: error: no such file or directory: 'option parsing'

llvm isn't a binary, but tab-complete suggestions include llvm-gcc and llvm-g++, their help screens don't mention a triple, but do talk about a --target, but that doesn't work either. Which is strange, because it's listed in the help output, so it seems like the error should be something about using it wrong, rather than "Unknown command line argument", so maybe this isn't the binary that it invokes for llvm:

$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '--target=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-stats=asmjs-unknown-emscripten'?

# same output for each of these potential variations:
$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target" -Xllvm "asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# for that last one, it says "swift (LLVM option parsing): Did you mean '-stats'?"
# so lets verify that llvm-gcc is the right binary by seeing if `-stats` is one of its options:
$ llvm-gcc --help | grep stats

# ...nope, what the fuck is llvm?
# searching implies its binary is named "clang" for some reason,
# but no dice here either:
$ clang --help | grep stats

# man pages have any ideas?
$ man llvm-gcc
No manual entry for llvm-gcc

$ man llvm-g++
No manual entry for llvm-g++

$ man clang # this one works!

# After looking through this man page, I try -arch, which doesn't do shit
# was going to try setting MACOSX_DEPLOYMENT_TARGET env var
# but then I realized there's some way to search all man pages for "-stats"
# some stupid searching (man man) eventually I figure out:
$ man -K stats

# go through this list, and `gcc-4.8` shows up, so maybe llvm is just gcc? *shrug* lets try it
# it turns out to be fruitless
# at this point, I'm considering trying to find in the emscripten code where they invoke it
# figure I'll try googling a bit, find some docs for `llc`, which isn't a binary on my system, but does have a `-mtriple` flag, so lets try that:
$ xcrun swiftc -Xllvm "-mtriple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# run the gamut of possible ways that I'm supposed to pass this thing
# (why can't I find a fucking example of how to do this?)

# eventually, one of the options suggests "-spiller" as a correction possibility
$ man -K spiller  # ...just some thing about crypto

At this point, documenting in code would add another hour, but many searches, fancy -k and -K uses of the man flag. A google search for llvm -spiller and -stats takes me back to that llc page, look through it a second time, this time notice that they have two dashes before them, even though the suggestion when I got it wrong only has 1 dash. So, try the permutations of possible ways to pass an argument to -Xllvm with two dashes. None of them work.

Get pissed, internet search like the fourth time for an example of how to use -Xllvm flag, no results, cd to xcode and search for the string 'Xllvm', nothing. Try emsdk_portable, nothing.

Look one more time at llvm-gcc, llvm-g++, lldb, apropos llvm, clang, nothing.

Decide to go see if I can find where this happens in emsdk_portable, looks like the makefiles set a build_triple but never use it. Probably some makefile magic, and I don't feel like reading a book about makefiles just to find out how this gets persisted. Notice some of the files use LLVM_TARGET, I don't know Python, maybe that's an env var, so try compiling with that set env LLVM_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc, but it doesn't do anything (triple is still set to "x86_64-apple-darwin13.2.0"), remember the clang man pages had an env var that seemed like it had potential, so env MACOSX_DEPLOYMENT_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc. Nope.

So, I'm like 3 hours in, and literally no further than when I started. Looks like someone got Rust compiling to asm.js, maybe that's a better choice. Someone got Go compiling directly to JavaScript. Might be sufficient. Could go suffer through the Elm type system for another week or two (it was basically like this: hours of slamming myself into the wall, but I kept at it for like 3 times longer), or maybe write the thing in C, since that at least compiles, but some part of me thinks that's masochism. If I did it in Opal, I wouldn't have to learn a new language... but that's part of the appeal, and I'm not sure it'll be able to scale in directions I might want to go with my project.

Going to go for a drink and weigh the shittiness of C as a language against the fact that I'll almost certainly succeed.

@endash

This comment has been minimized.

endash commented Feb 20, 2015

I was able to get very, very simple programs to build with the 6.3 beta at least. Unfortunately even something as simple as a string literal will not work, and regardless the resulting file has a bunch of missing symbols. Some of the symbols (swift_once) are defined in libswiftCore.dylib, so it's likely using Swift will be impossible until the standard library is open sourced.

@guidobouman

This comment has been minimized.

guidobouman commented Jul 13, 2015

So, luckily, this will help later on: https://developer.apple.com/swift/blog/?id=29

@nmn

This comment has been minimized.

nmn commented Jul 16, 2015

So I was trying to do the same thing, and I just stumbled upon this Issue. So glad that I won't waste my time again.

By the way did you try again with XCode 7 and the swiftc tool??

@stepanhruda

This comment has been minimized.

stepanhruda commented Oct 6, 2015

Since Apple now supports LLVM Bitcode, would that enable including dylibs that contain it?

@kripken

This comment has been minimized.

Owner

kripken commented Oct 6, 2015

LLVM bitcode is not portable. Bitcode for the App Store would be specific to Apple's hardware and OS.

@stepanhruda

This comment has been minimized.

stepanhruda commented Oct 8, 2015

Thanks, that makes sense. In case anyone is interested in more details as I was, LLVM FAQ: Can I compile C or C++ code to platform-independent LLVM bitcode?

@geelen

This comment has been minimized.

geelen commented Dec 3, 2015

It's been 18 months since I started this issue and I still really want this to happen. With Swift going open-source today let's hope that means it's now possible!

https://github.com/apple/swift 🎉

@kripken

This comment has been minimized.

Owner

kripken commented Dec 4, 2015

This will definitely make it a lot easier! :)

Ok, to move forward here, we need to do the following:

  1. Figure out how to use emscripten's llvm with swift's llvm. Does swift have a fork of llvm, or can it use stock? If the latter, then this should be easy, just make swift use emscripten's llvm. We just need to see which version of llvm it expects (the incoming branch is on trunk llvm from last week, hopefully that is good and swift doesn't need an older tag).
  2. Get swift to emit code with the asmjs-unknown-emscripten triple.
  3. Get swift to stop at the bitcode stage.
  4. Run emcc on that bitcode and see what happens.
  5. We would need similar bitcode of whatever runtime libraries swift requires, except for libc etc. which we already have.
@Gaelan

This comment has been minimized.

Gaelan commented Dec 4, 2015

Swift does have a fork of LLVM at apple/swift-llvm. Not sure what changes they have made.

@nmn

This comment has been minimized.

nmn commented Dec 4, 2015

the swift-cli has a bunch of handy options to output just the llvm byte code etc.

@jcampbell05

This comment has been minimized.

jcampbell05 commented Dec 6, 2015

Let me know what I can help with would love to have something working!

@kripken

This comment has been minimized.

Owner

kripken commented Dec 6, 2015

Thanks for the info @Gaelan. Ok, if they have their own fork, then the best workflow would probably be:

  1. Upstream our target triple to LLVM
  2. When they merge in new LLVM, they'll get that
  3. Get Swift to emit bytecode to our target triple
  4. Run that bytecode through emcc

The first step is to upstream our triple, which is already partially there (due to nacl upstreamings), so it's a small patch and probably not controversial. What we need to send is basically this:

https://gist.github.com/kripken/0b7ba068faf21d5449a3

Anyone interested to help upstream that?

@ianyh

This comment has been minimized.

ianyh commented Dec 7, 2015

It looks like Apple's llvm clone is pretty strict about changes being made upstream unless they are specifically swift-related. See: https://swift.org/contributing/#llvm-and-swift

@jcampbell05

This comment has been minimized.

jcampbell05 commented Dec 7, 2015

Could open an issue in the Swift LLVM github to ask them for changes

@ianyh

This comment has been minimized.

ianyh commented Dec 7, 2015

I think that reduces to asking them to merge the changes in themselves, which is against their policy. The patch would have to be submitted directly to llvm by someone here.

I have no familiarity with the llvm community, so I don't know how they would feel about, for example, introducing asmjs as an architecture. Anyone else have thoughts?

@sberan

This comment has been minimized.

sberan commented Dec 7, 2015

This is very exciting @kripken! In theory, should we be able to make these modifications locally and try things out using the swift LLVM clone? I modified the relevant files in the swift LLVM clone, according to your patch - here is the diff of the changes I made: https://gist.github.com/sberan/43bc5fbff78ea47658f0

Unfortunately I don't see a new target for asmjs in the swift recompiled LLVM:

./build/Ninja-DebugAssert/llvm-macosx-x86_64/bin/llc --version
LLVM (http://llvm.org/):
  LLVM version 3.8.0svn
  DEBUG build with assertions.
  Built Dec  6 2015 (19:49:40).
  Default target: x86_64-apple-macosx10.9
  Host CPU: haswell

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_be - AArch64 (big endian)
    arm        - ARM
    arm64      - ARM64 (little endian)
    armeb      - ARM (big endian)
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64

Maybe more work needs to be done here in order to register asmjs as a target? Forgive me if this is a noob question, I'm very new to llvm and emscripten, just very excited to try swift in the browser 😄

I'd be happy to try and push these changes upstream, but I'd like to verify that they'll work first! 😆

@ianyh

This comment has been minimized.

ianyh commented Dec 7, 2015

I'll try to give it a go when I get a chance!

@kripken

This comment has been minimized.

Owner

kripken commented Dec 8, 2015

I think you might not see a target because you can't build a full executable for js. But you should be able to build with -target asmjs-unknown-emscripten -emit-llvm and it should emit llvm for that specific target triple.

@kripken

This comment has been minimized.

Owner

kripken commented Jan 6, 2016

There is another possible path here. Upstream LLVM has a WebAssembly triple now. It is almost identical to our asm.js triple. It should be possible for swift to build to the wasm-wasm-wasm triple, then import that bitcode into emscripten which can compile it to asm.js.

(We would still need to make sure it's the same LLVM version, or close enough, to avoid e.g. debug info changes. But emscripten merges upstream every week or two now, so that should be easy, if someone tells me what is a good time to merge for Swift, i.e., when Swift merges.)

Note that the wasm triple is still experimental, so you need to build LLVM with it enabled, from the list of experimental targets.

@jcampbell05

This comment has been minimized.

jcampbell05 commented Jan 6, 2016

@kripken - This is an overview of the release process for Swift https://swift.org/blog/swift-2-2-release-process/

@naan

This comment has been minimized.

naan commented Feb 1, 2016

Swift uses llvm 3.8.0. They have their branch and not exactly sure their policy but once they finish swift 2.2, they'll probably merge their branch to upstream trunk.

3.8.0 already has wasm as kripken said, so I was able to build swift-llvm with wasm64 enabled and pass basic tests, but struggling to build swift with llvm-wasm and build swift stdlibs with with llvm-wasm + emscripten, main reason is that their build system is huge but not so flexible. (There're lots places that have hard-coded config like `if system==darwin then build for Mac, else build linux kind of thing) But I'm new for their build system so I may miss something.

I just wanted to make sure the process I'm doing is in the right course.

  • Build llvm with wasm enabled (already done)
  • Setup emscripten to use llvm-wasm enabled (not the one bundled with emscripten)
  • Build swift with llvm-wasm cross compile enabled, build swift stdlibs with llvm-wasm + emscripten
@ephemer

This comment has been minimized.

ephemer commented Feb 16, 2017

@goloveychuk not that I'm aware of

@cacaodev

This comment has been minimized.

cacaodev commented Jun 17, 2017

Any news ?

@helje5

This comment has been minimized.

helje5 commented Jun 17, 2017

I don't know how relevant this is for the question, but Swift Package Manager now supports cross compilation. I've built X toolchains for compiling Raspberry Pi Swift binaries on macOS and the reverse. Both however require the availability of Swift binaries (which we build on Raspi instead of x-compiling Swift itself).

My understanding is that CLang itself is always a cross compiler and comes w/ all supported targets builtin (i.e. the standard macOS clang can build for the Raspi armhf target out of the box, you just have to bring the toolchain). So if

Upstream LLVM has a WebAssembly triple now.

I suppose a lot may come for free. Has that 'upstream' be part of a release yet? Maybe the Xcode 9 one includes it? (is there a way to list the supported architectures by clang?)

Also, I think Swift 4 SPM defaults to building static libraries now.

@n8gray

This comment has been minimized.

n8gray commented Jun 23, 2017

Swift running in the browser at native-code speed? Heck yeah, let's make this happen!

I've downloaded the swift compiler and the emscripten sdk. I've successfully used emscripten to build and run "Hello world" in C. I tried passing "-target wasm32-wasm32-wasm32" to the latest Xcode 9 swiftc but it didn't work.

I'd like to do the process that @ephemer suggested above but I'll need a little guidance. Can you elaborate a bit on what the steps would be?

@n8gray

This comment has been minimized.

n8gray commented Jun 26, 2017

Oh hey, wasm in WebKit: https://webkit.org/blog/7691/webassembly/

Also, the Xcode 9 build of Clang supports the wasm32 and wasm64 targets:

% xcrun clang --version
Apple LLVM version 9.0.0 (clang-900.0.26)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

% xcrun clang -emit-llvm --target=wasm32 -Oz fib.c -c -o fib.bc
@Danappelxx

This comment has been minimized.

Danappelxx commented Feb 9, 2018

...any updates?

@goloveychuk

This comment has been minimized.

goloveychuk commented Feb 9, 2018

I tried to compile swift llvm ir code to wasm via emscripten - some syntax problems.
I guess we should try to build apple/swift-llvm and upstream llvm from source and try to compile ir code.
But this will likelly fail, sine apple frontend itself should support wasm32 target. This will require porting apple/swift for wasm32 target (same it was with android, but looks like, will requiremore significant changes)

@goloveychuk

This comment has been minimized.

goloveychuk commented Feb 10, 2018

Achieved small success.

  1. build llvm from sources. https://github.com/yurydelendik/wasmception
  2. swiftc -emit-ir -Onone code.swift -o code.ll
  3. open code.ll and remove
!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8, !9}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
!9 = !{!"-lobjc"}

and !5 from

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !10, !11}

(looks like swift-llvm is older than upstream llvm)
4) try llc -mtriple wasm32 code.ll
you will see WebAssembly doesn't support non-C calling conventions
So problem is that swift using swiftcc calling convention. Wasm supports only ccc calling convention.
https://llvm.org/docs/LangRef.html#calling-conventions
(if you comment this function llc will produce .s file.)
@kripken do you know, is it really possible to bring swiftcc convention to wasm target?

@kripken

This comment has been minimized.

Owner

kripken commented Feb 10, 2018

The calling conventions are mostly for the backends, so it should be possible to just ignore it. That is, if swift didn't emit it, it should be ok, and for now you can strip it from the IR manually for testing. (There are exceptions like tail call optimizations that are done on IR and depend on the convention, but I don't think that would be an issue here.)

If you do that, is the output runnable? The simplest way to check is probably to use the wasm backend with emscripten instead of calling llc and s2wasm etc. yourself, that is, run emcc fixed.ll for LLVM IR with the calling convention stuff fixed up, and where emcc is set up to use latest LLVM built with the wasm backend (can set the LLVM env var to point it to that build, or edit ~/.emscripten).

@goloveychuk

This comment has been minimized.

goloveychuk commented Feb 10, 2018

ok, i tried emcc.
After swiftc we have

define i32 @main(i32, i8**) #0 {
entry:
  %2 = bitcast i8** %1 to i8*
  ret i32 0
}

define hidden swiftcc void @_T04code4mainyyF() #0 {
entry:
  br label %0

; <label>:0:                                      ; preds = %entry
  br label %2

; <labe
...

So we have signature and implementation separately. Emcc catches first function, so we have no output. Ok, I merged it to one function.
This is whole code.

; ModuleID = 'code.ll'
source_filename = "code.ll"
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "asmjs-unknown-emscripten"

%struct._SwiftEmptyArrayStorage = type { %struct.HeapObject, %struct._SwiftArrayBodyStorage }
%struct.HeapObject = type { %struct.HeapMetadata*, %struct.InlineRefCounts }
%struct.HeapMetadata = type opaque
%struct.InlineRefCounts = type { i32, i32 }
%struct._SwiftArrayBodyStorage = type { i64, i64 }
%swift.type = type { i64 }
%swift.bridge = type opaque
%Any = type { [24 x i8], %swift.type* }
%TSi = type <{ i64 }>

@_swiftEmptyArrayStorage = external global %struct._SwiftEmptyArrayStorage, align 8
@_T0SiN = external global %swift.type, align 8
@__swift_reflection_version = linkonce_odr hidden constant i16 3
@llvm.used = appending global [1 x i8*] [i8* bitcast (i16* @__swift_reflection_version to i8*)], section "llvm.metadata", align 8

define i32 @main(i32, i8**) #0 {
entry:
  %2 = bitcast i8** %1 to i8*
  ; ret i32 0

  ; br label %0

; <label>:0:                                      ; preds = %entry
  ; br label %2

; <label>:1:                                      ; preds = %2
  br label %3

; <label>:2:                                      ; preds = %0
  ; br label %1

; <label>:3:                                      ; preds = %1
  %4 = call swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64 1)
  %5 = extractvalue { %swift.bridge*, i8* } %4, 0
  %6 = extractvalue { %swift.bridge*, i8* } %4, 1
  %7 = call %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge* %5) #2
  call void @swift_bridgeObjectRelease(%swift.bridge* %5) #2
  %8 = bitcast i8* %6 to %Any*
  %9 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 1
  store %swift.type* @_T0SiN, %swift.type** %9, align 8
  %10 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
  %11 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
  %12 = bitcast [24 x i8]* %11 to %TSi*
  %._value = getelementptr inbounds %TSi, %TSi* %12, i32 0, i32 0
  store i64 578, i64* %._value, align 8
  %13 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_()
  %14 = extractvalue { i64, i64, i64 } %13, 0
  %15 = extractvalue { i64, i64, i64 } %13, 1
  %16 = extractvalue { i64, i64, i64 } %13, 2
  %17 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_()
  %18 = extractvalue { i64, i64, i64 } %17, 0
  %19 = extractvalue { i64, i64, i64 } %17, 1
  %20 = extractvalue { i64, i64, i64 } %17, 2
  call swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge* %5, i64 %14, i64 %15, i64 %16, i64 %18, i64 %19, i64 %20)
  ret i32 0
}

; Function Attrs: noinline
declare swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge*, i64, i64, i64, i64, i64, i64) #1

declare swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64) #0

declare %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge*)

declare void @swift_bridgeObjectRelease(%swift.bridge*)

; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_() #1

; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_() #1

attributes #0 = { "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #1 = { noinline "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #2 = { nounwind }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !10, !11}

!0 = !{i32 1, !"Objective-C Version", i32 2}
!1 = !{i32 1, !"Objective-C Image Info Version", i32 0}
!2 = !{i32 1, !"Objective-C Image Info Section", !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!3 = !{i32 4, !"Objective-C Garbage Collection", i32 1280}
!4 = !{i32 1, !"Objective-C Class Properties", i32 64}
!10 = !{i32 1, !"PIC Level", i32 2}
!11 = !{i32 1, !"Swift Version", i32 5}

It compiles with emcc but writes messages like
warning: unresolved symbol: _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5
So, problem with linking. This could be caused by swiftcc or because I removed

!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
@kripken

This comment has been minimized.

Owner

kripken commented Feb 10, 2018

Ok, yeah, that's a runtime linking issue then (the !5 = ! etc. things at the end are metadata, and can be ignored for now). Swift has a runtime that implements necessary things, like I guess _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5 allocates an uninitialized array, and there is an "Objective C bridge" of some sort.

Is there an open source version of the swift runtime library? If so then the next step is to build that (to bitcode, and link that to the main program). That runtime might need to be ported to wasm as well, but if it's already portable enough to run on linux and macos then it might just work, or might only take a little work.

@helje5

This comment has been minimized.

helje5 commented Feb 11, 2018

there is an "Objective C bridge" of some sort.

It may be better to not use the macOS compiler but do this on Linux. On macOS Swift has Objective-C integration, which is not there on Linux.

Is there an open source version of the swift runtime library?

At least the Linux stuff is all open source and available here: https://github.com/apple/swift

@goloveychuk

This comment has been minimized.

goloveychuk commented Feb 11, 2018

ok, let's forget for now about linking, calling conventions, std/runtime libs.
Here is simple swift code without using stdlib and cleaned llvm ir.
https://gist.github.com/goloveychuk/76f842045f28a19c939b210937dce2d4
It fails both with upstream llvm and built-in llvm

Built-in:

Not handled: i64 sub (i64 ptrtoint ([13 x i8]* @2 to i64), i64 ptrtoint (<{ i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 }>* @_T04code5StrctVMn to i64))
LLVM ERROR: FlattenGlobals: ConstantExpr opcode not handled: sub

Which is ok, since emscripten's mostyle targeting c/cpp code.

With upstream llvm:

DEBUG:root:emscript: binaryen s2wasm: /Users/badim/github/wasm/emsdk-portable/clang/e1.37.33_64bit/binaryen/bin/s2wasm /tmp/emscripten_temp/tmp0FUxXQ.wb.s --emscripten-glue --global-base=1024 --initial-memory=16777216 -l /Users/badim/.emscripten_cache/wasm/wasm_libc_rt.a -l /Users/badim/.emscripten_cache/wasm/wasm_compiler_rt.a --trap-mode=allow
[[object_alias:]]:
==========
.int32	2
	.int32	3
	.int32	(.L__unnamed_2-_T04code5StrctVMn)
==========

Included .s in gist

@kripken

This comment has been minimized.

Owner

kripken commented Feb 11, 2018

Looks like that's a bug in s2wasm. It's actually being deprecated, though, so it's probably best to look at the new path, which is to use lld. Try building with EMCC_EXPERIMENTAL_USE_LLD=1 in the environment.

@goloveychuk

This comment has been minimized.

goloveychuk commented Feb 12, 2018

Great! It compiled with several warnings

warning: unresolved symbol: _T0SiN
warning: unresolved symbol: _swift_slowAlloc
warning: unresolved symbol: _swift_slowDealloc

caused by linking problem.
It running and fails in runtime bacause can't find those functions. Next step - fix linking with stdlib/runtime.

@patcheng

This comment has been minimized.

patcheng commented Feb 13, 2018

Great progress!

I have been doing trying to get everything to compile and linking.

Here is what I have so far (without success)
https://github.com/patcheng/swift/tree/swift-to-wasm

But it requires
https://github.com/patcheng/clang/tree/swift-to-wasm
and
https://github.com/patcheng/llvm/tree/swift-to-wasm

I had to build my own compile-rt, libcxx, libcxxabi and icu4c.

Here are the things I found (and attempted to address in my fork):

  • changed swift buildscripts to generate webassembly runtimes
  • change swift to have os(WebAssembly) and arch(wasm32) and arch(wasm64)
  • changed the runtime using those definitions.
  • clang and llvm doens't support swiftcall
  • llvm was not output wasm in a byte-aligned way that clang_ast expects.
  • llvm was not not generation atomic operations
  • llvm didn't have an asm parser yet.
  • llvm was failing to handle some "expression" that swift generates when compiling

Things are "compiling" and "linking".
Known issues:

  • The generated wasm file does not work. Firefox is compiling that the memory is not shared. It's probably missing in llvm/lld.
  • the generated wasm file is failing wasm-validation. The atomic operations are bad.
    probably a lot more things are broken.
@therealbnut

This comment has been minimized.

therealbnut commented Mar 10, 2018

@patcheng nice work - sounds promising. It seems that swift-clang & swift-llvm are now up to date with release_60 which includes WASM changes. You should be able to have your branches forked from those instead.

I've rebased your changes on those here if you want to use it:

Note, I did have to resolve some conflicts, so you may want to verify. Also I didn't have what I needed to resolve conflicts in combine-sdiv.ll so you may need to resolve that yourself (I suspect it's mostly generated code).

It'd be great to get some of these changes merged onto https://github.com/apple 😄

@benaubin

This comment has been minimized.

benaubin commented May 12, 2018

What is the status on this? Is there any way in which I can help? If so, where should I start?

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Sep 20, 2018

LLVM 7.0 was released yesterday with a lot of progress for WebAssembly target support. While the backend is still marked as experimental, not as much patching of upstream LLVM repositories should be needed. I've pulled 7.0 updates into Apple swift's fork repositories and rebased the changes in the main swift repository by @patcheng, while also fixing a few compilation issues.

Now the whole fresh toolchain with patches from above comments compiles for me, although I haven't checked yet if the actual WebAssembly support has improved. Going to look into that soon. I'm doing the work in these forks and branches:

https://github.com/maxdesiatov/swift
https://github.com/maxdesiatov/swift-compiler-rt
https://github.com/maxdesiatov/swift-llvm
https://github.com/maxdesiatov/swift-clang
https://github.com/maxdesiatov/lld

@paulshapiro

This comment has been minimized.

Contributor

paulshapiro commented Oct 27, 2018

Hey @MaxDesiatov, would love to hear of any updates.

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Oct 28, 2018

Hey @paulshapiro, I appreciate your interest! I've got most if not all of Swift standard library compiling, as well as patches from @patcheng. I've also added support for Swift calling conventions when targeting WebAssembly in my clang and llvm forks. The main problem now is linking the standard library with its dependencies when compiled to WebAssembly. The build process gets a bit complicated though, as Swift stdlib also depends on libc (musl fork supporting WebAssembly in this case), libc++ and libc++abi and ICU all compiled and linked together for WebAssembly target from scratch. It also looks like all of this can only be linked with lld.

Unfortunately, a build system used within Swift compiler infrastructure is not very modular, it uses a mix of shell scripts, Python scripts and CMake. In addition to that, it doesn't look like cross-compiling the whole toolchain from scratch is supported well.

I'm currently getting changes to the build system polished so that wider audience could just clone my fork and run a reproducible build. This doesn't mean that Swift standard library will be compiled, linked and working for WebAssembly end-to-end at that point, but at least the build system issues would become reproducible and allow more people to contribute their fixes to these problems.

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Oct 28, 2018

TWIMC, even when we're able to compile a few simple Swift executables to WebAssembly, there still a ton of infrastructure work than needs to be done for all this to work smoothly: Swift Package Manager support, helpers for managing multiple cross-compilation toolchains, IDE support, better bridging to JavaScript etc.

In my opinion, one of the most pressing issues is that it's not easy to produce a self-contained statically linked binary written in Swift be it a WebAssembly binary or an ELF binary for Linux. This is going to be a quite serious problem until we have dynamic linking working well in WebAssembly, which is probably not going to happen soon.

If you'd like to be able to easily distribute a self-contained WebAssembly builds of your Swift code, please support work being done on this (in the increasing order of complexity):

  1. Upvote Jira issues and PRs related to this:
  • SR-648 - SPM support for statically linked binaries
  • SR-3819 - cross-platform ICU bug that prevents static linking
  • SR-237 - build system improvement that splits the giant shell script build-script-impl that controls the build process of Swift compiler infrastructure
  • apple/swift#19860 - fix for SR-3819
  1. Reproduce these issues locally and provide relevant info in corresponding PRs and bug reports.
  2. Highlight importance of these issues on Swift Forums and wider in the community. These issues seem obscure and irrelevant if you're only targeting iOS and other Apple's platforms exclusively, but it's important to solve them for better support of other platforms.
  3. Provide fixes/PRs for these issues.

Thanks!

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Oct 30, 2018

I think I finally got the build script modified so that musl, icu4c, libcxx and libcxxabi are pulled and built automatically with a few more cross-compilation flags set. Unfortunately, there are still build errors caused by missing directories, which need to be investigated.

At least, no more manual steps or additional custom scripts are required now. You can reproduce this WebAssembly build (and errors until those are fixed) with this short script:

mkdir swift-source && cd swift-source && \
  git clone https://github.com/maxdesiatov/swift.git && \
  ./swift/utils/update-checkout --clone --scheme wasm-next \
    --target-platform WebAssembly && \
  time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
    --webassembly --libicu

This will pull the correct forks with correct branch scheme configured.

To rerun the build after you've done any changes you can do this:

time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
  --webassembly --libicu

This will also print the total build time, not the check out time though, but check out is supposed to be done only once anyway.

Be prepared for long build times for cold builds: on 2018 15" MBP with top CPU and RAM config it takes around 40 minutes until failing. I anticipate the complete successful build to take much more than this as it needs to assemble both the host macOS and the cross-compiled WebAssembly toolchains.

Thank you all for following. I'll post updates here when I get any significant fixes pushed to my fork. PRs welcome!

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Oct 30, 2018

For future reference, this is the error message during Swift stdlib compilation that's currently blocking me:

FAILED: stdlib/public/core/webassembly/wasm32/Swift.o 
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1039/1331] Generating /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule
FAILED: lib/swift/webassembly/wasm32/Swift.swiftmodule lib/swift/webassembly/wasm32/Swift.swiftdoc lib/swift/webassembly/wasm32/Swift.swiftinterface 
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftdoc && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftinterface && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -emit-module -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule -emit-parseable-module-interface -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1049/1331] Compiling /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/macosx/x86_64/Swift.o
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
@pannous

This comment has been minimized.

pannous commented Oct 30, 2018

error: no such file or directory: 'wasm32-unknown-unknown-wasm'

my ignorant suggestion:
-target=wasm32-unknown-unknown-wasm
or
-target:wasm32-unknown-unknown-wasm
instead of
-target wasm32-unknown-unknown-wasm
?

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Oct 30, 2018

It doesn't look like the argument is passed incorrectly, otherwise I'd expect line-directive or its proxied swiftc invocation to explicitly complain about the incorrect argument.

My current guess is that something's missing in build-script-impl or one of the numerous CMake files and their cross-compilation settings. Those seem to fail to create this directory within the main build directory tree. I plan to have a closer look at this line-directive script and possibly to debug it and swiftc to see where exactly the attempt to address that directory occurs and what's the absolute path of it. Then we'd need to retrace the build scripts to understand what's the best setting to tweak.

@patcheng

This comment has been minimized.

patcheng commented Nov 1, 2018

nice progress!
according to swiftc -h:

  -sdk <sdk>              Compile against <sdk>

so, looks like <sdk> is missing.

looking at SwiftConfigureSDK.cmake, configure_sdk_windows defines:

    set(SWIFT_SDK_${prefix}_ARCH_${arch}_PATH "/")

but we are not setting the variable in configure_sdk_webassembly.

@MaxDesiatov

This comment has been minimized.

MaxDesiatov commented Nov 1, 2018

Thank you @patcheng, that's a great shout!

I've fixed that missing directory problem and a few more header import errors following after that in my fork. Here's the error I'm currently stuck at:

[1049/1331] Compiling /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o
FAILED: stdlib/public/core/webassembly/wasm32/Swift.o 
cd /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt -- /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk / -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt
<unknown>:0: error: file '/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm' is not a valid precompiled module file
/Users/maxd/Documents/swift-source/swift/stdlib/public/core/ArrayBody.swift:18:8: error: no such module 'SwiftShims'
import SwiftShims

Looking at this precompiled module file we can see that it's compiled to WebAssembly:

% file module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm
module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm: WebAssembly (wasm) binary module version 0x1 (MVP)

Not sure if that's any good, a bit more googling leads to this clang doc on precompiled modules. There's also a mention of an unmerged patch for WebAssembly precompiled modules here and an unresolved clang bugreport probably related to this problem, both posted by @patcheng by the way 👍

Probably need to try and apply that patch or figure another way to make clang consume these precompiled WebAssembly modules. Or maybe these modules shouldn't be compiled to WebAssembly altogether? 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment