New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compiling to JavaScript with Emscripten #9430

Closed
tshort opened this Issue Dec 21, 2014 · 36 comments

Comments

Projects
None yet
@tshort
Member

tshort commented Dec 21, 2014

With Julia's improved static compilation, compilation to JavaScript is getting closer. This feature would allow Julia code to be run from static web pages. Eventually, it may be possible to compile all of Julia, REPL and all.

@tshort tshort added the feature label Dec 21, 2014

@tshort

This comment has been minimized.

Member

tshort commented Dec 21, 2014

I've tried and given up on this a couple of times, so I'm describing my progress to date in case anyone has advice or wants to try it. It's easy to generate a large LLVM bitcode file containing the julia/base library. Run the following from within the julia/base directory:

../julia --build XX --dump-bitcode=yes --compile=all -J ../usr/lib/julia/sys0.ji -f sysimg.jl

Now, you should have an XX.bc file in the base directory. To identify functions in the bc file, use the following:

llvm-nm XX.bc | grep _abs_

That'll give something like:

T jlcall_abs_24823
T jlcall_abs_57792
T jlcall_abs_57794
T jlcall_abs_57795
T jlcall_abs_57796
T julia_abs_24823
T julia_abs_42171
T julia_abs_42172
T julia_abs_42173
T julia_abs_42174
T julia_abs_42175
T julia_abs_57792
T julia_abs_57793
T julia_abs_57794
T julia_abs_57795
T julia_abs_57796
T julia_abs_57797
T julia_abs_57798
T julia_abs_57799
T julia_abs_57800
T julia_abs_57801
T julia_abs_57802
T julia_abs_57803
T julia_abs_57804
T julia_abs_57805
T julia_abs_57806
T julia_abs_57807

To compile one of these functions to JavaScript, use emcc:

emcc -v XX.bc -o XX.js -s EXPORTED_FUNCTIONS="['_julia_abs_42171]"

A few of the simplest functions will compile to working JavaScript code. Several others will error as described below.

Here are current issues:

  • 32-bit bitcode -- The most common error from the compilation step above is "i64 thing we can't legalize yet". Emscripten expects 32-bit bitcode. On a 64-bit platform, it can be difficult to generate 32-bit binaries. I tried modifying the contrib/build_sysimg.jl script to try to get 32-bit results by compiling with the i686 target. I wasn unsuccessful with that.

  • Struct issues -- Sometimes, we get the following error: Here's a bug report on this:

    https://code.google.com/p/nativeclient/issues/detail?id=3932

    Someone has compiled Rust with Emscripten and may have overcome similar issues:

    https://www.reddit.com/r/rust_gamedev/comments/2n0x08/emscripten_experiments/cmcakwr

    Also, here's a thread on using Emscripten with Rust that has interesting info:

    rust-lang/rust#2235

  • Symbol names -- Julia creates several symbol names with # in them. When converted to JavaScript, it causes errors. This one's probably not too bad to fix, either in Julia or in Emscripten.

Another issue is compiling libjulia to bitcode. I've gotten about 80% or more of it to compile with emcc. I had to strip out the libuv stuff. I haven't tried building any of the third-party libraries.

@tkelman tkelman referenced this issue Dec 21, 2014

Open

All the Maths #1

@nolta

This comment has been minimized.

Member

nolta commented Dec 29, 2014

Dup of #2418.

@nolta nolta closed this Dec 29, 2014

@ihnorton

This comment has been minimized.

Member

ihnorton commented Dec 30, 2014

If anything, I would close the other one because it is fairly speculative whereas this issue is substantive (if only partially actionable in base). Emscripten keeps coming up, so we might as well have a central place to point people to.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Dec 30, 2014

I'd favor that – #2418 was pretty vague.

@EricForgy

This comment has been minimized.

EricForgy commented Jan 24, 2015

This sounds awesome.

I am having some difficulty getting Emscripten installed, but once I do, I can try to play around with this.

Getting all of Julia including REPL compiled sounds ambitious, but it looks like bits and pieces already can be compiled. Would it makes sense to try to compile a small version of Julia (Juliet?) into Javascript as a start? I think it would already be super cool if we could do basic matrix algebra and maybe an svd or eig here and there and lean on JS proper for a lot of the other stuff.

For this smaller scope, we could probably even write a parser in JS if that helps.

Just so I understand, if we were able to do this, would we be able to write code like this in the JS console:

var a = rand(5,5);
var b = rand(5,5);
var c = .5*((a+b)+(a+b)');
var out = svd(c);
var u = out[0],
    v = out[1],
    w = out[2];
console.log(u);

One can dream :D

What would the usage look like?

By the way, Tom, I noticed it appears you managed to compile CLAPACK to JS. How is that? I'd love to learn how to use that.

@ViralBShah

This comment has been minimized.

Member

ViralBShah commented Jan 26, 2015

Or just translate LAPACK to Julia. However there are too many other libraries we depend on.

@EricForgy

This comment has been minimized.

EricForgy commented Jan 26, 2015

Baby steps...

Running a Julia function from the JS console in Chrome...

image

The JS code:

function _julia_abs_49197($0) {
$0 = +$0;
var $1 = 0, $2 = 0, $3 = 0.0, label = 0, sp = 0;
sp = STACKTOP;
$1 = (HEAPF32[tempDoublePtr>>2]=$0,HEAP32[tempDoublePtr>>2]|0);
$2 = $1 & 2147483647;
$3 = (HEAP32[tempDoublePtr>>2]=$2,+HEAPF32[tempDoublePtr>>2]);
STACKTOP = sp;return (+$3);
}

@ViralBShah ViralBShah removed the feature label Feb 14, 2015

@tshort

This comment has been minimized.

Member

tshort commented Feb 18, 2015

Someone is also looking to make this work with Numba (Python/NumPy that uses LLVM):

http://cyrille.rossant.net/numpy-browser-llvm/

@jskDr

This comment has been minimized.

jskDr commented Feb 19, 2015

It is interesting to know it. Everything go on Web now. However, still
Javascript is too realistic while desktop script languages become more and
more being abstract.

On Wed, Feb 18, 2015 at 4:56 PM, Tom Short notifications@github.com wrote:

Someone is also looking to make this work with Numba (Python/NumPy that
uses LLVM):

http://cyrille.rossant.net/numpy-browser-llvm/


Reply to this email directly or view it on GitHub
#9430 (comment).

Best regards,
James Sungjin Kim, Ph.D.
Samsung Advanced Institute of Technology (SAIT)
Samsung. Electronics Co., LTD

@tshort

This comment has been minimized.

Member

tshort commented Jun 19, 2015

Given the recent announcements about WebAssembly, the best path may be to wait until that is ready. Their development plan is to build an official LLVM backend. Emscripten uses an out-of-tree fork of LLVM, so version issues can be a problem. With a built-in backend for WebAssembly, compilation for the web might be an easier path.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Jun 19, 2015

WebAssembly is pretty exciting – it makes running Julia code in the browser a real prospect.

@EricForgy

This comment has been minimized.

EricForgy commented Jun 20, 2015

Very cool. Thanks for the heads up.

@ylluminate

This comment has been minimized.

ylluminate commented Jan 25, 2016

While WebAssembly is ultimately the way to go, has anyone actually put any effort into makin a Julia -> JavaScript transpiler for the time being a la Opal's methodology?

The more I look at Julia, the more desirous I become to be able to develop an isomorphic Julia stack so that it can be used for server and client side development. Volt is a fantastic example of that and is what Meteor should have been for JS.

I can imagine a Julia based isomorphic framework that would just burn down many performance and language barriers that cause debugging headaches at present.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Jan 25, 2016

It's pretty tricky largely because of big honking dependencies on LLVM, BLAS and OpenLibm: I don't think that EmScripten can transpile LLVM itself; and BLAS and OpenLibm are not pure C – they include inline assembly, which I'm pretty certain EmScripten cannot handle.

@jkroso

This comment has been minimized.

jkroso commented Jan 25, 2016

@ylluminate I spent a few hours writing a transpiler. It's easy to do but hard to make it fast. If I could figure out how to reuse parts of Julia's compilation pipeline like type inference and hopefully some function to make the AST more abstract that would make things a lot easier.

@ylluminate

This comment has been minimized.

ylluminate commented Jan 25, 2016

Has anyone explored this, which says it will "Run LLVM Assembly In Your Browser?" Just poking around without exploring yet...

@datnamer

This comment has been minimized.

datnamer commented Jan 27, 2016

@StefanKarpinski Once julia can be statically compiled those difficulties will be obviated correct?

@tkelman

This comment has been minimized.

Contributor

tkelman commented Jan 27, 2016

Not unless someone writes pure-julia replacements for all of the ccalls into every one of those libraries.

@datnamer

This comment has been minimized.

datnamer commented Jan 27, 2016

Wouldn't compiled binary emitted by a hypothetical slimmed down non-numerical Julia work with Web assembly?

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Jan 27, 2016

Not unless someone writes a transpiler from x86 assembly to WebAssembly, which is, of course, entirely possible, but would have to be done.

@datnamer

This comment has been minimized.

datnamer commented Jan 27, 2016

How about with this web assembly LLVM JIT VM? https://github.com/WebAssembly/wasm-jit-prototype

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Jan 27, 2016

OpenLibm and BLAS have inline assembly – not inline LLVM assembly.

@fasiha

This comment has been minimized.

fasiha commented Jan 27, 2016

OpenLibm and BLAS have inline assembly – not inline LLVM assembly.

I'm curious: do the functions containing assembly in these libraries lack
portable C/Fortran, non-assembly, versions?

@vtjnash

This comment has been minimized.

Member

vtjnash commented Jan 28, 2016

yes, there's a reference implementation in C. lapack is a bit harder, since it is in fortran, but it turns out there is an SO post on the topic: http://stackoverflow.com/questions/21990243/use-emscripten-with-fortran-lapack-binding

@tkelman

This comment has been minimized.

Contributor

tkelman commented Jan 28, 2016

Dragonegg is basically dead, unfortunately. End of this year the Nvidia/PGI Fortran LLVM front-end should get released.

@SalchiPapa

This comment has been minimized.

Contributor

SalchiPapa commented Jan 28, 2016

Wouldn't compiled binary emitted by a hypothetical slimmed down non-numerical Julia work with Web assembly?

@datnamer I think that is also the way we are going to, see:

Also currently @ScottPJones is working on a julia-lite version, that we are testing on RaspberryPi2, see:

Discussion at Gitter chat room:

@kripken

This comment has been minimized.

kripken commented Jan 29, 2016

It might be worth trying again to get this working with emscripten - around a year passed since the last attempt based on the history here. Fixes since then might have helped. In particular, the Rust work mentioned above has matured, including struct fixes and so forth.

There is also a current PR for another possibly relevant issue.

@ylluminate

This comment has been minimized.

ylluminate commented Feb 2, 2016

👍 @kripken

@datnamer

This comment has been minimized.

datnamer commented Mar 13, 2016

Maybe with the revamped Julia -> c transpiler, the c code can be compiled to web assembly.

@vtjnash

This comment has been minimized.

Member

vtjnash commented Mar 13, 2016

this will be cool when it is possible, and we intend to continue working towards this (going through the Julia2C transpiler is not the best path forward), but I'm going to close the issue since it's not particularly actionable as a bug report. (although discussion here and on julia-dev is still encouraged to continue)

@vtjnash vtjnash closed this Mar 13, 2016

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Mar 14, 2016

@vtjnash: The issue tracker is not just for bug reports, but also for feature requests. I'll leave this one closed since we don't really need an issue to track it, but please don't close everything that's not a bug.

@vtjnash

This comment has been minimized.

Member

vtjnash commented Mar 14, 2016

i'm not attempting to close all feature requests, only ones (like this one) that aren't specifically actionable and thus don't need an issue to track it

@datnamer

This comment has been minimized.

datnamer commented Mar 15, 2016

So what is being thought of as the most potentially successful path forward if not the c transpilation?

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Mar 15, 2016

Going through LLVM without C would be simpler.

@ylluminate

This comment has been minimized.

ylluminate commented Mar 15, 2016

Has anyone evaluated @kripken's remarks wrt emscripten?

@PallHaraldsson

This comment has been minimized.

Contributor

PallHaraldsson commented Aug 24, 2016

Is this "actionable" now (at least for @ScottPJones's Julia-lite branch, that is available), with or without Julia2C? The PR that @kripken mentioned is merged.

"Maybe with the revamped Julia -> c transpiler, the c code can be compiled to web assembly."

[or asm.js], anyway is Julia2C broken (or only for 0.3?), would if be much work to get it to work if it is, maybe not for this.

[Was this not actionable, only because of assembly/BLAS/LAPACK: "yes, there's a reference implementation in C. lapack is a bit harder, since it is in fortran"? I would be ok without BLAS/LAPACK. There are many functions besides matrix multiply, but how far would e.g. pure-Julia native [naive] matrix multiply get you (ans say pow)?]

[Also more discussion at https://github.com//issues/2418#issuecomment-168565188 (that was closed in favor of this one), since, I'm just not sure what "Biwa" is.]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment