Add Julia support

[mattgodbolt]

Consider adding Julia support, see: https://www.quora.com/How-long-does-it-take-to-develop-low-latency-trading-programming-skills-using-C%2B%2B/answer/Matt-Godbolt/comment/32014611?__snid3__=839274997&__nsrc__=4

[vchuravy]

Is there some documentations on how to add a new language to Compiler Explorer?

The standard way in Julia to get the LLVM IR or assembly of a function is:

using InteractiveUtils

function sum(X)
    acc = zero(eltype(X))
    @simd for x in X
       acc += x
    end
    return acc
end

code_llvm(stdout, sum, Tuple{Array{Float32, 1}})
# code_llvm(stdout, sum, Tuple{Array{Float32, 1}), optimize=false)
# code_native(stdout, sum, Tuple{Array{Float32, 1}})

# or @code_llvm f(zeros(30))

There are also more intermediate representations code_typed/code_lowered

How would one go about adding line mapping/coloring? Julia adds
debuginfo to the output:

julia> code_native(stdout, sum, Tuple{Array{Float32, 1}}, debuginfo=:source)
	.text
; ┌ @ REPL[3]:3 within `sum'
; │┌ @ simdloop.jl:71 within `macro expansion'
; ││┌ @ simdloop.jl:51 within `simd_inner_length'
; │││┌ @ REPL[3]:2 within `length'
	movq	8(%rdi), %rax

Which includes the inlining stacks and the LLVM output has also has the option to get the entire module with LLVM debuginfo:
code_llvm(stdout, sum, Tuple{Array{Float32, 1}}, raw=true, dump_module=true, debuginfo=:none)

Anyway I would be happy to see Julia in Compiler Explorer, since right now I end up generating LLVM IR and then putting it into Compiler Explorer.

[mattgodbolt]

Is there some documentations on how to add a new language to Compiler Explorer?
Yes indeed: https://github.com/mattgodbolt/compiler-explorer/blob/master/docs/AddingALanguage.md

To make it easier to integrate we'd need a single program to run to generate the output; else we need a full implementation of a "compiler" for julia.

Line mapping is handled in the asm.js parsers; either we'd need to find a way to make julia output standard debug information (DWARF) or else implement a new parser for it there.

[vchuravy]

To make it easier to integrate we'd need a single program to run to generate the output; else we need a full implementation of a "compiler" for julia.

Hm that is tricky since Julia compiles functions (not programs) and you need to provide it the name of the function and the types.
I guess I could write a Julia program that takes the name of the function and the types as a command-line argument and what you want to emit (LLVM,ASM,IR).

Line mapping is handled in the asm.js parsers; either we'd need to find a way to make julia output standard debug information (DWARF) or else implement a new parser for it there.

I think we can get around the non-standard output as well, I have implemented most of the infrastructure in https://github.com/vchuravy/MCAnalyzer.jl

[staticfloat]

I took a shot at this, my work is being stashed on this branch.

First off, Julia code often cannot be meaningfully compiled without the top-level function knowing what types it is being passed as arguments; I'm getting around this right now by simply wrapping all user code within a function main() and invoking that function, but this means that constant-prop eliminates a lot of computation. As an example, writing the
following:

function square(x)
	return x * x
end
square(2)

Results in just (minus boilerplate):

ret i64 4

This forces the user to resort to things like x = parse(Int, "4") to actually see what code is generated in the presence of non-consts. This is solvable, of course, especially if we have some kind of formal way of telling CE what types to pass in for (non-constant) parameters; that would be ideal. We could inspect the Julia code to see if there is a single top-level function defined with a single, fully-concretely-typed method, and if there is, show the result of compiling that. Example:

function square(x::Int64)
    return x * x
end

Our wrapper code would inspect, find that a definition for square() exists and there is only one valid set of types that it can possibly take, and would compile that.

All of this presupposes the existence of a wrapper script. Right now, I am shoe-horning this wrapper code into the compiler options, but that's really suboptimal. So, a question for the CE team: How would I bundle a compiler wrapper alongside CE so that we can (in essence) invoke julia compiler_wrapper.jl -o <output_file> <user_code> <compiler_args>?

[vchuravy]

Elliot, you might be able to add a script juliac.sh to https://github.com/mattgodbolt/compiler-explorer-image, when adding Julia in general.

[partouf]

So, a question for the CE team: How would I bundle a compiler wrapper alongside CE so that we can (in essence) invoke julia compiler_wrapper.jl -o <output_file> <user_code> <compiler_args>?

Elliot, you might be able to add a script juliac.sh to https://github.com/mattgodbolt/compiler-explorer-image, when adding Julia in general.

I think it's probably best to instead have a runCompiler() override, like for example done here https://github.com/mattgodbolt/compiler-explorer/blob/master/lib/compilers/pascal.js#L126

[staticfloat]

I think it's probably best to instead have a runCompiler() override, like for example done here /lib/compilers/pascal.js@master#L126

So right now, what I'm doing is basically shoe-horning the wrapper script into a command-line argument; but as this wrapper script will likely grow to be many dozens of lines long, it seems unwieldy to be generating a static file and passing that to Julia on the command line every time. To be clear, the purpose of this wrapper script is to interpret the options being passed from CE to Julia; because Julia is a JIT, half of our compilation options are exposed on the command-line, and half are exposed from within the language itself. For instance, targeting which function to compile will be done in code, as will things such as choosing which format to output, whether optimization is enabled, etc... So all the argument parsing, generating the compilation harness, etc... will be done in Julia code that needs to be passed to the Julia compiler.

I could write this Julia source as a string literal in the Javascript within runCompiler(), write that out to a temporary file, then pass it to the Julia compiler via julia /path/to/generated_script.jl -o /path/to/output.s /path/to/user_code.jl <compiler opts>, but it seems like it would be much simpler if there were a location for "supporting files" that gets shipped with CE, so that I can just point the julia compiler at it without having to generate the same static file every time it gets invoked.

Elliot, you might be able to add a script juliac.sh to mattgodbolt/compiler-explorer-image, when adding Julia in general.

I could, but that kind of precludes usage of Julia with the CE outside of that image; I'm not sure if that's a concern; if it's not, then that's probably what I'll do. Since we would be writing essentially an option parsing frontend that could work with multiple versions of Julia, the static script would not be tied to any particular version of Julia, it would be more tied to the options we expose in CE to interface with Julia.

[apmorton]

I suggest you take an approach similar to the recent python support - 78b3381

[staticfloat]

Perfect; exactly what I was looking for. Thank you @apmorton!

[staticfloat]

Alright, so I've added a compiler_wrapper.jl to etc/scripts in CE. It now works by assuming that the last expression within the file is the one to be compiled. This seems to provide the right balancing point between flexibility and ease of use so far.

Right out of the gates, it's allowing us to build some neat functionality; You can now pass options such as --format=<fmt> where <fmt> can be one of typed, llvm, native, and Julia will spit out the requested IR. If you are using --format=llvm, you can pass in --optimize to set whether or not it will be optimized, etc...

There are, of course, many, many different options we could build into the compiler_wrapper.jl in order to expose more and more of the internals but I think the most important things to do next are:

• Embed Julia syntax highlighting. We have a number of syntax highlighting rulesets already written up for things like vim, atom, Jupyter, etc... What's the best way to get this integrated into CE?
• Emit IR in a more easily-parsable manner for CE to be able to read it. I think the best way for this to happen is probably for us to get Julia to spit out DWARF information; this should be doable I believe.
• Add support for compiler options as dropdowns and such like we see for gcc within CE. I'm not quite sure how these are done yet, but it would be neat to have e.g. a dropdown for the various levels of lowering/optimization, e.g. ast -> typed ast -> optimized ast -> LLVM -> optimized LLVM -> native. Integrating this with DWARF information would allow a truly powerful inspection of Julia's compilation pipeline, as we would be able to see the flow of source lines to instructions as they pass through the pipelines.

[vchuravy]

Right out of the gates, it's allowing us to build some neat functionality; You can now pass options such as --format=<fmt> where <fmt> can be one of typed, llvm, native, and Julia will spit out the requested IR. If you are using --format=llvm, you can pass in --optimize to set whether or not it will be optimized, etc...

One thing to maybe add is the assembly style e.g. Intel or ATT

• Emit IR in a more easily-parsable manner for CE to be able to read it. I think the best way for this to happen is probably for us to get Julia to spit out DWARF information; this should be doable I believe.

Right now we have our own dissassembler in https://github.com/JuliaLang/julia/blob/b96a591c5846bc3ffdff3fa1f3b4659986669c32/src/disasm.cpp#L741 so that we get the consistent printing of debuginformation. Whereas https://github.com/vchuravy/MCAnalyzer.jl/blob/7f2a8309796281b81d2bcdcba7334943f9c34f7b/src/reflection.jl#L44-L51 just uses LLVM to obtain an assembly file directly.

• Embed Julia syntax highlighting. We have a number of syntax highlighting rulesets already written up for things like vim, atom, Jupyter, etc... What's the best way to get this integrated into CE?

IIUC CE is using Monaco and you have to add a highlighting scheme for that in in the step:

Add static/modes/language-mode.js and require it in static/panes/editor.js

x-ref: microsoft/monaco-editor#911 as well

[NHDaly]

@staticfloat it sounds like you have a solution here that works, it just doesn't have syntax highlighting or line mapping?

That seems like useful enough to add already, and once it's out there, presumably other ambitious users can add those things in follow-up PRs? :) What do you think? Would it make sense to open a PR now from your branch? :)

[MasonProtter]

Would JuliaLang/julia#40220 make Julia godbolt support any easier?