interaction with go runtime

[glycerine]

Thanks for this cool project!

@mewmew I'm wondering if llir/llvm could be used to generate code in a Go program that interacts with the Go (gc) runtime... allocating objects that are gc-ed; starting goroutines that are scheduled; making blocking system calls that the runtime handles with blocking... the kinds of things that would make llir/llvm viable for implementing an interpreter in Go.

What kind of interaction (if any) are you thinking could be supported? Or is that not on the radar at all?

[mewmew]

I'm wondering if llir/llvm could be used to generate code in a Go program that interacts with the Go (gc) runtime... allocating objects that are gc-ed; starting goroutines that are scheduled; making blocking system calls that the runtime handles with blocking... the kinds of things that would make llir/llvm viable for implementing an interpreter in Go.

Hej @glycerine! Happy to hear from you.

Implementing an interpreter in Go is definitely an intended use case of the llir/llvm project in the future; see issue go-interpreter/proposal#1 for some discussion on this topic.

The intention is also to make llir/llvm general enough to support compilers for Go to be written, so that the llgo compiler could eventually use llir/llvm as a backend instead of the LLVM C++ bindings for Go; see issue #3 for further background.

Personally, my friends and I are intending to use the llir/llvm project as the basis of a decompiler pipeline; which will translate LLVM IR assembly to Go source code; tracked by issue decomp/decomp#167.

As you have experience with implementing interpreters in Go (in particular glycerine/zygomys), I'd be very interested to learn from you what requirements may be needed as we continue to develop llir/llvm. Feel free to continue the discussion in #3 or here if you so wish : )

So, as of currently, the intended use cases of the llir/llvm project are compilers, decompilers and interpreters. Hopefully as time goes by, members of the Go community will come up with ever further use cases that we may not yet have thought of : )

Cheers /u

[mewmew]

@glycerine Just noticed you were already part of the discussion in go-interpreter/proposal#1 :)

[mewmew]

Oh, and to keep the list complete, @sangisos and I also developed a compiler for a subset of C which outputs LLVM IR; located at https://github.com/mewmew/uc

There are the projects which are currently known to use llir/llvm.

[mewmew]

@glycerine As the Go team has started to develop gollvm, a Clang for Go if you will, it may soon be feasible to connect the last pieces and implement an interpreter for Go, running on top of LLVM IR.

I'd still be very curious to hear back from you regarding the requirements of an interpreter (see #18 (comment)). If you wish, please add any such requirements either here or to issue #3, the issue dedicated to requirements of the llir/llvm project.

Within the next few days, v0.2 of llir/llvm will be released, at which point the project will be go-getable, have a fully functional parser, and have support for all instructions of LLVM IR. For a brief example of its current capabilities, see the usage section of the readme

Hope to hear back from you :)

Cheers /u

[glycerine]

Hey @mewmew! Thanks for the update. That sounds like great progress, and its also great to hear that gollvm is getting some love.

At a high level, it seems to me like there are three issues to address to make an (JIT-based) interpreter possible. Fortunately both Swift (from Apple) and Julia (from MIT) have produced languages that offer interpreter prompts but are still JIT-compiled by LLVM; Swift also provides a full AOT-compilation option. So these projects have already solved many of the problems, and could practically be used as templates/hints/examples on which to base an architecture.

The main three integration points that I see that would need to be addressed:

a) garbage collection; one would like jit-ted code from the interpreter to be able to create objects that get passed to aot-compiled code and still get GC-ed as usual.

b) scheduling with the go scheduler; code generated by a JIT/interpreter should be scheduled by the same scheduler, so that channels between aot-compiled and jit-generated code work.

lastly,

c) code update and a transaction-like system for replacing code atomically during runtime. Julia the language finally figured out (after many years) how to do this by threading a state token through, and since they are also llvm based I'm hoping that a similar trick can be used again. The principle thing that a jit-compilation based interpreter would need to address here is how much needs to be re-compiled when a function changes. See JuliaLang/julia#265. Swift has always had this, as far as I know. The basic problem is how to change code out in a partially running system, and how to update (recompile) functions that call changed code. The easy-to-start-with-but-slower solution is to have a more late-bound dispatch system that doesn't hardcode or inline any function calls. How Swift solved this would be a very interesting place to begin thinking about this.

Very exciting to see the possibility starting to coalesce!

[mewmew]

@glycerine Thanks for the feedback and for documenting the primary requirements of an interpreter for Go running on top of LLVM IR. To me, it is very clear that you have more in-depth experience with this than anyone on the team currently working on llir/llvm. Care to join? :) We could really use the expertise in these areas. I expect that you have little time to spare, as it seems most people do in this day and age, so of course you would be free to contribute in any way you may desire. Either through code or discussions :) That you've helped spark these discussions is already great!

Cheerful regards /u & i

[glycerine]

@mewmew Hi Robin, thank you; that's flattering. Unfortunately I won't have time to contribute on this.

[mewmew]

@mewmew Hi Robin, thank you; that's flattering. Unfortunately I won't have time to contribute on this.

Hi Jason. Thank you for your reply. Understandably so, and as mentioned it seems everyone is busy in this day and age. I'm still very glad that you decided to document these requirements, so that those interested in getting an interpreter running for Go may choose to take up the torch and carry this forward. As you've already mentioned, I agree that its very exciting to see the possibility starting to coalesce! :)

Wish you a most lovely summer.

Cheerful regards from Sweden,
Robin

[Nv7-GitHub]

There's also tinygo which creates LLVM from Go code, and is compatible with most of Go.