Cross-compilation

[bitwalker]

Hi! I'm really glad to see this project!

I'm curious if there is a documented path for cross-compiling NIFs using rustler when used with releases. More specifically, if I have a cross-compiled ERTS from some other architecture, I can build a release for that architecture, but NIFs must also be cross-compiled for that to work. I would like to build NIFs using Rust, and I know it has facilities for cross-compilation, but reading through the documentation it wasn't clear how I would specify that in my configuration. To be clear, I'm very new to Rust, and I don't have immediate plans to write any NIFs, but as I'm the maintainer of distillery, one of the things I'd like to do this year is make building cross-compiled releases easier to set up.

If this is something that is already provided by the toolchain, perhaps some notes on how to configure it would be a good idea to add in the docs?

[mrluc]

I've been taking the path of doing all of my building through a Dockerfile.build, so I can just build the whole release (elixir and rust) under the target archi. It's slower to build, but only when creating an image that will be pushed to the registry.

And it's been almost unavoidable for me if the project vendors interesting non-NIF dependencies anyway (for instance, if in addition to an Elixir codebase you're also vendoring ffprobe, or some client binary that needs to be built in Go, and the resulting artifact needs to run on Alpine).

For others interested in building elixir+rust releases with builder containers, the Dockerfile I used as a base for the elixir+rust builder is ekidd/rust-musl-builder combined with msaraiva's, because at least a 4-6 months ago building Rust on Alpine was a bit of a bear to get working right (so if that's changed, maybe ignore this).

[bitwalker]

Yeah that's more or less what I recommend when people ask about it, but was hoping it might be easier in the case of Rust, not that building in Docker is hard, it's just extra tooling it would be nice to not need.

[OvermindDL1]

For note, some of us are forced to run Elixir on painWindows, so docker-like solutions do not work well. ^.^

[scrogson]

@bitwalker one thing that we've been thinking about is allowing NIF libraries to ship precompiled binaries in the hex package. I suppose some other options would be to fetch a precompiled binary from a GitHub release based on the target architecture.

We could also support something in the rustler compiler that would specify the rust target. That would work well for most targets (except for Windows).

Just some thoughts.

[bitwalker]

@scrogson The problem with shipping pre-compiled binaries is that you have to have a binary compiled for each architecture the package might be used on, which isn't practical when you have projects like Nerves targeting potentially a very large set of different architectures - basically, you have to ship the source anyway. That said, one could generate binaries for the most common architectures and save a lot of pain for the majority of users.

Your suggestion about specifying the target is more or less what I was getting at - it would be nice if something like that could be unified across all NIFs, since my understanding is that target triples are more or less universal, but even if it was just rustler specific it would make things easier. Is there a reason why it won't work for Windows though? I was under the impression Rust worked on Windows, or is it just that there is no cross-compilation target for Windows yet?

[hansihe]

@bitwalker Being able to prebuild binaries for the most common architectures was more or less what I was thinking. The author should be able to choose what architectures they want to build and easily bundle them along. If any of the users of the NIF are running on something where there isn't a prebuilt binary, it should build from source.

As for Nerves, they use buildroot, but buildroot doesn't support Rust yet. There was some work on supporting it a while ago, but I think the person doing it gave up. As far as I know, there is no easy way for us to work with Nerves without buildroot support.

[bitwalker]

Makes sense :). I like the idea of putting control in the library author's hands to generate binaries for a preset list of architectures, but I think it would also be key to provide a way for consumers of NIF libraries to specify their desired target architecture. I think in a perfect world, both mechansims would be used by NIFs written not just in Rust, but other languages as well, that way there is a standard way of both generating such libraries, as well as generating cross-compiled releases. If there is anything I can do to help, let me know :)

I'll let you decide if you want to leave this open for further discussion, or close it, but I think my question is answered for the immediate future.

[hansihe]

@bitwalker Could you give us a quick overview of how distillery handles cross-compiled releases? I think that would be helpful to know when looking at solutions.

I will leave the issue open, this is something we should think about more.

[bitwalker]

Sure, today, it basically works like this (assuming NIFs are part of the equation):

• Grab a copy of ERTS from a machine running on the desired target architecture, this is the version of ERTS you will target, and as an example from my machine, would be found in ~/.asdf/installs/erlang/19.2/lib/erlang/erts-8.2, but can be located on any host with Path.wildcard("#{:code.root_dir()}/erts-*")
• Set include_erts: "path/to/target/erts" in rel/config.exs
• Ensure the compiler toolchain for your NIFs are configured for cross-compiling as necessary
• Run MIX_ENV=prod mix release --env=some_target where some_target is an environment definition in rel/config.exs with the include_erts setting configured as described above. I would guess other environment variables are also required for the NIFs, but I omit those here since it varies depending on what your NIFs are written in.

This will ensure that the correct ERTS is bundled with the release, and that your NIFs are also cross-compiled for the target. Getting a copy of ERTS for the target is easy, and in the future it may be something that could be automated by Distillery (or another tool which hooks into Distillery). Currently, the big missing piece for most people is "how do I cross-compile a release when I have NIFs in my dependency tree?". Right now, I tell people that they either have to setup the toolchain for the NIFs to do cross-compilation, which is pretty impractical, as many devs have no familiarity with systems languages, or the specific requirements for the NIFs in question (i.e. dependencies on other system libraries/packages); OR, to build their release using Vagrant/Docker/VirtualBox/etc. and copy the resulting tarball out for deployment. This is not ideal, but is the least painful choice.

What I'd like to see is a way to specify a target triple to build for, and have a plugin for Distillery which automatically fetches the toolchain/binaries needed for that architecture (if one can be discovered), and set up the environment appropriately to do the cross-compilation. Obviously this can't work for all situations, but I think there is a good chance it would benefit the majority, and would likely encourage NIF authors to ensure their library can work within this framework for ease of use. I may be missing some key details here though, but that's at least the general idea I have in mind.

[MikaAK]

Hi there, I'm struggling with this currently, ideally I would love to cross compile but I'm unsure of what "setting up the toolchain for cross compilation" actually means. I know Rust can be cross compiled with cargo, but I'm unsure of how that process links in with distillery when using rustler. Would love it if anyone has more info on hooking up the toolchain or what the future plans for this issue are!

[bitwalker]

My understanding is that you just need to make sure that when the rust compiler is called when building your project, that it gets passed the appropriate cross-compilation flags to generate the .so file for the target platform. The responsibility for this is on rustler to give you the appropriate configuration hooks to set these flags based on environment, and on you to wire up your build to generate the release using that environment. Distillery provides environments separate from MIX_ENV specifically for this type of use case. Ideally rustler would provide a way, perhaps via environment variables, to tell it which target you are building for, do you could invoke the build like MIX_ENV=prod TARGET=foo mix release --env=foo, and have it produce a cross compiled release including your nif. That said, this only solves the rust part of the equation, I expect you'd have to do extra work to cross compile C NIFs at the same time. I would like to provide a tutorial on this at some point, but I'd have to sit down and work out the details, since I'm not familiar with the ins and outs of cross compiling C NIFs.