BEAM packages support umbrella tracking issue (Erlang, Elixir, LFE, etc.)

[gleber]

I am starting this issue to track progress of the effort started in #11651. I will be updating this issue with further developments and my implementation plans. If someone wants to jump in and work on it, please let me know (here or privately) to avoid work duplication.

Assumptions

I am working on getting Erlang packages into Nix. The approach is similar to what is done for Haskell. Here are main points:

• use Hex.pm as a source of tarballs (that's the only reasonable package respository for Erlang AFAIK)
• use rebar3 and Mix as foundation
• aim at automatically importing all of newest packages on Hex (including their transitive closures) use not-yet-developed set of tools
• packages should include both source code and bytecode files

Current status (2017-09-24)

The following basic features are implemented already:

• fetchHex automates fetching packages from Hex.pm CDN
• buildHex which automates definition of Erlang packages to be fetched and built from Hex.pm CDN
• buildMix to build Mix packages
• support building packages which depend on other Erlang packages
• support adding rebar3 plugins for compilation of packages
• support building of packages with ports (using rebar3-pc plugin)
• implemented hermetic bootstrap and build for rebar3
• two versions of rebar3 are provided:
  • rebar3 is one with applied patch which makes it hermetic (forbids any attempts to download stuff from the internet)
  • rebar3-open which is unmodified version of rebar3
• implemented support for nix-shell using ERL_LIBS (just run nix-shell -A erlangPackages.lager.env) Add shell env to buildHex packages. #11818
• [NEEDS UPDATE] hex2nix automates creation of pkgs/development/erlang-modules/hex-packages.nix file based on hex.pm registry, which includes all buildable versions of packages, including multiple versions of the packages
• doc/erlang-users-guide.xml contains documentation
• support for multiple erlang versions
• support for Elixir and LFE
• support building Elixir/LFE and all packages from hex-packages.nix with various Erlang versions.

Rebar3 depends on a concept of registry, which is usually fetched from S3. I've created a separate github repo in which I plan to snapshot that single file from S3. This is automated in https://github.com/erlang-nix/hex-pm-registry-snapshots/blob/master/update.sh

Further work

Short term

• extract out of hex2nix Nix expression creation for Erlang/Mix packages into a common library;
• adapting hex2nix to use the common library;
• reuse the library to implement rebar2nix as a rebar3_nix plugin able to create shell.nix and default.nix in two modes:
  • self-contained with all deps explicitly written in the default.nix (like in

    nixpkgs/pkgs/development/tools/build-managers/rebar3/default.nix

    Line 10 in b1140f9

    erlware_commons = fetchHex {

    ) - useful e.g. to upgrade rebar3 and mix in nixpkgs
  • reusing beamPackages as much as possible - the default mode;
• implement mix2nix (or a mix plugin) doing equivalent of rebar3_nix;
• add "shell" command to both rebar3 and mix plugins to enter a self-contained shell mode with a specific OTP version and all deps available in PATH and in ERL_LIBS
• upgrading all of the non-mass-imported Beam packages in nixpkgs using the self-contained mode;
• upgrade the mass-imported beamPackages using the default mode.

Long term

• implement running tests as part of the build
• investigate if it makes sense to provide "binary" and/or "binary and sources" or just "sources" packages (there are some packages on Hex which provide some binaries which are useful to be put into user's environment)
• investigate integration Erlang releases with Nix ecosystem
• instigate discussion with Erlang community about some sort of LTS package set

Support for nix-env

Support for nix-env has been discouraged by @peti and @domenkozar as unsustainable and giving very little benefits.

[gleber]

I've redone erlang/rebar3#982 according to owners of rebar3, waiting for feedback.

[gleber]

Tooling for this will be placed in: https://github.com/erlang-nix

[gleber]

Here are various changes to the tooling from last week:

#12429
#12444
erlang-nix/hex2nix#6
erlang-nix/rebar3-nix-bootstrap#1
erlang-nix/rebar3-nix-bootstrap#2
https://github.com/erlang-nix/nixpkgs/pull/2

[rasendubi]

(triage) @gleber any update?

[rasendubi]

Closing because of inactivity.

[domenkozar]

@rasendubi let's not close meta issues. We can't expect people just deliver code every week or month.

[rasendubi]

@domenkozar, no problem. I've closed because the issue hasn't been updated for ~9 months - seems like nobody cares about it and the work is abandoned. I would be glad to be proven wrong.

[gleber]

Updates which I have forgotten to link to this issue:

• 2016
  • Erlang: update tooling and hex packages #12473
• 2017
  • erlang: refactor to generalize Erlang/OTP derivations #26381
  • erlang: refactor: build packages per Erlang/OTP version. #26668
  • erlang: use makeExtensible #26811

Current rough plan of actions is outlined in this comment: #26279 (comment)

[ankhers]

I'm also going to add the following PR's to this list.

• 2017
  • Generalize building of Elixir interpreter #26716
  • Generalize building of LFE #27295

[gleber]

@ankhers Thanks for linking to those PRs which I've missed.

I've updated the issue main text with current status and future plans.

[ankhers]

I will start working on the common library and the mix2nix plugin.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.