Add erlang modules #11651

gleber · 2015-12-12T13:16:11Z

Hello Nix community

This is rebase of #11614

I am working on getting Erlang packages into Nix and here's my first stab at the problem. The approach is similar to what is done for Haskell. Here are main points:

use Hex.pm as a source of tarballs (that's the only reasonable package respository for Erlang AFAIK)
use rebar3 (beta-quality version of rebar Erlang built tool which works with Hex.pm)
aim at automatically importing all of newest packages on Hex (including their transitive closures) use not-yet-developed set of tools

This establishes basic build functions and provides 5 Erlang packages and 1 rebar3 plugin. Implemented features:

buildHex which automates definition of Erlang packages to be fetched from Hex.pm CDN
support building packages which depend on other Erlang packages
support adding rebar3 plugins for compilation of packages
support building of packages with ports (using rebar3-pc plugin)

Rebar3 depends on a concept of registry, which is usually fetched from S3. I've created a separate github repo in which I plan to snapshot that single file from S3 (to be done automatically using

Rebar3 is imported separately from rebar2 since they are very different. @the-kenny

Further work:

automate creation of pkgs/development/erlang-modules/hex/*.nix files based on rebar.config
fetch all packages from hex.pm and create .nix files for them
figure out how to deal with conflicting dependencies and diamond dependency graph
figure out how if to provide multiple versions of the same package
implement support for Elixir
implement support for nix-shell using ERL_LIBS
instigate discussion with Erlang community about some sort of LTS package set
and much more along the way...

I'd love to hear some initial comments on the code to get guidance. This is my first Nixpkgs contribution.

Thanks!

@sjmackenzie @peti @7c6f434c @pSub @copumpkin

mention-bot · 2015-12-12T13:16:14Z

By analyzing the blame information on this pull request, we identified @vcunat, @civodul and @ip1981 to be potential reviewers

jagajaga · 2015-12-12T14:22:12Z

cc @manpages

peti · 2015-12-12T14:54:23Z

None of the Erlang packages have a meta attribute, i.e. there's no homepage URL, no short description, and -- most importantly -- no meta.platforms. I've inherited that latter attribute from Erlang in 7fb8050e402f699e068842c65697f08669e54669 to make these packages compile on Hydra: http://hydra.cryp.to/jobset/nixpkgs/erlang-updates.

peti · 2015-12-12T14:55:52Z

According to http://hydra.cryp.to/jobset/nixpkgs/erlang-updates, rebar3-3.0.0-beta.4 won't compile. Consequently, none of the packages compile either.

gleber · 2015-12-12T15:51:48Z

@peti i, thank you for taking a look. I've tested it with the following command and it worked for me:

$ nix-build ~/code/erl/nixpkgs/ -v -A erlangPackages 2>&1 | ix
http://ix.io/mLs

How can i test it on Hydra? Do I need to set up my own Hydra for that?

peti · 2015-12-12T15:55:01Z

Are you building with chroot isolation enabled? If your build succeeds in a chroot environment, then it's almost certainly going to succeed on Hydra as well.

sjmackenzie · 2015-12-13T04:11:43Z

I've tested this with nix.useChroot = true; and the build succeeds. (@peti)

sjmackenzie · 2015-12-13T04:52:49Z

@gleber, typically you'll need to setup hydra yourself, which can be a bit of a headache. We've had a noble yet tentative step up by @aszlig to be Hydra's release manager, so Hydra will be incorporated into NixOS' module infrastructure (via configuration.nix). Which will automate Hydra greatly. Though you probably don't need Hydra at this stage.

gleber · 2015-12-13T08:04:31Z

Looks like rebar3 bootstrap script is not hermetic as I thought:

===> Fetching rebar3_hex ({pkg,<<"rebar3_hex">>,<<"1.11.0">>})
===> Successfully downloaded https://s3.amazonaws.com/s3.hex.pm/tarballs/rebar3_hex-1.11.0.tar
===> Downloaded package, caching at ./.cache/rebar3/hex/default/packages/rebar3_hex-1.11.0.tar

I will work on making sure it doesn't fetch anything.

There are some packages on Hex which have custom hex-specific names, but inside there's a base project name. Remove most packages andadd ibrowse, meck, jiffy

If compilePorts is enabled plugin "pc" is added to the rebar.config during postPatch. This makes rebar3 compile download the plugin, hence added TODO.

Allow specifying pluginDeps for buildHex listing rebar3 plugins which are required for given package to be built by rebar3.

This allows to avoid fetching registry file from S3 at build time, making the build hermetic and much faster. Automatic tools will be used to update said external repo content when Hex packages are imported/update.

Adds "meta" for meck package as a test.

This will allow to reuse this code to fetch rebar3's dependencies to ensure it's hermetic build.

This makes rebar3 build a bit more hermetic. Next stage is to pull in rebar3 plugins dependencies for bootstrap to pick up.

derivation.

gleber · 2015-12-15T14:09:29Z

@peti @sjmackenzie @RumataEstor I've made rebar3 compilation hermetic and cleaned up branch history. Please take a look again.

gleber · 2015-12-15T14:10:12Z

And the build now works on my NixOS done like this:

nix-build ~/code/nixpkgs/ -vv -A erlangPackages --option build-use-chroot true

peti · 2015-12-15T16:53:07Z

Okay, all builds succeed now: http://hydra.cryp.to/eval/746730.

peti · 2015-12-15T16:55:16Z

I think it's fine to merge the current state into master so that everyone who's interested in the subject can experiment with it.

Add erlang modules

peti · 2015-12-15T16:56:27Z

@gleber, at some point, it might be a good idea to post some kind of announcement to the nix-dev mailing list to ensure that people are aware of the work you've been doing. Also, user documentation in doc/* would be very welcome at some point.

sjmackenzie · 2015-12-15T17:07:26Z

Great step forward, kudos @gleber!

copumpkin · 2015-12-15T17:13:03Z

🍻

gleber · 2015-12-15T19:37:18Z

Thanks, Peter. I've created a tracking issue which will remind me about docs: #11748

I will subscribe to nix-dev and let people know about this effort.

domenkozar · 2015-12-16T17:36:19Z

@gleber awesome work.

peti added 0.kind: enhancement 6.topic: non-nixos 8.has: package (new) 9.needs: clean-up labels Dec 12, 2015

gleber force-pushed the add-erlang-modules branch from 4f1eb61 to 37474f9 Compare December 15, 2015 00:01

gleber added 13 commits December 15, 2015 01:02

Add rebar3 package

bcc4ca0

Add few erlang modules

a05e2db

Automagically download files from hex.pm tarball S3 repo

6dcbe05

Add support for Hex package name

2251a94

There are some packages on Hex which have custom hex-specific names, but inside there's a base project name. Remove most packages andadd ibrowse, meck, jiffy

Add "compilePorts" enabling rebar3 "pc" plugin

406b974

If compilePorts is enabled plugin "pc" is added to the rebar.config during postPatch. This makes rebar3 compile download the plugin, hence added TODO.

Simplify patch and move pc compile call into buildPhase

eac25ea

Add support for rebar3 plugins to buildHex.

3a9110b

Allow specifying pluginDeps for buildHex listing rebar3 plugins which are required for given package to be built by rebar3.

Fetch hex.pm registry snapshot with external github repo.

5d3680d

This allows to avoid fetching registry file from S3 at build time, making the build hermetic and much faster. Automatic tools will be used to update said external repo content when Hex packages are imported/update.

Replace deprecated uniqList with unique function

13a1ab4

Add rebar3-pc to plugin deps automatically if compilePorts is enabled

7e57b20

Move registrySnapshot into it's own file for easier automation.

2a7e77c

Allow specifying meta for buildHex and buildErlang.

acaf2c1

Adds "meta" for meck package as a test.

Refactor fetchHex out of buildHex.

033902d

This will allow to reuse this code to fetch rebar3's dependencies to ensure it's hermetic build.

gleber force-pushed the add-erlang-modules branch from 37474f9 to ccf5e81 Compare December 15, 2015 13:01

gleber added 2 commits December 15, 2015 14:05

Move registry to rebar3, since it needs it itself to be bootstrapped.

4ba1a51

This makes rebar3 build a bit more hermetic. Next stage is to pull in rebar3 plugins dependencies for bootstrap to pick up.

Add patch which makes rebar compilation hermetic

108f3cf

gleber force-pushed the add-erlang-modules branch from ccf5e81 to 6e3df42 Compare December 15, 2015 13:06

gleber mentioned this pull request Dec 15, 2015

Allow bootstrap to pick up existing Hex cache and deps erlang/rebar3#982

Merged

gleber added 2 commits December 15, 2015 15:06

Add sources of specific rebar bootstrap deps via new fetchHex

474da3b

derivation.

Update hex.pm registry

bb1b5c7

gleber force-pushed the add-erlang-modules branch from 6e3df42 to bb1b5c7 Compare December 15, 2015 14:07

peti added a commit that referenced this pull request Dec 15, 2015

Merge pull request #11651 from gleber/add-erlang-modules

82c673d

Add erlang modules

peti merged commit 82c673d into NixOS:master Dec 15, 2015

gleber mentioned this pull request Dec 15, 2015

BEAM packages support umbrella tracking issue (Erlang, Elixir, LFE, etc.) #11748

Open

gleber deleted the add-erlang-modules branch December 15, 2015 19:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add erlang modules #11651

Add erlang modules #11651

gleber commented Dec 12, 2015

mention-bot commented Dec 12, 2015

jagajaga commented Dec 12, 2015

peti commented Dec 12, 2015

peti commented Dec 12, 2015

gleber commented Dec 12, 2015

peti commented Dec 12, 2015

sjmackenzie commented Dec 13, 2015

sjmackenzie commented Dec 13, 2015

gleber commented Dec 13, 2015

gleber commented Dec 15, 2015

gleber commented Dec 15, 2015

peti commented Dec 15, 2015

peti commented Dec 15, 2015

peti commented Dec 15, 2015

sjmackenzie commented Dec 15, 2015 via email

copumpkin commented Dec 15, 2015

gleber commented Dec 15, 2015

domenkozar commented Dec 16, 2015

Add erlang modules #11651

Add erlang modules #11651

Conversation

gleber commented Dec 12, 2015

mention-bot commented Dec 12, 2015

jagajaga commented Dec 12, 2015

peti commented Dec 12, 2015

peti commented Dec 12, 2015

gleber commented Dec 12, 2015

peti commented Dec 12, 2015

sjmackenzie commented Dec 13, 2015

sjmackenzie commented Dec 13, 2015

gleber commented Dec 13, 2015

gleber commented Dec 15, 2015

gleber commented Dec 15, 2015

peti commented Dec 15, 2015

peti commented Dec 15, 2015

peti commented Dec 15, 2015

sjmackenzie commented Dec 15, 2015 via email

copumpkin commented Dec 15, 2015

gleber commented Dec 15, 2015

domenkozar commented Dec 16, 2015