Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildRebar3 and buildMix depend less on hex registry #25655

Merged
merged 1 commit into from Feb 25, 2019
Merged

Conversation

@elitak
Copy link
Contributor

elitak commented May 9, 2017

Motivation for this change

I was trying to build one of my own projects using packages newer than those in the hex registry and found it was impossible because both Mix and rebar3 were trying to pull in the updated registry after failing to find the requested versions.

Mix has a new environment flag that fixes the behavior, so that change was easy.

Rebar3, on the other hand, still insists on updating the registry even when the built dependencies are present in ./_build/default/lib. The best solution I could come up with short of fixing this in upstream (erlang/rebar3#958) was to change the rebar3 bootstrap to have it generate all of what was symlinked said directory to ./_checkouts instead, sans src subdirs, since rebar3 will try to recompile the source (always, because the derivations lack .rebar3/erlcinfo metadata) into a RO store directory.

My changes to the rebar3 bootstrapping could use review and probably more testing, but as it stands, ./pkgs/development/beam-modules/hex-packages.nix is plenty broken. Does anyone know where is the script that generated this package listing? I need to make a fix there so that packages that depend on broken, elided packages are also elided. It could use a refresh, too.

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@mention-bot

This comment has been minimized.

Copy link

mention-bot commented May 9, 2017

@elitak, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ericbmerritt and @yurrriq to be potential reviewers.

@yurrriq

This comment has been minimized.

Copy link
Member

yurrriq commented May 9, 2017

@elitak

This comment has been minimized.

Copy link
Contributor Author

elitak commented May 9, 2017

@yurrriq I don't see the script that emits hex-packages.nix, if that's what you meant to link...

@ericbmerritt

This comment has been minimized.

Copy link
Contributor

ericbmerritt commented May 9, 2017

The script that updates the registry is update.sh. The code that actually creates hex-packages.nix is here https://github.com/erlang-nix/hex2nix.

@elitak

This comment has been minimized.

Copy link
Contributor Author

elitak commented May 9, 2017

Ok thanks.

@elitak

This comment has been minimized.

Copy link
Contributor Author

elitak commented May 10, 2017

Hex2nix isnt building now because I somehow broke rebar3 escriptize with these changes. This is turning into a real mess; I'm getting pretty aggravated by rebar3 not being able to reliably use dependencies I'm laying at its feet. The lack of documentation, code clarity, command-line options, and debug logging in rebar3 is wearing my patience thin.

I might just abandon this and port all my projects to Mix, unless someone who's familiar with the rebar3+nix-bootstrap can offer a better solution than my moving _build/default/lib/* to _checkouts/*, because even if I get that to work, it feels like way too many hacks to accommodate what should be a more robust build tool.

@ericbmerritt

This comment has been minimized.

Copy link
Contributor

ericbmerritt commented May 10, 2017

@elitak rebar3 doesn't want to give up dependency management in any way shape or form. I have had conversations with the maintainers and they are not interested in fixing that problem. We had to do quite a number of hacky things to get it working. You are running into all that stuff. Mix was much simpler.

@elitak elitak force-pushed the elitak:beam branch May 10, 2017
@elitak

This comment has been minimized.

Copy link
Contributor Author

elitak commented May 10, 2017

Alright, I'm just backing out the rebar3 changes for now, leaving the innocuous buildMix change.

@elitak elitak force-pushed the elitak:beam branch to a6e35f2 Aug 18, 2017
@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Apr 21, 2018

@GrahamcOfBorg build hexPackages.airbrakify

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Apr 21, 2018

@GrahamcOfBorg build beam.packages.erlang.airbrakify

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

Failure on x86_64-darwin (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)


Unsafe variable found at:
  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
patching script interpreter paths in /nix/store/jv4lb2i4rrpsair2swr227xcf7xh3dcx-httpoison-0.8.3
cannot build derivation '/nix/store/r4vs9i6m0f905wwma8xhxvn5cx5f1mmg-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/r4vs9i6m0f905wwma8xhxvn5cx5f1mmg-airbrakify-0.0.1.drv' failed

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

Failure on x86_64-linux (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)

  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3
patching script interpreter paths in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3
checking for references to /build in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3...
cannot build derivation '/nix/store/cw6l4r6qhrbfgwwsz14h0ask513njmlm-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/cw6l4r6qhrbfgwwsz14h0ask513njmlm-airbrakify-0.0.1.drv' failed

Copy link
Member

matthewbauer left a comment

Looks good. Anyone have objections?

@GrahamcOfBorg

This comment has been minimized.

Copy link

GrahamcOfBorg commented Apr 21, 2018

Failure on aarch64-linux (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)

  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3
patching script interpreter paths in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3
checking for references to /build in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3...
cannot build derivation '/nix/store/pzbx3zjzb9pm0k8fjh96zayz6cbla7g4-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/pzbx3zjzb9pm0k8fjh96zayz6cbla7g4-airbrakify-0.0.1.drv' failed

@veprbl

This comment has been minimized.

Copy link
Member

veprbl commented Jan 12, 2019

@ryantm

This comment has been minimized.

Copy link
Member

ryantm commented Feb 23, 2019

@GrahamcOfBorg build beam.packages.erlang.airbrakify

@yurrriq

This comment has been minimized.

Copy link
Member

yurrriq commented Feb 25, 2019

Anyone have objections?

None from me.

@ryantm ryantm merged commit fcc2834 into NixOS:master Feb 25, 2019
13 of 14 checks passed
13 of 14 checks passed
continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
beam.packages.erlang.airbrakify on aarch64-linux Failure
Details
beam.packages.erlang.airbrakify on x86_64-darwin Failure
Details
beam.packages.erlang.airbrakify on x86_64-linux Failure
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@yorickvP

This comment has been minimized.

Copy link
Contributor

yorickvP commented Jul 2, 2019

Removing --no-deps-check here broke a lot of builds, since it now runs the deps check. Is running the deps check the intended effect?

cc: @yegortimoshenko

@yorickvP

This comment has been minimized.

Copy link
Contributor

yorickvP commented Jul 2, 2019

Example of a build that doesn't work anymore:

{
  jason = buildMix rec {
    name = "jason";
    version = "1.1.2";

    src = fetchzip {
      url = "https://github.com/michalmuskala/jason/archive/v${version}.tar.gz";
      sha256 = "0fh87vrfqsyiaazsangsg992i1azad8cmzyzvg7fdm9z6b3v7lm0";
    };
  };
}
@yegortimoshenko

This comment has been minimized.

Copy link
Member

yegortimoshenko commented Jul 2, 2019

Can confirm, --no-deps-check should stay.

@yegortimoshenko

This comment has been minimized.

Copy link
Member

yegortimoshenko commented Jul 2, 2019

@elitak rebar3 doesn't want to give up dependency management in any way shape or form. I have had conversations with the maintainers and they are not interested in fixing that problem. We had to do quite a number of hacky things to get it working. You are running into all that stuff. Mix was much simpler.

You can use fake-hex-registry, it was created specifically for that purpose. (I'm going to try to upstream mix-to-nix to Nixpkgs once it doesn't require eval-time builds and fetches).

@yorickvP yorickvP mentioned this pull request Jul 16, 2019
3 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.