Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Nixpkgs support for Dhall #79900

Merged
merged 3 commits into from Feb 25, 2020
Merged

Conversation

@Gabriel439
Copy link
Contributor

@Gabriel439 Gabriel439 commented Feb 12, 2020

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as pkgs.fetchFromGitHub)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds. That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with dhallPackages, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks. If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the -f-with-http flag, which disables
the interpreter's support for HTTP imports. If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep /nix/store
utilization as compact as possible.

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as `pkgs.fetchFromGitHub`)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds.  That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with `dhallPackages`, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks.  If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the `-f-with-http` flag, which disables
the interpreter's support for HTTP imports.  If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep `/nix/store`
utilization as compact as possible.
There is no need to provide a separate `kubernetesVersion` argument
since the `file` argument works just fine
lib.mapAttrs makePrelude {
# Prelude versions older than 7.0.0 use old-style union literals, which are
# no longer supported by the latest version of the standard
"7.0.0" = {

This comment has been minimized.

@Mic92

Mic92 Feb 12, 2020
Contributor

Do we really need all those versions?

This comment has been minimized.

@Gabriel439

Gabriel439 Feb 12, 2020
Author Contributor

@Mic92: Not really. I just added as many as possible because it was easy to do. I can remove them if desired

@Profpatsch
Copy link
Member

@Profpatsch Profpatsch commented Feb 13, 2020

Will review tomorrow.

Copy link
Member

@Profpatsch Profpatsch left a comment

I haven’t done a full review, but it looks to me like this was mostly inspired by the haskellPackages code, which is a horribly clunky thing in my opinion. I’d advise to keep it more simplistic.

pkgs/development/dhall-modules/Prelude.nix Outdated Show resolved Hide resolved
callPackage ../development/dhall-modules/dhall-kubernetes.nix { };

dhall-packages =
callPackage ../development/dhall-modules/dhall-packages.nix { };

This comment has been minimized.

@Profpatsch

Profpatsch Feb 16, 2020
Member

what’s the reasoning behind this attribute? It doesn’t seem to contain anything that the others don’t expose.

This comment has been minimized.

@Gabriel439

Gabriel439 Feb 17, 2020
Author Contributor

dhall-packages does not just re-export other Dhall packages. dhall-packages provides new functionality of its own (like Argo CD bindings, for example)

This comment has been minimized.

@Profpatsch

Profpatsch Feb 17, 2020
Member

Oh, I see, dhall-packages is a (mono)repository containing other dhall packages. The naming is kind of confusing.

let
packages = self:
let
callPackage = newScope self;

This comment has been minimized.

@Profpatsch

Profpatsch Feb 16, 2020
Member

Not a fan of the complexity introduced by this scoping override (not a fan of how haskellPackages overcomplicates many things in general). Can we keep it simple? let is a great idea.

This comment has been minimized.

@Gabriel439

Gabriel439 Feb 17, 2020
Author Contributor

@Profpatsch: Actually, I was following how pythonPackages did things (which happens to be the same as haskellPackages)

I'm not exactly sure what alternative approach you are suggesting, but it sounds like you are recommending that I use pkgs.callPackage (with no scoping override). I gave that a try and ran into the following issues:

  • It's now more difficult for one Dhall package to depend upon another Dhall package

    ... since the non-overriden callPackage doesn't bring any Dhall packages in scope

  • You lose the ability to override buildDhallPackage

    ... which is really useful, for the same reason that it's useful to override pkgs.haskellPackage.mkDerivation. For example, you could globally override buildDhallPackage to set source = true; to build source distributions for Dhall packages

Or did I misunderstand what you are requesting? Perhaps if you included a short code sample of what you had in mind it would be more clear

This comment has been minimized.

@Profpatsch

Profpatsch Feb 17, 2020
Member

I’m sorry, I was in a bad mood yesterday and did this review against my better judgement. I should do better than that.

We should use the approach in this PR because of two reasons:

  1. It’s the default, with precedence set by haskellPackages and pythonPackages. There’s a lot of value in regularity, even if the usability or design is not the greatest.
  2. There hasn’t been a better approach so far.
... as requested by @Profpatsch
(Prelude."12.0.0".override { file = "Map/Type"; })
];
};
}

This comment has been minimized.

@Profpatsch

Profpatsch Feb 17, 2020
Member

So if dhall-packages (confusing name!) is a monorepository providing vendored versions of other packages, this marks a shortcoming of the proposed approach; how would I as a nix maintainer keep these definitions up to date?
Do I have to rummage through the repository to find all dependencies and manually resolve their version to patch them here? Can dhall help me with this?

This comment has been minimized.

@Gabriel439

Gabriel439 Feb 17, 2020
Author Contributor

@Profpatsch: My plan was to follow this up with a dhall-to-nixpkgs utility, which would play a role similar to other *2nix utilities like cabal2nix. You would give it a source directory or a URL as input and it would translate that to a derivation matching the style as those underneath ./pkgs/development/dhall-modules

Essentially the tool would:

  • Guess the name of the Dhall package from the directory name or URL
  • Obtain all external dependencies by scanning transitive relative imports
  • Turn those external dependencies into the dependencies field of buildDhallPackage (using the same rule for inferring package names)

For example, if it were to see an external dependency of the form:

https://raw.githubusercontent.com/${user}/${repo}/${version}/${path}

Then it would add this to the dependencies list:

  (repo."${version}".override { file = path })

... or we could include the user in the package name, too, in order to minimize conflicts.

Then the next logical step would be writing a callDhall2Nixpkgs analogous to callCabal2nix

Copy link
Member

@Profpatsch Profpatsch left a comment

Before I block this any further, let’s merge.

I’m sure you are going to come up with a satisfactory solution iteratively, like you always seem to do 👍

@Profpatsch Profpatsch merged commit 8246c35 into NixOS:master Feb 25, 2020
12 checks passed
12 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Feb 26, 2020
Add Nixpkgs support for Dhall

(cherry picked from commit 8246c35)
@Gabriel439 Gabriel439 deleted the Gabriel439:gabriel/dhall_packages_2 branch Feb 26, 2020
@ehmry
Copy link
Member

@ehmry ehmry commented Mar 17, 2020

How does this work without any setup or shell hooks? Why merge if it isn't used anywhere in the tree?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.