Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Nixpkgs support for Dhall #79900

Open
wants to merge 3 commits into
base: master
from

Conversation

@Gabriel439
Copy link
Contributor

Gabriel439 commented Feb 12, 2020

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as pkgs.fetchFromGitHub)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds. That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with dhallPackages, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks. If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the -f-with-http flag, which disables
the interpreter's support for HTTP imports. If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep /nix/store
utilization as compact as possible.

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as `pkgs.fetchFromGitHub`)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds.  That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with `dhallPackages`, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks.  If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the `-f-with-http` flag, which disables
the interpreter's support for HTTP imports.  If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep `/nix/store`
utilization as compact as possible.
There is no need to provide a separate `kubernetesVersion` argument
since the `file` argument works just fine
lib.mapAttrs makePrelude {
# Prelude versions older than 7.0.0 use old-style union literals, which are
# no longer supported by the latest version of the standard
"7.0.0" = {

This comment has been minimized.

Copy link
@Mic92

Mic92 Feb 12, 2020

Contributor

Do we really need all those versions?

This comment has been minimized.

Copy link
@Gabriel439

Gabriel439 Feb 12, 2020

Author Contributor

@Mic92: Not really. I just added as many as possible because it was easy to do. I can remove them if desired

@Profpatsch

This comment has been minimized.

Copy link
Member

Profpatsch commented Feb 13, 2020

Will review tomorrow.

Copy link
Member

Profpatsch left a comment

I haven’t done a full review, but it looks to me like this was mostly inspired by the haskellPackages code, which is a horribly clunky thing in my opinion. I’d advise to keep it more simplistic.

pkgs/development/dhall-modules/Prelude.nix Outdated Show resolved Hide resolved
callPackage ../development/dhall-modules/dhall-kubernetes.nix { };

dhall-packages =
callPackage ../development/dhall-modules/dhall-packages.nix { };

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Feb 16, 2020

Member

what’s the reasoning behind this attribute? It doesn’t seem to contain anything that the others don’t expose.

This comment has been minimized.

Copy link
@Gabriel439

Gabriel439 Feb 17, 2020

Author Contributor

dhall-packages does not just re-export other Dhall packages. dhall-packages provides new functionality of its own (like Argo CD bindings, for example)

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Feb 17, 2020

Member

Oh, I see, dhall-packages is a (mono)repository containing other dhall packages. The naming is kind of confusing.

let
packages = self:
let
callPackage = newScope self;

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Feb 16, 2020

Member

Not a fan of the complexity introduced by this scoping override (not a fan of how haskellPackages overcomplicates many things in general). Can we keep it simple? let is a great idea.

This comment has been minimized.

Copy link
@Gabriel439

Gabriel439 Feb 17, 2020

Author Contributor

@Profpatsch: Actually, I was following how pythonPackages did things (which happens to be the same as haskellPackages)

I'm not exactly sure what alternative approach you are suggesting, but it sounds like you are recommending that I use pkgs.callPackage (with no scoping override). I gave that a try and ran into the following issues:

  • It's now more difficult for one Dhall package to depend upon another Dhall package

    ... since the non-overriden callPackage doesn't bring any Dhall packages in scope

  • You lose the ability to override buildDhallPackage

    ... which is really useful, for the same reason that it's useful to override pkgs.haskellPackage.mkDerivation. For example, you could globally override buildDhallPackage to set source = true; to build source distributions for Dhall packages

Or did I misunderstand what you are requesting? Perhaps if you included a short code sample of what you had in mind it would be more clear

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Feb 17, 2020

Member

I’m sorry, I was in a bad mood yesterday and did this review against my better judgement. I should do better than that.

We should use the approach in this PR because of two reasons:

  1. It’s the default, with precedence set by haskellPackages and pythonPackages. There’s a lot of value in regularity, even if the usability or design is not the greatest.
  2. There hasn’t been a better approach so far.
... as requested by @Profpatsch
(Prelude."12.0.0".override { file = "Map/Type"; })
];
};
}

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Feb 17, 2020

Member

So if dhall-packages (confusing name!) is a monorepository providing vendored versions of other packages, this marks a shortcoming of the proposed approach; how would I as a nix maintainer keep these definitions up to date?
Do I have to rummage through the repository to find all dependencies and manually resolve their version to patch them here? Can dhall help me with this?

This comment has been minimized.

Copy link
@Gabriel439

Gabriel439 Feb 17, 2020

Author Contributor

@Profpatsch: My plan was to follow this up with a dhall-to-nixpkgs utility, which would play a role similar to other *2nix utilities like cabal2nix. You would give it a source directory or a URL as input and it would translate that to a derivation matching the style as those underneath ./pkgs/development/dhall-modules

Essentially the tool would:

  • Guess the name of the Dhall package from the directory name or URL
  • Obtain all external dependencies by scanning transitive relative imports
  • Turn those external dependencies into the dependencies field of buildDhallPackage (using the same rule for inferring package names)

For example, if it were to see an external dependency of the form:

https://raw.githubusercontent.com/${user}/${repo}/${version}/${path}

Then it would add this to the dependencies list:

  (repo."${version}".override { file = path })

... or we could include the user in the package name, too, in order to minimize conflicts.

Then the next logical step would be writing a callDhall2Nixpkgs analogous to callCabal2nix

Copy link
Member

Profpatsch left a comment

Before I block this any further, let’s merge.

I’m sure you are going to come up with a satisfactory solution iteratively, like you always seem to do 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.