Add `dhall-to-nixpkgs` #1826

Gabriella439 · 2020-05-31T04:17:00Z

This adds a new dhall-nixpkgs project which builds a dhall-nixpkgs
executable. This executable is the Dhall analog of cabal2nix, which
takes a directory or GitHub project as input and generates a buildable
Nix package as its output.

Example output:

$ dhall-to-nixpkgs github https://github.com/Gabriel439/dhall-semver.git

{ buildDhallGitHubPackage, Prelude }:
  buildDhallGitHubPackage {
    name = "dhall-semver";
    githubBase = "github.com";
    owner = "Gabriel439";
    repo = "dhall-semver";
    rev = "7f5b7bf0e89664571522de9919edcc8469a55320";
    fetchSubmodules = false;
    sha256 = "00vi74npwbhwn2lnjpw4zv5y9kjqysc2f1wg1zncps6k5wljnwb7";
    directory = "";
    file = "package.dhall";
    source = false;
    dependencies = [ (Prelude.override { file = "package.dhall"; }) ];
    }

$ dhall-to-nixpkgs directory ~/proj/dhall-packages/

{ buildDhallDirectoryPackage, dhall-kubernetes, Prelude, dhall-packages }:
  buildDhallDirectoryPackage {
    name = "dhall-packages";
    src = /home/gabriel/proj/dhall-packages;
    file = "package.dhall";
    source = false;
    dependencies = [
      (dhall-kubernetes.override { file = "1.15/package.dhall"; })
      (Prelude.override { file = "JSON/Type"; })
      (Prelude.override { file = "JSON/package.dhall"; })
      (dhall-kubernetes.override { file = "1.16/package.dhall"; })
      (dhall-kubernetes.override { file = "1.17/package.dhall"; })
      (dhall-kubernetes.override {
        file = "types/io.k8s.api.core.v1.ServiceSpec.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.Time.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.api.core.v1.PodTemplateSpec.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.api.core.v1.SecretKeySelector.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.MicroTime.dhall";
        })
      (dhall-kubernetes.override {
        file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta.dhall";
        })
      (Prelude.override { file = "Map/Type"; })
      (dhall-packages.override { file = "kubernetes/k8s/1.15.dhall"; })
      (Prelude.override { file = "package.dhall"; })
      ];
    }

This requires the following matching changes to Nixpkgs in order to work:

Two new buildDhall{Directory,GitHub}Package utilities have to be
added

... which are used by the generated output
The Dhall package set in Nixpkgs now needs to select default versions
for packages

In other words, pkgs.dhallPackages.Prelude should select a default
version of the Prelude (ideally the latest version)

This parallels how the Haskell package set works, by selecting a
default version for each package, the latest version possible that
produces a consistent package set.

This adds a new `dhall-nixpkgs` project which builds a `dhall-nixpkgs` executable. This executable is the Dhall analog of `cabal2nix`, which takes a directory or GitHub project as input and generates a buildable Nix package as its output. Example output: ```bash $ dhall-to-nixpkgs github https://github.com/Gabriel439/dhall-semver.git ``` ```nix { buildDhallGitHubPackage, Prelude }: buildDhallGitHubPackage { name = "dhall-semver"; githubBase = "github.com"; owner = "Gabriel439"; repo = "dhall-semver"; rev = "7f5b7bf0e89664571522de9919edcc8469a55320"; fetchSubmodules = false; sha256 = "00vi74npwbhwn2lnjpw4zv5y9kjqysc2f1wg1zncps6k5wljnwb7"; directory = ""; file = "package.dhall"; source = false; dependencies = [ (Prelude.override { file = "package.dhall"; }) ]; } ``` ```bash $ dhall-to-nixpkgs directory ~/proj/dhall-packages/ ``` ```nix { buildDhallDirectoryPackage, dhall-kubernetes, Prelude, dhall-packages }: buildDhallDirectoryPackage { name = "dhall-packages"; src = /home/gabriel/proj/dhall-packages; file = "package.dhall"; source = false; dependencies = [ (dhall-kubernetes.override { file = "1.15/package.dhall"; }) (Prelude.override { file = "JSON/Type"; }) (Prelude.override { file = "JSON/package.dhall"; }) (dhall-kubernetes.override { file = "1.16/package.dhall"; }) (dhall-kubernetes.override { file = "1.17/package.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.api.core.v1.ServiceSpec.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.Time.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.api.core.v1.PodTemplateSpec.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.api.core.v1.SecretKeySelector.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.MicroTime.dhall"; }) (dhall-kubernetes.override { file = "types/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta.dhall"; }) (Prelude.override { file = "Map/Type"; }) (dhall-packages.override { file = "kubernetes/k8s/1.15.dhall"; }) (Prelude.override { file = "package.dhall"; }) ]; } ``` This requires the following matching changes to Nixpkgs in order to work: * Two new `buildDhall{Directory,GitHub}Package` utilities have to be added ... which are used by the generated output * The Dhall package set in Nixpkgs now needs to select default versions for packages In other words, `pkgs.dhallPackages.Prelude` should select a default version of the Prelude (ideally the latest version) This parallels how the Haskell package set works, by selecting a default version for each package, the latest version possible that produces a consistent package set.

Gabriella439 · 2020-05-31T04:20:56Z

Here's the matching change to Nixpkgs: NixOS/nixpkgs#89237

sjakobi

So is the main purpose of this tool to make it easy to integrate Dhall packages into Nix setups? Could you maybe say a bit about the background / context?

I've started reviewing the code a bit, but I haven't made it all the way through yet. I'm sure this is the most complex code I've seen in this repository so far! A good learning opportunity I guess! ;)

Of course, I'm also still way too unfamiliar with Nix…

Feel free to merge if you want to get this out quickly. Or keep it open a bit longer if you want a more in-depth review…

dhall-nixpkgs/Main.hs

sjakobi · 2020-05-31T07:59:04Z

dhall-nixpkgs/Main.hs

+toListWith _  Nothing  = [ ]
+
+nub :: Ord a => [a] -> [a]
+nub = Foldl.fold Foldl.nub


This is also available from containers BTW: http://hackage.haskell.org/package/containers-0.6.2.1/docs/Data-Containers-ListUtils.html#v:nubOrd

But apparently only from v0.6.0.1 / GHC 8.6 on…

sjakobi · 2020-05-31T08:15:20Z

dhall-nixpkgs/Main.hs

+    This function finds all remote imports that are transitive dependencies of
+    the given expression, failing if any of them are missing integrity checks.
+-}
+findExternalDependencies :: Expr Src Import -> StateT Status Shell URL


What's the purpose of the Shell monad here? I don't quite understand how it's being used here.

@sjakobi: The Turtle.Shell module explains a bit how it works:

http://hackage.haskell.org/package/turtle-1.5.19/docs/Turtle-Shell.html

... as well as the turtle tutorial:

http://hackage.haskell.org/package/turtle-1.5.19/docs/Turtle-Tutorial.html#g:9

dhall-nixpkgs/Main.hs

german1608

I don't know to much about Nix so I only reviewed code structure and LGTM!

I see that you used a lot of NeatInterpolations. Why don't we use them in other places of our project? Is it more efficient than concatenating Texts with (<>)?

I think that error messages construction in other modules could benefit from using NeatInterpolation since I see them way more readable, but that could go in another PR 😃

It's no longer necessary

@sjakobi

... as suggested by @sjakobi Co-authored-by: Simon Jakobi <simon.jakobi@gmail.com>

@sjakobi

... as suggested by @sjakobi

@sjakobi

... as requested by @sjakobi

@sjakobi

... to clarify some questions that @sjakobi raised Note that really the long-term place where this documentation will go is in a new "Dhall" subsection of the Nixpkgs manual. This module header is just a temporary stopgap until the Nixpkgs manual entry can be fully fleshed out.

…all-haskell into gabriel/dhall-to-nixpkgs_2

Gabriella439 · 2020-06-02T03:49:32Z

At this point I'm mainly waiting on NixOS/nixpkgs#89237 to be approved, too, before merging this

sjakobi

Thanks for addressing all my comments, @Gabriel439! :)

I still haven't properly grokked this though. I'll have to learn more about Nix first, I guess! ;)

Playing with it should also help…

@Gabriel439 Maybe also add this to the packages list in the main README?

dhall-nixpkgs/Main.hs

@sjakobi

... as caught by @sjakobi

Anton-Latukha · 2020-06-05T15:39:10Z

You can review this code later. HNix is 0.8.0, and it now uses GHC 8.8 by default: https://matrix.hackage.haskell.org/#/package/hnix and all its dependencies are up to the day, and we would look to keep it so and progress further.

The motivation for this change is to enable a new Dhall command-line utility called `dhall-to-nixpkgs` which converts Dhall packages to buildable Nix packages. You can think of `dhall-to-nixpkgs` as the Dhall analog of `cabal2nix`. You can find the matching pull request for `dhall-to-nixpkgs` here: dhall-lang/dhall-haskell#1826 The two main changes required to support `dhall-to-nixpkgs` are: * Two new `buildDhall{Directory,GitHub}Package` utilities are added `dhall-to-nixpkgs` uses these in the generated output * `pkgs.dhallPackages` now selects a default version for each package using the `prefer` utility All other versions are still buildable via a `passthru` attribute

Gabriella439 mentioned this pull request May 31, 2020

Nixpkgs support for dhall-to-nixpkgs NixOS/nixpkgs#89237

Merged

2 tasks

sjakobi approved these changes May 31, 2020

View reviewed changes

german1608 approved these changes May 31, 2020

View reviewed changes

Gabriella439 and others added 7 commits May 31, 2020 20:12

Remove -Wno-name-shadowing

5352ab3

It's no longer necessary

s/loop/loop with GHC 8.4/

81a409c

... as suggested by @sjakobi Co-authored-by: Simon Jakobi <simon.jakobi@gmail.com>

Document dependencyToNix result

81bff5b

Don't use RecordWildCards for destructing records

7ed946e

... as suggested by @sjakobi

Factor out error messages into a top-level function

1cbcc94

... as requested by @sjakobi

Merge branch 'gabriel/dhall-to-nixpkgs_2' of github.com:dhall-lang/dh…

0944eeb

…all-haskell into gabriel/dhall-to-nixpkgs_2

sjakobi reviewed Jun 2, 2020

View reviewed changes

dhall-nixpkgs/Main.hs Outdated Show resolved Hide resolved

dhall-nixpkgs/Main.hs Show resolved Hide resolved

german1608 mentioned this pull request Jun 3, 2020

feat(dhocs): add dhocs tool skeleton #1833

Merged

Gabriella439 added 3 commits June 3, 2020 19:31

Fix missing newline at end of error messages

bd0748e

Add missing newline at end of pretty-printed Nix expressions

0e6f2d2

... as caught by @sjakobi

Merge branch 'master' into gabriel/dhall-to-nixpkgs_2

e0b2149

Fix bug in stripping trailing slash

b45f1ae

Gabriella439 added 2 commits June 18, 2020 21:33

Merge branch 'master' into gabriel/dhall-to-nixpkgs_2

cb481d6

Relax upper bounds for dhall-nixpkgs dependencies

3316401

Gabriella439 merged commit 50df284 into master Jun 19, 2020

Gabriella439 deleted the gabriel/dhall-to-nixpkgs_2 branch June 19, 2020 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `dhall-to-nixpkgs` #1826

Add `dhall-to-nixpkgs` #1826

Gabriella439 commented May 31, 2020

Gabriella439 commented May 31, 2020

sjakobi left a comment

sjakobi May 31, 2020

sjakobi May 31, 2020

Gabriella439 Jun 1, 2020

german1608 left a comment

Gabriella439 commented Jun 2, 2020

sjakobi left a comment

Anton-Latukha commented Jun 5, 2020 •

edited

Add dhall-to-nixpkgs #1826

Add dhall-to-nixpkgs #1826

Conversation

Gabriella439 commented May 31, 2020

Gabriella439 commented May 31, 2020

sjakobi left a comment

Choose a reason for hiding this comment

sjakobi May 31, 2020

Choose a reason for hiding this comment

sjakobi May 31, 2020

Choose a reason for hiding this comment

Gabriella439 Jun 1, 2020

Choose a reason for hiding this comment

german1608 left a comment

Choose a reason for hiding this comment

Gabriella439 commented Jun 2, 2020

sjakobi left a comment

Choose a reason for hiding this comment

Anton-Latukha commented Jun 5, 2020 • edited

Add `dhall-to-nixpkgs` #1826

Add `dhall-to-nixpkgs` #1826

Anton-Latukha commented Jun 5, 2020 •

edited