dhallPackages.dhallDirectoryToNix: add function to turn a directory of Dhall files into Nix

[cdepillabout]

Motivation for this change

This is a continuation of #142825 and adds functionality for easily turning a directory of Dhall files into Nix (as long as all remote imports have integrity checks).

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• 21.11 Release Notes (or backporting 21.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[cdepillabout]

Note that this PR based on #142825 and must be rebased on master after #142825 is merged in.

This PR requires support for the --fixed-output-derivations flag to dhall-to-nixpkgs, which was added in dhall-lang/dhall-haskell#2318 and hasn't been released to Hackage yet.

[SuperSandro2000]

Suggested change

	generateDhallDirectoryPackage =
	callPackage ../development/interpreters/dhall/generate-dhall-directory-package.nix { };
	generateDhallDirectoryPackage = callPackage ../development/interpreters/dhall/generate-dhall-directory-package.nix { };

[SuperSandro2000]

Suggested change

	dhall-to-nixpkgs directory --fixed-output-derivations --file "${file}" "${src}" ${if document then "--document" else ""} > $out
	dhall-to-nixpkgs directory --fixed-output-derivations --file "${file}" "${src}" ${lib.optionalString document "--document"} > $out

[cdepillabout]

I don't necessarily feel strongly about either way, but lib.optionalString document "--document" is actually more characters than if document then "--document" else "".

[SuperSandro2000]

Suggested change

	'';
	'';

[cdepillabout]

This PR now contains all the necessary code I'd like to add. It should be able to be reviewed.

However, there are two points to watch out for:

• This PR requires a new release of dhall-nixpkgs before the new functions added in this PR work. (Although this PR wouldn't break any of the existing functions/derivations in Nixpkgs if it was merged before a release of dhall-nixpkgs.)

  Specifically, dhallPackageToNix, dhallDirectoryToNix, and generateDhallDirectoryPackage won't work until dhall-nixpkgs gets the --fixed-output-derivations flag.

• The dhallPackageToNix and dhallDirectoryToNix functions use IFD, so we can't write a test for them that can run on Hydra.

In order to work around these two points, I created a .nix file that you can copy to your local Nixpkgs checkout and use to test this PR.

Save this in your local Nixpkgs checkout as test.nix:

let
  overlay = final: prev: {
    # dhall-nixpkgs needs to be overridden because the other attributes
    # in this PR require the `dhall-nixpkgs directory --fixed-output-derivation`
    # flag.
    dhall-nixpkgs = final.haskell.lib.justStaticExecutables (
      final.haskell.lib.overrideCabal final.haskellPackages.dhall-nixpkgs (oldAttrs: {
        src = final.stdenv.mkDerivation {
          name = "dhall-nixpkgs-src";
          src = final.fetchurl {
            # This is the latest commit from
            # https://github.com/dhall-lang/dhall-haskell/pull/2326 as of
            # 2021-11-08.
            url = "https://github.com/dhall-lang/dhall-haskell/archive/3d56794eef338962d623b55521a427cc58de88bb.tar.gz";
            sha256 = "sha256-0E6+bQV6hShHL3JmtjvSo4rGmpUFm00Jmr66csJ4bGA=";
          };
          # This derivation is a hacky way to get only the `dhall-nixpkgs/`
          # directory from the above archive.  I imagine there is probably
          # a better way to do this...
          installPhase = ''
            cp -r ./dhall-nixpkgs $out/
          '';
        };
        # some new libs have been added in this PR
        editedCabalFile = null;
        executableHaskellDepends = oldAttrs.executableHaskellDepends ++ [ final.haskellPackages.memory ];
      })
    );

    # This is just a repo of Dhall files that are used as an example
    # in the following attributes.
    #
    # You may be interested in seeing what the `mydhallfile.dhall`
    # file from this Dhall repo evaluates to, since it is used in all
    # the following examples:
    #
    # $ nix-shell -I nixpkgs=./. -p dhall --run 'dhall <<< https://raw.githubusercontent.com/cdepillabout/example-dhall-nix/e6a675c72ecd4dd23d254a02aea8181fe875747f/mydhallfile.dhall'
    # ["BILLBILLbillbill", "JANEJANEjanejane", "TESTTESTtesttest", "TESTTESTtesttest", "TESTTESTtesttest"]
    myRawDhallRepo = final.fetchFromGitHub {
      owner = "cdepillabout";
      repo = "example-dhall-nix";
      rev = "e6a675c72ecd4dd23d254a02aea8181fe875747f";
      sha256 = "sha256-c/EZq76s/+hmLkaeJWKqgh2KrHuJRYI6kWry0E0YQ6s=";
    };

    # This tests that `dhallPackages.generateDhallDirectoryPackage` is working.
    # This is actually the same as `tests.dhall.generateDhallDirectoryPackage`,
    # but it has been redefined here to make it easier to work with, and to make
    # this file more understandable.
    #
    # Note that `dhallPackages.generateDhallDirectoryPackage` produces a Nix file
    # that calls `dhallPackages.buildDhallDirectoryPackage` on
    # `myRawDhallRepo`.
    #
    # You can verify the produced Nix file with the following commands:
    #
    # $ nix-build ./test.nix -A myGeneratedTestDhallPackage
    # $ cat ./result
    myGeneratedTestDhallPackage = final.dhallPackages.generateDhallDirectoryPackage {
      src = final.myRawDhallRepo;
      file = "mydhallfile.dhall";
    };

    # This just builds the Nix file produced from `myGeneratedTestDhallPackage`.
    #
    # Building `myGeneratedTestDhallPackage` produces a standard Nixpkgs Dhall
    # package (so it just contains a `binary.dhall` file and `.cache/`
    # directory).
    #
    # You can confirm this with the following command:
    #
    # $ nix-build ./test.nix -A myTestDhallPackage
    # $ ls ./result/
    myTestDhallPackage = final.dhallPackages.callPackage final.myGeneratedTestDhallPackage {};

    # This tests the `dhallPackageToNix` function added in this PR.
    # `dhallPackageToNix` converts a Nixpkgs Dhall package into Nix
    # code.
    #
    # You can confirm this with the following command (using nix-2.4):
    #
    # $ nix eval -f ./test.nix myTestNixDhallPackage
    # [ "BILLBILLbillbill" "JANEJANEjanejane" "TESTTESTtesttest" "TESTTESTtesttest" "TESTTESTtesttest" ]
    myTestNixDhallPackage = final.dhallPackageToNix final.myTestDhallPackage;

    # This tests the `dhallDirectoryToNix` that was added in this PR.
    # `dhallDirectoryToNix` is similar to `dhallPackageToNix`, but takes
    # a directory of Dhall files as input, instead of a fully-built
    # Nixpkgs Dhall package.
    #
    # `dhallDirectoryToNix` is effectively a small wrapper around
    # `dhallPackages.generateDhallDirectoryPackage` and `dhallPackageToNix.
    #
    # You can confirm this works with the following command (using nix-2.4):
    #
    # $ nix eval -f ./test.nix myTestNixDhallDirectoryPackage
    # [ "BILLBILLbillbill" "JANEJANEjanejane" "TESTTESTtesttest" "TESTTESTtesttest" "TESTTESTtesttest" ]
    myTestNixDhallDirectoryPackage = final.dhallDirectoryToNix {
      src = final.myRawDhallRepo;
      file = "mydhallfile.dhall";
    };
  };

in

import ./. { overlays = [ overlay ]; }

Test the generateDhallDirectoryPackage function that has been added in this PR:

$ nix-build ./test.nix -A tests.dhall.generateDhallDirectoryPackage
$ cat ./result

Test that building this Nix Dhall Package actually works:

$ nix-build ./test.nix -A myTestDhallPackage
$ ls ./result/

Test that the dhallPackageToNix function (added in this PR) can successfully turn the above Nix Dhall Package into Nix code:

$ nix eval -f ./test.nix myTestNixDhallPackage
[ "BILLBILLbillbill" "JANEJANEjanejane" "TESTTESTtesttest" "TESTTESTtesttest" "TESTTESTtesttest" ]

Test that the dhallDirectoryToNix function (added in this PR) is able to perform all the above steps at once:

$ nix eval -f ./test.nix myTestNixDhallDirectoryPackage
[ "BILLBILLbillbill" "JANEJANEjanejane" "TESTTESTtesttest" "TESTTESTtesttest" "TESTTESTtesttest" ]

See the comments in the above test.nix for a more in-depth explanation of each of these attributes.

[cdepillabout]

@Gabriel439 This PR should be ready for review, but it uses dhall-to-nixpkgs directory --fixed-output-derivations, so we may not want to merge this PR until there is a release of dhall-to-nixpkgs that includes the --fixed-output-derivations flag.

[cdepillabout]

Gabriella recently released dhall-nixpkgs-1.0.7 to Hackage which includes the --fixed-output-derivations flag needed by dhallDirectoryToNix.

That version of dhall-nixpkgs should be available on master when #148359 has been merged in.

[sternenseemann]

Why not merge into haskell-updates? Should eliminate the dependency-between-PRs pitfall…

[cdepillabout]

@sternenseemann I'm not in a huge rush to get this in, and I'd definitely like a review by Gabriella before merging it in.

I could retarget this on haskell-updates, but my guess is that I'd probably just end up retargeting it on master before actually merging it in.

[cdepillabout]

@ofborg build dhall-nixpkgs

I've rebased this on master. This should be ready to merge in. I'll merge as soon as ofborg finishes.

[cdepillabout]

I wrote a blog post about this dhallDirectoryToNix function: https://functor.tokyo/blog/2021-12-10-dhallDirectoryToNix

Thought you might be interested @Gabriel439.

[Gabriella439]

Thanks for writing that up! I scheduled a tweet tomorrow to mention it