Fails to find repositories specified in `pin-depends` #1

brendanzab · 2022-01-10T00:01:25Z

Describe the bug
Opam-nix seems not to look for pin-depends.

I've been trying to experiment with yocaml. It seems not to have been published to opam yet, and people seem to be using pin-depends as a way to use it in their projects?

I searched on the opam-nix repo and it seems like pin-depends is not mentioned, so maybe it is not yet supported? I understand that it might be tricky to do this while maintaining reproducibility however? So I'm not sure what that would look like.

To Reproduce

flake.nix

{
  description = "A very basic flake";

  inputs = {
    # Utilities for writing flakes
    flake-utils.url = github:numtide/flake-utils;
    # Precisely filter files copied to the nix store
    nix-filter.url = "github:numtide/nix-filter";
    # Generate derivations from Dune and OCaml files
    opam-nix.url = github:tweag/opam-nix;
  };

  outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # Legacy packages that have not been converted to flakes
        legacyPackages = nixpkgs.legacyPackages.${system};
        # OCaml packages available on nixpkgs
        ocamlPackages = legacyPackages.ocamlPackages;

        # OCaml source files
        ocaml-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            "dune-project"
            "garden.opam.template"
            (nix-filter.lib.inDirectory "bin")
          ];
        };

        # Nix source files
        nix-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            (nix-filter.lib.matchExt "nix")
          ];
        };
      in
      {
        # Executed by `nix build .#<name>`
        packages.garden =
          (opam-nix.lib.${system}.buildDuneProject { } "garden" ocaml-src { }).garden;

        # Executed by `nix build`
        defaultPackage = self.packages.${system}.garden;

        # Executed by `nix run .#<name>`
        apps.garden = {
          type = "app";
          program = "${self.packages.${system}.garden}/bin/garden";
        };

        # Executed by `nix run`
        defaultApp = self.apps.${system}.garden;

        # Used by `nix develop`
        devShell = legacyPackages.mkShell {
          nativeBuildInputs = [
            ocamlPackages.dune_2
            ocamlPackages.ocaml
            legacyPackages.opam

            # Editor support
            # pkgs.ocamlformat # FIXME: fails to build `uunf` on my M1 mac :(
            ocamlPackages.merlin
            ocamlPackages.ocaml-lsp
          ];
          inputsFrom = [
            self.defaultPackage.${system}
          ];
        };
      });
}

dune-project

(lang dune 2.9)

(generate_opam_files true)

(package
 (name garden)
 (synopsis "Static page generator for my digital garden")
 (depends
  (preface (>= "0.1.0"))
  (yocaml :pinned)
  (yocaml_yaml :pinned)
  (yocaml_markdown :pinned)
  (yocaml_unix :pinned)))

bin/dune

(executable
 (name garden)
 (public_name garden)
 (promote (until-clean))
 (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))

bin/garden.ml

let () =
  ()

garden.opam.template

pin-depends: [
  ["yocaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_unix.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_yaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_markdown.dev" "git://github.com/xhtmlboi/yocaml.git"]
]

After running nix build I get:

nix log ...-garden-local.drv

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/l2y5prikdgks7i4a5awc7vkxdxzxi2h9-source-copy
source root is source-copy
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
@nix { "action": "setPhase", "phase": "buildPhase" }
building
File "bin/dune", line 5, characters 55-66:
5 |  (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))
                                                           ^^^^^^^^^^^
Error: Library "yocaml_unix" not found.
Hint: try:
  dune external-lib-deps --missing --no-config --root . --ignore-promoted-rules --default-target @install --always-show-command-line --promote-install-files false --promote-install-files --release --only-packages garden -p garden --profile release -j 10 @install

The generated opam file is:

/nix/store/...-source-copy/garden.opam

# This file is generated by dune, edit dune-project instead
opam-version: "2.0"
depends: [
  "dune" {>= "2.9"}
  "preface" {>= "0.1.0"}
  "yocaml" {pinned}
  "yocaml_yaml" {pinned}
  "yocaml_markdown" {pinned}
  "yocaml_unix" {pinned}
  "odoc" {with-doc}
]
build: [
  ["dune" "subst"] {dev}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "--promote-install-files=false"
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
  ["dune" "install" "-p" name "--create-install-files" name]
]
pin-depends: [
  ["yocaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_unix.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_yaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_markdown.dev" "git://github.com/xhtmlboi/yocaml.git"]
]

Expected behavior

The build should find the dependencies

Environment

OS name + version: aarch64-darwin, macOS 12.1

Version of the code:

Inputs:
├───flake-utils: github:numtide/flake-utils/74f7e4319258e287b0f9cb95426c9853b282730b
├───nix-filter: github:numtide/nix-filter/bd03eb5242e707bc270f7a7e4f835b55713247ac
├───nixpkgs: github:NixOS/nixpkgs/568e0bc498ee51fdd88e1e94089de05f2fdbd18b
└───opam-nix: github:tweag/opam-nix/03ff01c4d228814e03566ef36667599848408738
    ├───flake-compat: github:edolstra/flake-compat/12c64ca55c1014cdc1b16ed5a804aa8576601ff2
    ├───flake-utils: github:numtide/flake-utils/74f7e4319258e287b0f9cb95426c9853b282730b
    ├───nixpkgs: github:nixos/nixpkgs/eac07edbd20ed4908b98790ba299250b5527ecdf
    ├───opam-repository: github:ocaml/opam-repository/c7b45b3cbcfbe22a7b3b8af1e5bb9998f3aa0e8a
    └───opam2json: github:tweag/opam2json/7652f94ac18ec94bc1ae697169c7f7f8b809270a
        └───nixpkgs: github:NixOS/nixpkgs/5c37ad87222cfc1ec36d6cd1364514a9efc2f7f2

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

balsoft · 2022-01-10T06:28:20Z

It's indeed not possible to support pin-depends reproducibly without an opam lockfile (since it doesn't have any notion of actually pinning the dependency version in the package description). It can probably be supported with --impure in some way.

For now, you can try passing yocaml as a repo. First, add an input to your flake:

  yocaml = {
    url = "github:xhtmlboi/yocaml";
    flake = false;
  };

Then, pass it (together with the default opam-repository) to the buildDuneProject:

      packages.garden = (with opam-nix.lib.${system}; buildDuneProject {
        repos = [
          (makeOpamRepo yocaml)
          opam-nix.inputs.opam-repository
        ];
      } "garden" ocaml-src { }).garden;

balsoft · 2022-01-10T08:21:24Z

opam-nix now supports pin-depends, but in your particular case it won't work because yocaml has a non-master default branch. See NixOS/nix#5139 and NixOS/nix#4456

brendanzab · 2022-01-10T08:24:26Z

Oh nice, thanks!

Hmm for some reason the suggestion above doesn't seem to be working for me?

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/l2y5prikdgks7i4a5awc7vkxdxzxi2h9-source-copy
source root is source-copy
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
@nix { "action": "setPhase", "phase": "buildPhase" }
building
File "bin/dune", line 5, characters 55-66:
5 |  (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))
                                                           ^^^^^^^^^^^
Error: Library "yocaml_unix" not found.
Hint: try:
  dune external-lib-deps --missing --no-config --root . --ignore-promoted-rules --default-target @install --always-show-command-line --promote-install-files false --promote-install-files --release --only-packages garden -p garden --profile release -j 10 @install

balsoft · 2022-01-10T08:31:52Z

Ah, yeah, sorry. I forgot to makeOpamRepo and also opam doesn't resolve pinned packages with opam admin list, so we have to ask for those ourselves:

        packages.garden = (with opam-nix.lib.${system};
          buildDuneProject {
            pinDepends = false;
            repos = [ (makeOpamRepo yocaml) opam-nix.inputs.opam-repository ];
          } "garden" ocaml-src {
            yocaml = null;
            yocaml_unix = null;
            yocaml_yaml = null;
            yocaml_markdown = null;
          }).garden;

(Note that you'll also need the latest version of opam-nix since there was an overlay issue)

brendanzab · 2022-01-10T10:31:49Z

Wooo, I think that worked! Thanks a bunch. Here's the full flake I made if it's of any use to anyone:

{
  description = "Static page generator for my digital garden";

  inputs = {
    # Utilities for writing flakes
    flake-utils.url = github:numtide/flake-utils;
    # Precisely filter files copied to the nix store
    nix-filter.url = "github:numtide/nix-filter";
    # Generate derivations from Dune and OCaml files
    opam-nix.url = github:tweag/opam-nix;
    # Static site generator for OCaml (not yet published to opam)
    yocaml.url = "github:xhtmlboi/yocaml";
    yocaml.flake = false;
  };

  outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix, yocaml }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # Legacy packages that have not been converted to flakes
        legacyPackages = nixpkgs.legacyPackages.${system};
        # OCaml packages available on nixpkgs
        ocamlPackages = legacyPackages.ocamlPackages;

        # OCaml source files
        ocaml-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            "dune-project"
            "garden.opam.template"
            (nix-filter.lib.inDirectory "bin")
          ];
        };
      in
      {
        # Executed by `nix build .#<name>`
        packages.garden =
          (opam-nix.lib.${system}.buildDuneProject
            {
              pinDepends = false;
              repos = [
                (opam-nix.lib.${system}.makeOpamRepo yocaml)
                opam-nix.inputs.opam-repository
              ];
            }
            "garden"
            ocaml-src
            {
              yocaml = null;
              yocaml_unix = null;
              yocaml_yaml = null;
              yocaml_markdown = null;
            }).garden;

        # Executed by `nix build`
        defaultPackage = self.packages.${system}.garden;

        # Executed by `nix run .#<name>`
        apps.garden = {
          type = "app";
          program = "${self.packages.${system}.garden}/bin/garden";
        };

        # Executed by `nix run`
        defaultApp = self.apps.${system}.garden;

        # Used by `nix develop`
        devShell = legacyPackages.mkShell {
          nativeBuildInputs = [
            # Editor support
            legacyPackages.fswatch # for `dune build --watch --terminal-persistence=clear-on-rebuild`
            # legacyPackages.ocamlformat # FIXME: fails to build `uunf` on aarch64-darwin :(
            ocamlPackages.merlin
            ocamlPackages.ocaml-lsp
          ];
          inputsFrom = [
            self.defaultPackage.${system}
          ];
        };
      });
}

brendanzab · 2022-01-10T11:07:52Z

Ahh wierd, for some reason when I do dune build --watch --terminal-persistence=clear-on-rebuild using the above nix file, builds a bin/garden.exe, rather than putting it in the _build directory?

brendanzab · 2022-01-10T11:26:00Z

Ahh wait, I think that must be because of the recommended dune file:

(executable
 (name garden)
 (public_name garden)
 (promote (until-clean)) ; <-- this is the culprit!
 (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))

balsoft · 2022-01-20T14:32:02Z

Thanks to work of @rizo now non-master main branches are also supported! E.g. this works: https://github.com/tweag/opam-nix/blob/main/examples/readme/my-package/my-package.opam#L16

brendanzab · 2022-01-21T07:20:55Z

Should I be looking at https://github.com/tweag/opam-nix#examples-8 for this?

balsoft · 2022-01-21T07:44:20Z

No, it should just work with buildOpamProject and buildDuneProject now.

brendanzab · 2022-01-22T03:03:35Z

So I switched on pinDepends = true:

flake.nix

{
  description = "Static page generator for my digital garden";

  inputs = {
    # Utilities for writing flakes
    flake-utils.url = github:numtide/flake-utils;
    # Precisely filter files copied to the nix store
    nix-filter.url = "github:numtide/nix-filter";
    # Generate derivations from Dune and OCaml files
    opam-nix.url = github:tweag/opam-nix;
    # Static site generator for OCaml (not yet published to opam)
    yocaml.url = "github:xhtmlboi/yocaml";
    yocaml.flake = false;
  };

  outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix, yocaml }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # Legacy packages that have not been converted to flakes
        legacyPackages = nixpkgs.legacyPackages.${system};
        # OCaml packages available on nixpkgs
        ocamlPackages = legacyPackages.ocamlPackages;

        # OCaml source files
        ocaml-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            "dune-project"
            "garden.opam.template"
            (nix-filter.lib.inDirectory "bin")
          ];
        };

        # Nix source files
        nix-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            (nix-filter.lib.matchExt "nix")
          ];
        };
      in
      {
        # Used for `nixpkgs` packages, also accessible via `nix build .#<name>`
        legacyPackages =
          (opam-nix.lib.${system}.buildDuneProject
            {
              pinDepends = true;
              repos = [
                opam-nix.inputs.opam-repository
                (opam-nix.lib.${system}.makeOpamRepo yocaml)
              ];
            }
            "garden"
            ocaml-src
            {
              # Include `yocaml` packages
              yocaml = null;
              yocaml_markdown = null;
              yocaml_mustache = null;
              yocaml_unix = null;
              yocaml_yaml = null;

              # Ask `opam-nix `to include editor tools in the same scope as our
              # package, ensuring that they will be built using the same
              # compiler version etc.
              merlin = null;
              ocaml-lsp-server = null;
            });

        # Executed by `nix build`
        defaultPackage = self.legacyPackages.${system}.garden;

        # Executed by `nix run .#<name>`
        apps.garden = {
          type = "app";
          program = "${self.legacyPackages.${system}.garden}/bin/garden";
        };

        # Executed by `nix run`
        defaultApp = self.apps.${system}.garden;

        # Used by `nix develop`
        devShell = legacyPackages.mkShell {
          nativeBuildInputs = [
            # Editor support
            legacyPackages.fswatch # for `dune build --watch --terminal-persistence=clear-on-rebuild`
            # legacyPackages.ocamlformat # FIXME: fails to build `uunf` on aarch64-darwin :(
            self.legacyPackages.${system}.merlin
            self.legacyPackages.${system}.ocaml-lsp-server
          ];
          inputsFrom = [
            self.defaultPackage.${system}
          ];
        };
      });
}

but now I am getting warnings like:

trace: warning: pin-depends without an explicit sha1 is not supported in pure evaluation mode; try with --impure

I also get a build error:

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/4d0ix5djms3n2jnjdc58l916cwack1rp-empty-directory
source root is empty-directory
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
cp: cannot create regular file '/tmp/opam-subst': Permission denied

brendanzab · 2022-01-22T08:19:31Z

Oh, so I need to set a sha in .opam.template? (sorry, I was expecting it to be ok, given I had supplied the flake input)

brendanzab · 2022-01-22T08:25:57Z

I'm wondering if it's possible to use an opam lockfile to simplify this kind of thing? This is what naersk does, but I'm not too clear on the specifics of how opam works to know if it's possible here too… eg. I'm not sure how to generate the lockfile in the first place. I'm guessing it might require opam to be installed globally?

balsoft · 2022-01-22T08:35:08Z

but now I am getting warnings like:

Well, the warning tells you exactly what to do: run your command again with --impure, or set the git commit sha1 (a.k.a rev) in the opam file (like done in the example: https://github.com/tweag/opam-nix/blob/main/examples/readme/my-package/my-package.opam#L14). To be clear, this is also supported by opam, and is a good idea for reproducibility purposes regardless of opam-nix.

I also get a build error:

That error looks weird, I'll look into it

brendanzab · 2022-01-22T09:23:27Z

Well, the warning tells you exactly what to do

Yeah, I didn't want to run it in --impure, as I thought this was the point of flakes: offering isolation. I also couldn't tell from the warning where to set the sha - I was looking for somewhere in the nix file, not the .opam.template.

To be clear, this is also supported by opam, and is a good idea for reproducibility purposes regardless of opam-nix.

Ahh yeah, this might be me confused by opam's model… (which I find confoundingly frustrating at the best of times). Coming from Cargo git dependencies are automatically pinned in a lockfile - ie. the 'correct way' is enabled by default. I'll go ahead and add the sha to the pin-depends URLs, but it is kind of frustrating as it loses information on what branch it is supposed to be tracking, and can't be updated with tooling. :(

Sorry again for all my confusion!

brendanzab · 2022-01-23T00:08:09Z

Could it be that there is some weirdness in how opam-nix is understanding my pinned deps?

pin-depends: [
  # The sha1 hashes currently point to the main branch of the yocaml repository
  ["yocaml.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_markdown.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_mustache.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_unix.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_yaml.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
]

balsoft · 2022-01-23T07:43:11Z

Certainly possible :)

What is the exact problem you're experiencing?

I've just checked and that snippet does what I would expect. Are you on latest opam-nix?

brendanzab changed the title ~~Fails to download repositories specified in pin-depends~~ Fails to find repositories specified in pin-depends Jan 10, 2022

brendanzab mentioned this issue Jan 10, 2022

Show how to use opam-nix to add dependencies brendanzab/ocaml-flake-example#7

Open

brendanzab closed this as completed Jan 10, 2022

rizo mentioned this issue Jan 20, 2022

Pin-depends do not work for packages outside of opam repository. #2

Closed

rizo mentioned this issue Jan 24, 2022

Materialization does not work with local projects and pin-depends. #6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fails to find repositories specified in `pin-depends` #1

Fails to find repositories specified in `pin-depends` #1

brendanzab commented Jan 10, 2022 •

edited

Loading

balsoft commented Jan 10, 2022 •

edited

Loading

balsoft commented Jan 10, 2022

brendanzab commented Jan 10, 2022

balsoft commented Jan 10, 2022 •

edited

Loading

brendanzab commented Jan 10, 2022 •

edited

Loading

brendanzab commented Jan 10, 2022

brendanzab commented Jan 10, 2022

balsoft commented Jan 20, 2022

brendanzab commented Jan 21, 2022

balsoft commented Jan 21, 2022

brendanzab commented Jan 22, 2022

brendanzab commented Jan 22, 2022 •

edited

Loading

brendanzab commented Jan 22, 2022

balsoft commented Jan 22, 2022 •

edited

Loading

brendanzab commented Jan 22, 2022 •

edited

Loading

brendanzab commented Jan 23, 2022

balsoft commented Jan 23, 2022 •

edited

Loading

Fails to find repositories specified in pin-depends #1

Fails to find repositories specified in pin-depends #1

Comments

brendanzab commented Jan 10, 2022 • edited Loading

balsoft commented Jan 10, 2022 • edited Loading

balsoft commented Jan 10, 2022

brendanzab commented Jan 10, 2022

balsoft commented Jan 10, 2022 • edited Loading

brendanzab commented Jan 10, 2022 • edited Loading

brendanzab commented Jan 10, 2022

brendanzab commented Jan 10, 2022

balsoft commented Jan 20, 2022

brendanzab commented Jan 21, 2022

balsoft commented Jan 21, 2022

brendanzab commented Jan 22, 2022

brendanzab commented Jan 22, 2022 • edited Loading

brendanzab commented Jan 22, 2022

balsoft commented Jan 22, 2022 • edited Loading

brendanzab commented Jan 22, 2022 • edited Loading

brendanzab commented Jan 23, 2022

balsoft commented Jan 23, 2022 • edited Loading

Fails to find repositories specified in `pin-depends` #1

Fails to find repositories specified in `pin-depends` #1

brendanzab commented Jan 10, 2022 •

edited

Loading

balsoft commented Jan 10, 2022 •

edited

Loading

balsoft commented Jan 10, 2022 •

edited

Loading

brendanzab commented Jan 10, 2022 •

edited

Loading

brendanzab commented Jan 22, 2022 •

edited

Loading

balsoft commented Jan 22, 2022 •

edited

Loading

brendanzab commented Jan 22, 2022 •

edited

Loading

balsoft commented Jan 23, 2022 •

edited

Loading