Allow flakes to refer to other flakes by relative path

[edolstra]

E.g.

{
  inputs.foo = { url = ./../foo; };
}

This is primarily useful when you have multiple flakes in the same repository.

However this means that we can't have a lock file entry for the foo input.

[roberth]

A lock file shouldn't be necessary as long as the input is going to be committed to the same repository.

It only becomes a problem when the reference points outside the repository. Doing is useful for testing a change in a dependency, but a warning seems appropriate.

[zimbatm]

It would be good to disallow .. in the relative paths to encourage composing things reliably.

[Kloenk]

Can we only do one lockfile for this, like cargo workspaces does it?

[roberth]

@zimbatm Disallowing .. in relative paths forces the flakes to form a tree rather than a DAG. Suppose you have a repo with flakes with dependencies A -> P -> X and A -> Q -> X, you'd have to duplicate flake X for flakes P and Q.
.. is also useful for ad-hoc testing. For example, you may want to use this feature temporarily to iterate on a dependency that's outside the repo.

@Kloenk I think you get half of this behavior for free if you don't invoke nix flake commands on the dependency flakes. The other half is that if you do invoke it, it can use the parent's lockfile. In cargo this seems to be implicit. Another example is Maven, where you reference the parent pom explicitly. If we require an explicit reference to the parent flake, we preserve the ability to opt out of the single lockfile, which can be important for monorepos. I also like that it reinforces that the flake isn't quite independent. For example, you can't just copy its files and expect nix flake commands to work the same outside the context of the parent. This is true whether the parent reference is explicit or not, but if it's explicit, you'll get an error message to "manage" that expectation; a useful message that's easy to resolve.

[stephank]

So, I’m considering if this is something I want/need. In my case, I have a repository of nixosConfiguration flakes in subdirectories (each a project with one or more machines), and a shared lib. There’re currently hacks in place to make the lib stuff work, because it technically lives outside any of the flake contexts.

• Workspaces sound like a different piece of functionality (pinning all flake dependencies as a whole), and in my case, I want separate lockfiles (per project).

• At the same time, workspaces could be a more sensible way to declare a toplevel than ‘discover git/vcs root’, if we want logic to prevent/warn escaping a certain context.

• My current setup would require .. to work, because lib is outside the flake context.

[MagicRB]

This can sort of be simulated with https://github.com/edolstra/flake-compat, you must make your build impure, but even though it sounds bad, it isn't that bad. If you know your build is pure, you're fine, if you aren't sure, you can temporarily disable the sub-flakes and test each one separately build nix build-ing it. To get this to work with nixos-rebuild you have to edit the script and add "--impure" to extraBuildFlags. Or add an option into the switch

oh and all the sub-flakes have their own lock-files, therefore its necessary to update them recursively, I'll write a script for that.

[MagicRB]

@stephank

* My current setup would require `..` to work, because `lib` is outside the flake context.

perhaps something like this could work, though I have no clue how to actually implement it

{
  inputs = {
    your-lib = {
      path = ./your-lib; # different from url to distinguish it
    };
    
    machine-1 = {
      path = ./machine-1;
      pass-on = your-lib; # I don't really know what `your-lib` should be, it could be a string, but that could cause confusion as registry references look the same. Also, I already saw something like this somewhere, can't remember what was the option, if you know then please tell me.
    };
  };
}

{
  inputs = {
    your-lib = {
      follows = "your-lib"; # this is the option used for referencing flake registries, I'm sure this time
    };
  };
}

[jaen]

I just want to add another two cents to the bandwagon – I have two use cases that keep me wanting to reach for flakes with a relative path (and possibly some kind of arguments, but that's a separate discussion).

One is the already mentioned idea to factor out common code into a lib flake shared across some nixosConfiguration flakes and possibly version things like nixpkgs on a per-configuration basis. Another is using nix to build a multi-language repo I have for work, where each sub-project could be a flake I then bring together.

I'm not sure if it's the right instinct to try reaching out for things like that, as I've been using NixOS mostly for Terraform/Ansible angle of reproducible configuration and I'm not sure how that fits in the package building conventions Nix has – maybe there are other options that are better suited to those use-cases I'm not aware of.

[MagicRB]

I'd add that sub-flakes are a useful tool to "partition" your repo, for example my dotfiles (EDIT: I moved those packages to my systems repo) themselves provide custom packages I created and I propagate those up from "half-flakes", flakes whose inputs are declared at the toplevel flake, but look like separate flakes and mostly behave the same way. Just call the outputs lambda with the unified inputs and self

[jaen]

Another semi-related thing I've bumped against just now is that after a few tries to use things like submodules or impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this) I bumped into enough issues that I decided to just split it out as separate flakes.

Unfortunately this now incurs the change->commit->push->update-flake penalty to make a simple change which is kind of annoying, is there plans for some way to develop flakes alongside a la yarn link (basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it) for flakes. If not, would it make sense to open an issue for it?

[MagicRB]

impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this)

definitely not

Unfortunately this now incurs the change->commit->push->update-flake penalty

look at my dotfiles, I had the same issue and I solved it by those half-flakes

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/5

[jorsn]

Currently, relative paths work already with the path fetcher. However, this comment looks like the restrictions are just not implemented. flake-compat doesn't allow relative paths, because builtins.path needs an absolute path string or a path (e.g. ./. + "/lkjsfl") as attribute path in the argument set.

Edit: What is the problem with a lock file?

[BBBSnowball]

basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it

nix flake update --override-input ... might do what you want. If you use it with path:/something, it will record the hash of that path so you have to run the command again after changing any file in /something. This should remove the "commit->push" part but you still have to update the flake.

This was still too much overhead in my edit-test-loop. Therefore, I have merged most of my repos and use a similar approach as MagicRB.

I hope relative paths with ".." will remain possible for path: and I will switch to plain .. if this becomes possible. I use (misuse?) them for these purposes:

1. I have a repo with modules and configurations (nixosConfigurations) for several hosts. Later, I wanted to change the nixpkgs input for one of them. I did that with a sub-flake (in the same repo) that uses the main flake via path:../... The sub-flake has its own lockfile.
2. A friend wanted to use a package from my flake. He is using nixos-unstable while I'm forcing a specific version of nixos-20.09 in the flake. Therefore, any package from my flake will download lot's of dependencies for him. I have created a sub-flake that sets inputs.nixpkgs.url = "nixpkgs" and doesn't have a lock file. That way, he can add ?dir=nolock to the flake url and all is well.

[nrdxp]

Currently this already works with inputs.subflake.url = "path:./adirectory", but if a lock file is not already generated, it can fail if your not in the directory with the flake.nix. Once the lock file exists, you can reference the flake from anywhere and it works as expected.

Perhaps a more elegant solution to this, which would also solve the above mentioned issue, would be to pass a special self reference to the flake inputs, in a similar manner as is passed to outputs. Then you could reference a subflake as inputs.subflake.url = "self:adirectory".

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/1

[DieracDelta]

This issue with path is that you cannot mark submodules = true;. This complicates things if you want to build from a local checkout of a GitHub repo and all its submodules.

[zhaofengli]

The current path:../sub-dir-of-same-repo solution doesn't actually work as it comes with a giant catch: Everything, including .gitignore'd files, in that directory is copied. This is unacceptable when you are developing and your flake directory contains loads of build artifacts. In the current implementation you will need to cargo clean or manually move the offending directories out from the input flake in order to not copy lots (in my case, gigabytes) of data into the Nix store.

[roberth]

@zhaofengli does git+file:../sub-dir-of-same-repo work?

[zhaofengli]

@roberth

Thanks for the suggestion. I tried the following and got:

• git+file:../sub-dir -> fetching Git repository 'file:../sub-dir'ssh: Could not resolve hostname file: Name or service not known
• git+file://../sub-dir -> error: file:// URL 'git+file://../sub-dir' has unexpected authority '..'
• git+file:/..?dir=sub-dir -> error: file:// URL 'git+file://..?dir=sub-dir' has unexpected authority '..'
• git+file:///home/zhaofeng/repo?dir=sub-dir -> Works, but it's not really scalable when you have multiple flakes and collaborate with other people

[eclairevoyant]

@mikepurvis see #3978 (comment)

[r-vdp]

@mikepurvis in addition to @eclairevoyant's response, the comment just above should help to unblock you: #3978 (comment)

If you run nix flake lock with the right arguments, hydra should create the needed store path and then be able to evaluate the flake.

It's an ugly work-around until this issue gets resolved.

[mikepurvis]

If you run nix flake lock with the right arguments, hydra should create the needed store path and then be able to evaluate the flake.

Hm, I appreciate this thought, but I don't think it works for me as it looks like that hack isn't just necessary on Hydra, but necessary anywhere the outer flake might be evaluated (I got the same error in an empty container). So I think in the short term I'll leave the skeleton of this structure in place, but without the actual flake part, and then adopt that at some point after this issue is properly resolved.

[aviallon]

I wasn't able to get even the trivial case of inputs.thing.url = "path:./thing" working on Hydra. It was doing the expected thing locally, but then on Hydra (with Nix 2.13.3):

in job ‘<redacted>.x86_64-linux’:
error: cannot fetch input 'path:./thing?lastModified=1&narHash=sha256-tk7BEyl8BW4OSrT%2fRCAhYBzJx3LQJxPcV1e2MFRl7kM=' because it uses a relative path

Is there some trivial error I'm making with this? I'd really like this setup to work as being able to pass an --override-input thing <other-thing> provides some limited parameterization that helps my workflow.

I finally got a nice workaround: if you are using git for your flake (which you probably are), you can do:

inputs = {
  subflake.url = "git+file:.?dir=mysubflake_subdir";
};

And it should Just Work™

[irisjae]

@aviallon Does your solution require the subflake to be in its own git submodule or something to work? With both my subflake and the parent flake in the same repo, I got no luck getting it working.

[aviallon]

@irisjae No, it does not. However, only commited files are taken into account by the flake.
And by default, the master branch is used. You have to pass &rev=HEAD to use the last (even if unpushed) commit.

[mikepurvis]

@aviallon The . in this solution is unfortunately relative to the invocation location rather than the top-level file. That makes it unsuitable for me, since I need my top level flake to be invokable from anywhere (using a flake-registry mapping).

[aviallon]

@mikepurvis I didn't even know that... I wonder if we could get the current flakes's path while in the inputs scope... I guess not.

[irisjae]

Is there documentation available for the "git+file:...?..." URL scheme? I've only been able to find examples on this on Nix Flakes documentation.

[golddranks]

I'm not able to get git+file: URL scheme working. It tries to search for host "file". It only works when specifying git+file://, and then it expects an absolute path.

$ nix --version
nix (Nix) 2.13.3

[aiotter]

@irisjae Check out nix flake --help

[MatrixManAtYrService]

As a workaround, instead of this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    # not supported:
    # foo.url = "path:./subflake/foo";

Maybe something like this?

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
      };
    in rec
    {

      # workaround
      foo-flake = import ./subflake/foo/flake.nix;
      foo-outputs = foo-flake.outputs {
        inherit nixpkgs;
        inherit flake-utils;
      };
      foo = foo-outputs.packages.${system}.default;

      packages = rec {
        mypkg = pkgs.buildBarPackage {
          buildInputs = [ foo ];
        };
        default = mypkg;
...

It lets me use the top-level flake to explain how to package my component, while hiding the details re: how to package its dependencies in a different flake, kept in a subfolder. Of course it would be nice if this wasn't necessary and all dependencies were already packaged in a nix-friendly way, but that's not always the case.

(If there's already something in nixpkgs that does this in a more succinct way, then I'd like to know about it)

[colonelpanic8]

I'm not able to get git+file: URL scheme working. It tries to search for host "file". It only works when specifying git+file://, and then it expects an absolute path.

$ nix --version
nix (Nix) 2.13.3

Yeah I'm seeing the same thing.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/experimental-does-not-mean-unstable-detsyss-perspective-on-nix-flakes/32703/12

[MatrixManAtYrService]

Does this maybe belong here: https://github.com/NixOS/nix/milestone/27 ?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-re-locking-necessary-at-each-evaluation-when-import-sub-flake-by-path/34465/1

[13x1]

Just gonna leave this here in case someone else gets a weird "error: relative path x points outside of its parent's store path y" with no explanation when using relative paths in flakes: If you have flakes in the subdirectories of a Git repo and get this error, make sure that you actually have the .git folder in the directory above the flakes. Or stuff will break with no explanation and you will waste hours trying to debug this (like me) :')

[domenkozar]

When using devenv I find best to do:

inputs:
  root:
    url: git+file://.
    flake: false
imports:
- root/examples/compose/projectA