Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flake support #3573

Draft
wants to merge 740 commits into
base: master
from
Draft

Flake support #3573

wants to merge 740 commits into from

Conversation

@edolstra
Copy link
Member

edolstra commented May 6, 2020

This PR adds experimental support for flakes. The flake file format is defined by RFC 49. To use it, you have to add

experimental-features = nix-command flakes

to nix.conf.

Other changes:

  • The nix command is flake-based now. For instance, nix build nixpkgs#hello builds the hello package from the nixpkgs flake. Non-flake-based operation is still supported, e.g. nix build -f /path/to/expr.nix. However, nix no longer uses NIX_PATH so it's no longer affected by channels.

  • New subcommand nix flake for working with flakes.

  • New subcommand nix profile for imperative package management (intended to replace nix-env). (This is not completely finished. Note that once a profile has been modified using nix profile, you can't use nix-env on it anymore.)

  • New subcommand nix run (previously named nix app) to run an application defined by an app or defaultApp output of a flake.

  • The nix command has an evaluation cache to speed up querying flake attributes. For instance, nix search uses this instead of the old JSON cache.

edolstra added 30 commits Feb 5, 2020
E.g.

  $ nix edit .#nixosConfigurations.bla

now works.
This allows all supported fetchers to be used, e.g.

  builtins.fetchTree {
    type = "github";
    owner = "NixOS";
    repo = "nix";
    rev = "d4df99a3349cf2228a8ee78dea320afef86eb3ba";
  }
Example usage:

  $ nix eval-hydra-jobs -f '<nixpkgs/pkgs/top-level/release.nix>' '' \
      --max-memory-size 2048 --workers 8
Fixes #3361.
Fixes #3363.
Looks like this got accidentally revived after 204291f.
@edolstra
Copy link
Member Author

edolstra commented Jun 17, 2020

@andir Thanks for the suggestion. builtins.getFlake now only exists if the flakes feature is enabled (and getFlake is gone).

edolstra added 3 commits Jun 17, 2020
E.g.

  $ nix run nixpkgs#hello
  error: --- Error ---------- nix
  flake 'flake:nixpkgs' does not provide attribute 'apps.x86_64-linux.hello' or 'hello'

instead of

  $ nix run nixpkgs#hello
  error: --- Error ---------- nix
  flake 'flake:nixpkgs' does not provide attribute 'hello'
@@ -440,6 +440,16 @@ ref<eval_cache::EvalCache> openEvalCache(
}));
}

static std::string showAttrPaths(const std::vector<std::string> & paths)
{
std::string s;

This comment has been minimized.

Copy link
@andir

andir Jun 17, 2020

Member

This might be a bit nit-picky but I think we can improve readability and reduce duplicate code (e.g. the concat and quote code) by just implementing it like this:

static std::string showAttrPaths(const std::vector<std::string> & paths)
{
        std::string s;
        auto quotedPaths = quoteStrings(paths);
        auto it = quotedPaths.begin();

        if (it != quotedPaths.end())
                s = *it++;

        if (it != quotedPaths.end()) {
                auto rem = concatStringsSep(", ", std::vector<std::string>(it, quotedPaths.end()));
                s += " or " + rem;
        }

        return s;
}

This removes the ternary operator and the line that contains three different statements that mostly consists of quote & operator characters. Worst case this will create a copy of the remaining elements of the vector but IMHO that shouldn't hurt given the benefits.

This comment has been minimized.

Copy link
@gilligan

gilligan Jun 17, 2020

Contributor

I sort of got nerd-sniped reading this and tried to write something that i can parse quite easily:

/**
 * Takes a list of paths and turns them into a comma separated list
 * of the format shown below:
 *
 * { }               -> ""
 * { "x" }           -> "'x'"
 * { "x", "y" }      -> "'x' or 'y'
 * { "x", "y", "z" } -> "'x' or 'y', 'z'"
 */
static std::string showAttrPaths(const std::vector<std::string> & paths)
{
    if (paths.size() == 0)
        return "";

    if (paths.size() == 1)
        return "'" + paths[0] + "'";

    auto quotedPaths = quoteStrings(paths);
    auto it = std::next(quotedPaths.begin());

    return "'"
        + paths[0]
        + "' or "
        + concatStringsSep(", ", std::vector<std::string>(it, quotedPaths.end()));
}

By the way: In the case of an empty paths vector, should the result really just be an empty string?

edolstra added 2 commits Jun 18, 2020
@B4dM4n
Copy link

B4dM4n commented Jun 21, 2020

I recently started experimenting with flakes and hit a similar issue as @Kloenk with nix flake check.

When using the forAllSystems idiom to support multiple systems and adding an apps or defaultApp output nix flake check fails with something like this on a x86_64-linux system:

a 'aarch64-linux' with features {} is required to build '/nix/store/9wpnn4am8jc1vyp40ks6hizk3m53d4r0-hello.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

example flake gist

I fixed this error with bf1deda (will open a PR later). Opened #3726 with a fix.

Working on the fix made me realize that checks can currently only be build for the host system, which limits their usability a bit. For CI environments this is not a big issue as the check can be run on many hosts, but for personal usage I would like to run nix flake check on my local machine and build the checks for all supported systems (settings.thisSystem, settings.extraPlatforms and remote builds).

A possible solution I came up with is 5fb4cc3, which replaces the --no-build flag with a --build-systems flag. The new flag defaults to the local system to retain the current behavior, but allows to specify the exact set of systems that will be build (empty list is equivalent to --no-build).

Example usage:

nix flake check --build-systems ""
nix flake check --build-systems "x86_64-linux x86_64-darwin i686-linux"

I can open a PR for this proposal for further discussion if desired.

@Ericson2314
Copy link
Member

Ericson2314 commented Jun 21, 2020

I am not up date with flakes, so perhaps I misunderstand what @B4dM4n but it sounds like more evidence that we really need to move away from building and running tests in the same derivation and towards 3 derivations:

  1. Build regular thing
  2. Build tests
  3. Run tests

It's mainly on Nixpkgs to just do this, but any way we can get the tooling to nudge us in that direction is a plus.

@B4dM4n
Copy link

B4dM4n commented Jun 21, 2020

@Ericson2314 The issue I described has nothing to do with regular package tests. nix flake check just helps to ensure that a flake.nix follows the basic structure for Nix flakes and also builds some of the outputs (currently apps, defaultApp and checks).

@Ericson2314
Copy link
Member

Ericson2314 commented Jun 21, 2020

At least checks is it's own derivation. I suppose 1 and 3 needs names, but 2 doesn't, in my list above, at a bare minimum.

edolstra added 3 commits Jun 22, 2020
This was inconsistent since we're not building 'packages' or
'defaultPackage' either.

Closes #3726.
@mkenigs
Copy link
Contributor

mkenigs commented Jun 27, 2020

I've been thinking about usability, and I guess some of this might be more of my thoughts related to nix in general. But if I was trying to to convince someone to use nix, I would want a flake to look something more like this:

{
  description = "optional description";
  inputs.build = [ boost ];
  inputs.test = [ testframework ];
  outputs = { self, inputs }: {
    mkFlake {
      name = "name";
      build =
        ''
          make
        '';
      install =
        ''
          cp name $out/bin/
        '';
      run = "$out/bin/name";
      test =
        ''
          test
        '';
    }
  }
}

I don't know how things like nixpkgs, system, mkDerivation, and other outputs would be handled, so I guess this is more just my thoughts rather than something concrete. But if it was possible, I think something along these lines would be way easier to digest as a newcomer to nix. It's pretty easy to understand what's going on: you say what dependencies you need, nix does it's nix magic, and you get something you can build/run/develop/test anywhere with nix build/run/develop/test (flake check). I think it's hard to realize that's what nix can do when it's hidden behind complexity, but it's actually incredibly appealing once you see it.

edolstra added 5 commits Jun 29, 2020
…efinition

'nix run' will try to run $out/bin/<name>, where <name> is the
derivation name (excluding the version). This often works well:

  $ nix run nixpkgs#hello
  Hello, world!

  $ nix run nix -- --version
  nix (Nix) 2.4pre20200626_adf2fbb

  $ nix run patchelf -- --version
  patchelf 0.11.20200623.e61654b

  $ nix run nixpkgs#firefox -- --version
  Mozilla Firefox 77.0.1

  $ nix run nixpkgs#gimp -- --version
  GNU Image Manipulation Program version 2.10.14

though not always:

  $ nix run nixpkgs#git
  error: unable to execute '/nix/store/kp7wp760l4gryq9s36x481b2x4rfklcy-git-2.25.4/bin/git-minimal': No such file or directory
@FRidh

This comment has been minimized.

Copy link
Member

FRidh commented on 26cf0c6 Jul 1, 2020

Nice feature! Could Nix first check whether the executable exists before running it, and describe what should be done in case the executable does not exist? This may otherwise be a bit too much magic and get confusing. Typically this type of checking is avoided because the file could be created/moved, but with the read-only store we do not have this issue.

This comment has been minimized.

Copy link
Member

zimbatm replied Jul 1, 2020

Regarding pkgs.gitMinimal, maybe we could move the variants into the version, like git-2.26.2+minimal. Another idea is to introduce a passthru.entrypoint that can be used to override the location of the binary.

edolstra added 6 commits Jul 1, 2020
This allows you to refer to an input from another flake. For example,

  $ nix run --inputs-from /path/to/hydra nixpkgs#hello

runs 'hello' from the 'nixpkgs' inputs of the 'hydra' flake.

Fixes #3769.
@nixos-discourse

This comment has been minimized.

Copy link

nixos-discourse commented on 68f524d Jul 6, 2020

This commit has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-flakes-problems-with-dev-output/7934/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.