builtins.path with builtins.filterSource copies all files to /nix/store

[misuzu]

Describe the bug

My issue is with the way builtins.path works with builtins.filterSource.
After compiling rust package using cargo, it dumps gigabytes of its files to target directory. When I later run nix-build, the entire package directory with gigabytes of garbage gets copied to /nix/store and only then builtins.path takes only what is needed.
After running cargo build and nix-build several times, /nix/store hogs all available disk space.

Steps To Reproduce

I'm using nix-gitignore.gitignoreSourcePure in my rust package:
src = nix-gitignore.gitignoreSourcePure [ "target" ] ./.;

Expected behavior

Maybe it's possible to filter sources without copying everything to /nix/store?

nix-env --version output
nix-env (Nix) 2.5.1

[roberth]

A solution for this problem exists in Nixpkgs. The cleanSourceWith family of functions returns attrsets that only add files to the store on demand, optimizing away intermediate store additions.
Almost all use cases of builtins.path and builtins.filterSource are better solved with cleanSourceWith calls instead.

For example:

src = cleanSourceWith {
  filter = path: type: f path type;
  src = nix-gitignore.gitignoreSourcePure [ "target" ] ./.;
};

Here the nix-gitignore source's outPath attribute that would copy the files to the store is ignored. Instead, cleanSourceWith takes the filter and original path from the nix-gitignore source, then adds its own filter.
The final src attrset can then be used in an derivation or string interpolation, which will evaluate the outPath attr and add it to the store.

[misuzu]

A solution for this problem exists in Nixpkgs.

It doesn't actually work. The entire ./. is still copied to /nix/store before filtering.

I've tried this:

src = lib.cleanSource (nix-gitignore.gitignoreSourcePure [ "target" ] ./.);

And this:

src = nix-gitignore.gitignoreSourcePure [ "target" ] (lib.cleanSource ./.);

And this (Cargo.* files are missing when building, so it fails, but ./. along with target directory is still getting dumped to /nix/store):

src = lib.sourceByRegex ./. ["src"];

[roberth]

Oh, you must be using flakes then, where ./. isn't what you think it is. Those always copy the whole flake to the store before evaluation. #5549

[misuzu]

Oh, you must be using flakes then, where ./. isn't what you think it is. Those always copy the whole flake to the store before evaluation. #5549

I'm using flakes for my system, yes, but not to build my rust package (#5549 is one of the reasons).
I'm using regular nix-build command.

[misuzu]

I've tried building my package using nix 2.3.16 - it has the same issue. It also spits a warning:
warning: dumping very large path (> 256 MiB); this may run out of memory

[misuzu]

I've figured out what's wrong with my setup. I've had the following function to simplify package management in my monorepo:

callPackages = callPackage: path: with pkgs.lib;
  mapAttrs
    (n: v: callPackage "${path}/${n}/default.nix" { })
    (filterAttrs
      (n: v: v == "directory" && (builtins.pathExists "${path}/${n}/default.nix"))
      (builtins.readDir path));

I can use this function to get an attrset with all my packages without manually defining all of them:
packages = callPackages pkgs.callPackage ./directory-with-packages;

The actual issue was with "${path}/${n}/default.nix" part. For some reason, it forces copying ${path} to /nix/store. I haven't noticed this because I've not used rust before.

I've modified the first function to use (path + "/${n}/default.nix") instead, which workarounds my issue.

callPackages = callPackage: path: with pkgs.lib;
  mapAttrs
    (n: v: callPackage (path + "/${n}/default.nix") { })
    (filterAttrs
      (n: v: v == "directory" && (builtins.pathExists (path + "/${n}/default.nix")))
      (builtins.readDir path));