add path to 'very large path' error

[bburdette]

This is to address issue #1184, the vague 'dumping very large path' warning. Rather than changing the warning message itself, we're creating an Activity when we copy a path, similar to what happens in store-api.cc.

Here's the output with my test case:

[nix-shell:~/code/nix-error-project/1184/test-case]$ ./newbuild.sh 
copying path '/home/bburdette/code/zknotes/zknotes/server'...
warning: dumping very large path (> 256 MiB); this may run out of memory
^Cerror: interrupted by the user

I put in an isInStore check before adding the Activity, otherwise there's quite a bit of extra output:

[nix-shell:~/code/nix-error-project/1184/test-case]$ ./newbuild.sh 
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/stdenv/generic/default-builder.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/stdenv/linux/bootstrap-tools/scripts/unpack-bootstrap-tools.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/stdenv/generic/builder.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-hooks/move-docs.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-hooks/make-symlinks-relative.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-hooks/compress-man-pages.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-hooks/strip.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-hooks/patch-shebangs.sh'...

                      <a few hundred more lines of these>

copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/setup-d
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/development/compilers/rust/setup-hook.sh'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/build-support/rust/patch-registry-deps'...
copying path '/nix/store/rrhid0i04x3v4majkx9sjzg6jrpxl3hz-nixos-20.09.2787.6e7f25001fe/nixos/pkgs/development/compilers/llvm/10/llvm-outputs.patch'...
copying path '/home/bburdette/code/zknotes/zknotes/server'...
warning: dumping very large path (> 256 MiB); this may run out of memory

[Ekleog]

Interesting, I hadn't been aware of this PR when opening #4625! Guess that's an alternative, and both ways make reasonable sense to me — I kind of like #4625 just because it adds metadata, thus hopefully opening the way for other improved diagnostics, but OTOH it's quite a bit more code so I guess some balance will have to be found :)

Something I think is missing from here is the fact that currently the logic is to display the warning only once. I changed said logic in this part, this part, this part and this part of my PR if I'm not missing any.

[domenkozar]

@Ekleog Have you also seen the discussion at #4605

[domenkozar]

@edolstra what's missing here?

[domenkozar]

@edolstra would be great to know if this is a good way forward to debug these warnings.

[stale]

I marked this as stale due to inactivity. → More info

[longregen]

It would be really great to have some way to debug this. I'm not a seasoned NixOS user, and I was trying to install it on a 16GB machine, and I can't tell what is making the nixos-install procedure to fail. It could be a real downer for a newbie to stumble upon this error!

copying channel...
building the configuration in /mnt/etc/nixos/configuration.nix...
warning: dumping very large path (> 256 MiB); this may run out of memory
fatal flex scanner internal error--end of buffer missed

[jceb]

I an into the same error message - fatal flex scanner internal error--end of buffer missed shortly after I started the installation of NixOS via nixos-install. As a workaround, I massively increased swap space. Currently, the installation is using ~50 GB of RAM in total. A bit excessive 🤯

[jceb]

Passing the option --no-channel-copy to nixos-install solved the issue for me.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-17-nix-team-meeting-minutes-41/26614/1

[bburdette]

@Ericson2314 I'm happy to get this branch working again, if it looks like its something that's desired.

[Ericson2314]

@bburdette Yeah maybe. But perhaps it would be better to put the activities in the underlying implementations?

[Ericson2314]

I think this is no longer needed because InputAccessor::fetchToStore does this:

nix/src/libfetchers/input-accessor.cc

Line 36 in 02bd821

Activity act(*logger, lvlChatty, actUnknown, fmt("copying '%s' to the store", showPath(path)));

Do leave a comment if this is not the case!