Various Nix commands ignore Ctrl-C (SIGTERM) #7245

9999years · 2022-10-31T20:52:07Z

Describe the bug

Several Nix commands seem to ignore Ctrl-C for lengthy periods. I'm not sure exactly what conditions cause this, but I've seen it with nix flake update, nix develop, and nix-shell.

$ nix flake update
[111.7 MiB DL]^C^C^C^C^C^C^C^C^C^C^C^C

Steps To Reproduce

With this flake.nix:

{
  description = "A very basic flake";

  inputs = {
    nixpkgs = {
      type = "github";
      owner = "NixOS";
      repo = "nixpkgs";
      ref = "master";
    };
  };

  outputs = {
    self,
    nixpkgs,
  }: {
    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;
    packages.x86_64-linux.default = self.packages.x86_64-linux.hello;
  };
}

Run nix flake update, and wait until the downloading 'https://api.github.com/repos/NixOS/nixpkgs/tarball/... message disappears. Pressing Ctrl-C at that time will be ignored.

On subsequent runs, no message will be printed to stdout before nix flake update starts ignoring Ctrl-C. UPDATE: It looks like on subsequent runs, nix flake update deadlocks entirely after Ctrl-C is pressed. I've been watching this do nothing for 10 minutes (far longer than it should take to get the commit hash of master on the nixpkgs repo):

$ nix flake update
^C^C^C^C^C

^C^C^C^C^C^C^C^C

Expected behavior

The program exits cleanly and immediately.

nix-env --version output:

$ nix-env --version
nix-env (Nix) 2.11.0

Additional context

Observed on M1 macOS (aarch64-darwin).

The text was updated successfully, but these errors were encountered:

matthewbauer · 2022-10-31T21:04:55Z

I think #6995 might help, but I have seen this happen even with that change.

9999years · 2022-10-31T21:08:17Z

Can the Nix project at least cut a release with #6995 included? There have been 220 commits since the last release:

$ git log --oneline 2.11.1..master | wc -l
     220

nrdxp · 2022-11-01T21:02:57Z

I don't think this is isolated to a mac. My most annoying experience with this is when using direnv, and while I won't blame Nix for my decision to automatically load devshells, occassionally I just need to do something very briefly in the repo without the need for the devshell and a Nix eval can sometimes be expensive, not being able to quickly cancel out of it is more annoying than it needs to be.

matthewbauer · 2022-11-02T00:50:58Z

My most annoying experience with this is when using direnv, and while I won't blame Nix for my decision to automatically load devshells, occassionally I just need to do something very briefly in the repo without the need for the devshell and a Nix eval can sometimes be expensive, not being able to quickly cancel out of it is more annoying than it needs to be.

Does it happen outside of direnv though? I tried pretty hard to repro it on Linux in the past and could never get it to happen. Could be it's just much rarer though. It could also be a bug in direnv.

nrdxp · 2022-11-02T15:37:21Z

Does it happen outside of direnv though?

I don't see how it could be direnv when I have to send a kill -9 to the nix cli process to get it to quit since a regular SIGTERM is ineffective, meaning the Nix process itself is ignoring the signal. I have not found a reliable way to repro, unfortunately, which is why I never bothered to write a ticket before, but it's really annoying and happens to me usually at least once a day.

IIRC, this has also happened to be a few times during a heavy nix build, but I can't say for sure.

matthewbauer · 2022-11-02T15:44:43Z

Yeah that sounds like nix :). It is really hard to repro. I was getting it to happen like 1 out of 10 times or something. I believe it happens the most during download threads.

thufschmitt · 2022-11-03T15:39:39Z

I've seen this happening a few times indeed. It might a duplicate of #5438.

I can't manage to reproduce it reliably (the reproducer in the issue doesn't work for me, I only get a few seconds of delay between my C-C and Nix stopping)<. However, I suspect it's either linked to the download threads as @matthewbauer suggests or the serialization that happens when copying the files to the store.

9999years · 2022-11-03T15:50:43Z

I'm on a 20-core machine which may exacerbate the issue.

aakropotkin · 2022-11-16T05:03:49Z

If this is helpful to anyone I'm almost certain the issue is specifically hitting <CTRL-C> while a fetch is occurring.

There might be a valid reason for this, but in practice it means I usually have to kill -KILL.

I hadn't restarted my box for about a week and found ~30 hung nix processes.

I can reproduce the issue reliably in most large repos with a dirty tree, or by fetching a large number of tarballs with builtins.fetchTree. If you are trying to trigger it using -vvvvv to find the right moment to signal is helpful. I basically use this flag to know when I can safely SIGINT.

If you want a test subject with a gross number of tarballs ( ~500 ) you can try

nix eval 'github:aakropotkin/flocoPackages?dir=info/unscoped/a/aws-sdk#fetchInfo'  \
  -vvvvv  \
  --apply 'builtins.mapAttrs ( _: builtins.fetchTree )';

You can <CTRL-C> after ~1 second and call kill -INT all you want and it'll lock up.

arianvp · 2023-06-05T10:34:32Z

Not stale. Is there any way we can get this issue prioritised? I've been in a place with poor network conditions for the past week, and the amount of hours waiting for nix to exit because it does not respond to interrupt signals is extremely frustrating to the point that it's basically unusable.

edolstra · 2023-06-05T10:51:00Z

It's annoying if Ctrl-C doesn't work, but you can just kill it. No need to wait for hours.

arianvp · 2023-06-05T11:54:07Z

Yes but it's not good UX. E.g. these lockups also happen when you autocomplete a nix flake. So it's not even clear to the user a nix process is running

roberth · 2023-06-05T13:17:22Z

Overheard on matrix:

You can always use 2.3 which doesn't suffer from this.

Adding new-cli label.

arianvp · 2023-06-05T17:24:30Z

It also happens in the old-cli commands for sure though. Run into this at work every day and we're still using the old nix cli.

I think for the old CLI the trigger is fetchGit and friends at eval time

stephenjudkins · 2023-06-06T20:14:28Z

I have also noticed this and had to go and kill these processes. Additionally, when I've done so, it's left some things in a locked state that's difficult/slow to restart. So it's a significant UX issue

Ericson2314 · 2023-06-06T21:22:27Z

Yeah this one is annoying.

kjeremy · 2023-08-11T14:13:32Z

I'm seeing this when a nix operation triggers a garbage collect

arianvp · 2023-08-30T10:09:58Z

Since 2.17 I also experience this whilst nix is doing evaluation and no fetching

e.g. nix profile install nixpkgs#jq will hang if you C-c immediately.

marijanp · 2023-10-19T10:51:42Z

Anyone who is investigating this: @aakropotkin's command can be used to reliably reproduce this issue. Here is an updated command using the revision when he posted the comment:

nix eval 'github:aakropotkin/flocoPackages/7e6a0eabbe46394ebc3c10154cbbd90b01f57efb?dir=info/unscoped/a/aws-sdk#fetchInfo'  \
  -vvvvv  \
  --apply 'builtins.mapAttrs ( _: builtins.fetchTree )';

marijanp · 2023-10-19T11:28:02Z

This is a backtrace I was able to obtain:

Thread 1 "nix" received signal SIGUSR1, User defined signal 1.
0x00007f3d9f89c9e6 in __futex_abstimed_wait_common () from /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libc.so.6
(gdb) bt
#0  0x00007f3d9f89c9e6 in __futex_abstimed_wait_common () from /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libc.so.6
#1  0x00007f3d9f8a1833 in __pthread_clockjoin_ex () from /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libc.so.6
#2  0x00007f3d9fae0637 in std::thread::join() () from /nix/store/xq05361kqwzcdamcsxr4gzg8ksxrb8sg-gcc-12.3.0-lib/lib/libstdc++.so.6
#3  0x00007f3d9fea3ca5 in nix::RemoteStore::ConnectionHandle::withFramedSink(std::function<void (nix::Sink&)>) [clone .cold] ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixstore.so
#4  0x00007f3d9ffb7e43 in nix::RemoteStore::addCAToStore(nix::Source&, std::basic_string_view<char, std::char_traits<char> >, nix::ContentAddressMethod, nix::HashType, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> > const&, nix::RepairFlag) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixstore.so
#5  0x00007f3d9ffb8159 in virtual thunk to nix::RemoteStore::addToStoreFromDump(nix::Source&, std::basic_string_view<char, std::char_traits<char> >, nix::FileIngestionMethod, nix::HashType, nix::RepairFlag, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> > const&) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixstore.so
#6  0x00007f3d9ffcd409 in nix::Store::addToStore(std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::FileIngestionMethod, nix::HashType, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&, nix::RepairFlag, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> > const&) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixstore.so
#7  0x00007f3da01ca389 in nix::fetchers::downloadTarball(nix::ref<nix::Store>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixfetchers.so
#8  0x00007f3da01a8732 in nix::fetchers::GitArchiveInputScheme::fetch(nix::ref<nix::Store>, nix::fetchers::Input const&) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixfetchers.so
#9  0x00007f3da01996ab in nix::fetchers::Input::fetch(nix::ref<nix::Store>) const () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixfetchers.so
#10 0x00007f3da0430d14 in nix::FlakeRef::fetchTree(nix::ref<nix::Store>) const () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixexpr.so
#11 0x00007f3da04237ec in nix::flake::fetchOrSubstituteTree(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixexpr.so
#12 0x00007f3da0425994 in nix::flake::getFlake(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixexpr.so
#13 0x00007f3da0426c9a in nix::flake::getFlake(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixexpr.so
#14 0x00007f3da0429f85 in nix::flake::lockFlake(nix::EvalState&, nix::FlakeRef const&, nix::flake::LockFlags const&) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixexpr.so
#15 0x00007f3da010f350 in nix::InstallableFlake::getLockedFlake() const () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#16 0x00007f3da01107db in nix::InstallableFlake::getCursors(nix::EvalState&) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#17 0x00007f3da011150b in nix::InstallableValue::getCursor(nix::EvalState&) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#18 0x00007f3da0108888 in nix::InstallableFlake::toValue(nix::EvalState&) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#19 0x0000555f6c1a0126 in CmdEval::run(nix::ref<nix::Store>, nix::ref<nix::InstallableValue>) ()
#20 0x00007f3da00fd76e in nix::InstallableValueCommand::run(nix::ref<nix::Store>, nix::ref<nix::Installable>) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#21 0x00007f3da0120204 in nix::InstallableCommand::run(nix::ref<nix::Store>) () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#22 0x00007f3da00feb97 in nix::StoreCommand::run() () from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixcmd.so
#23 0x0000555f6c1c227e in nix::mainWrapped(int, char**) ()
#24 0x00007f3da0219ec1 in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) ()
   from /nix/store/7kcvfjnsmfn0h8az5q38whz00dqd92am-nix-2.17.0/lib/libnixmain.so
#25 0x0000555f6c11b817 in main ()

marijanp · 2023-10-20T12:49:53Z

I'm able to consistently trigger this after the following output occurs

performing daemon worker op: 7

tomberek · 2023-11-22T22:51:58Z

Likely related to fetchTree clean up when interrupted. If we fix it in that one spot, it might resolve much of this. Is there a handler similar to https://github.com/NixOS/nix/blob/master/src/libstore/filetransfer.cc#L560 in the fetchTree codepath?

arianvp · 2023-12-27T11:08:51Z

performing daemon worker op: 7

I can attest to this.

Nix prints this just before starting to evaluate the downloaded nixpkgs sources. if I hit C-c when this prints the CLI completely locks up and hangs forever.

Otherwise Nix deadlocks when Ctrl-C is received in withFramedSink(): the parent thread will wait forever for the stderr thread to shut down. Fixes the hang reported in NixOS#7245 (comment).

edolstra · 2024-01-04T16:06:12Z

#9687 should fix the hang reported by @aakropotkin.

Otherwise Nix deadlocks when Ctrl-C is received in withFramedSink(): the parent thread will wait forever for the stderr thread to shut down. Fixes the hang reported in #7245 (comment). (cherry picked from commit 24e7048)

9999years · 2024-02-05T20:29:06Z

I can no longer reproduce these issues, I believe the fixes in #6995 and #9687 worked. Please reopen if anyone can find another reproducer.

9999years · 2024-02-23T22:56:43Z

Nevermind, I'm reopening this:

error (ignored): error: interrupted by the user

9999years added the bug label Oct 31, 2022

stale bot added the stale label May 21, 2023

stale bot removed the stale label Jun 5, 2023

roberth added the new-cli Relating to the "nix" command label Jun 5, 2023

RaitoBezarius mentioned this issue Nov 22, 2023

[Tracking Issue] Nix 2.3 parity #9412

Open

edolstra mentioned this issue Jan 3, 2024

withFramedSink(): Receive interrupts on the stderr thread #9687

Merged

9999years closed this as completed Feb 5, 2024

9999years mentioned this issue Feb 5, 2024

Add release notes #9943

Merged

9999years reopened this Feb 23, 2024

9999years mentioned this issue Apr 10, 2024

[2.20] Backport changes to release notes #10459

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various Nix commands ignore Ctrl-C (SIGTERM) #7245

Various Nix commands ignore Ctrl-C (SIGTERM) #7245

9999years commented Oct 31, 2022 •

edited

Loading

matthewbauer commented Oct 31, 2022

9999years commented Oct 31, 2022

nrdxp commented Nov 1, 2022 •

edited

Loading

matthewbauer commented Nov 2, 2022

nrdxp commented Nov 2, 2022 •

edited

Loading

matthewbauer commented Nov 2, 2022

thufschmitt commented Nov 3, 2022

9999years commented Nov 3, 2022

aakropotkin commented Nov 16, 2022 •

edited

Loading

arianvp commented Jun 5, 2023

edolstra commented Jun 5, 2023

arianvp commented Jun 5, 2023

roberth commented Jun 5, 2023

arianvp commented Jun 5, 2023 •

edited

Loading

stephenjudkins commented Jun 6, 2023

Ericson2314 commented Jun 6, 2023

kjeremy commented Aug 11, 2023

arianvp commented Aug 30, 2023

marijanp commented Oct 19, 2023

marijanp commented Oct 19, 2023

marijanp commented Oct 20, 2023

tomberek commented Nov 22, 2023

arianvp commented Dec 27, 2023

edolstra commented Jan 4, 2024

9999years commented Feb 5, 2024

9999years commented Feb 23, 2024

Various Nix commands ignore Ctrl-C (SIGTERM) #7245

Various Nix commands ignore Ctrl-C (SIGTERM) #7245

Comments

9999years commented Oct 31, 2022 • edited Loading

matthewbauer commented Oct 31, 2022

9999years commented Oct 31, 2022

nrdxp commented Nov 1, 2022 • edited Loading

matthewbauer commented Nov 2, 2022

nrdxp commented Nov 2, 2022 • edited Loading

matthewbauer commented Nov 2, 2022

thufschmitt commented Nov 3, 2022

9999years commented Nov 3, 2022

aakropotkin commented Nov 16, 2022 • edited Loading

arianvp commented Jun 5, 2023

edolstra commented Jun 5, 2023

arianvp commented Jun 5, 2023

roberth commented Jun 5, 2023

arianvp commented Jun 5, 2023 • edited Loading

stephenjudkins commented Jun 6, 2023

Ericson2314 commented Jun 6, 2023

kjeremy commented Aug 11, 2023

arianvp commented Aug 30, 2023

marijanp commented Oct 19, 2023

marijanp commented Oct 19, 2023

marijanp commented Oct 20, 2023

tomberek commented Nov 22, 2023

arianvp commented Dec 27, 2023

edolstra commented Jan 4, 2024

9999years commented Feb 5, 2024

9999years commented Feb 23, 2024

9999years commented Oct 31, 2022 •

edited

Loading

nrdxp commented Nov 1, 2022 •

edited

Loading

nrdxp commented Nov 2, 2022 •

edited

Loading

aakropotkin commented Nov 16, 2022 •

edited

Loading

arianvp commented Jun 5, 2023 •

edited

Loading