Abort (assertion failure) in nix::daemon::TunnelLogger::enqueueMsg

[the-sun-will-rise-tomorrow]

Describe the bug

I saw this when trying to build a derivation:

error: Nix daemon disconnected unexpectedly (maybe it crashed?)

Looking at the system journal, I see that the daemon exited with SIGABRT. Here is the demangled stack trace from the journal:

#0  0x00007fe506c9916c __pthread_kill_implementation (libc.so.6 + 0x9916c)
#1  0x00007fe506c40e86 raise (libc.so.6 + 0x40e86)
#2  0x00007fe506c2893a abort (libc.so.6 + 0x2893a)
#3  0x00007fe5070adc3d __gnu_cxx::__verbose_terminate_handler() [clone .cold] (libstdc++.so.6 + 0xadc3d)
#4  0x00007fe5070bf20a __cxxabiv1::__terminate(void (*)()) (libstdc++.so.6 + 0xbf20a)
#5  0x00007fe5070ad7e9 std::terminate() (libstdc++.so.6 + 0xad7e9)
#6  0x00007fe5070ad800 __cxa_rethrow.cold (libstdc++.so.6 + 0xad800)
#7  0x00007fe5078a2ab1 nix::daemon::TunnelLogger::enqueueMsg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .cold] (libnixstore.so + 0xa2ab1)
#8  0x00007fe507955e66 nix::daemon::TunnelLogger::log(nix::Verbosity, std::basic_string_view<char, std::char_traits<char> >) (libnixstore.so + 0x155e66)
#9  0x00007fe507d508dc nix::ignoreExceptionExceptInterrupt(nix::Verbosity) (libnixutil.so + 0x608dc)
#10 0x00007fe507d4dfe9 nix::ThreadPool::doWork(bool) [clone .cold] (libnixutil.so + 0x5dfe9)
#11 0x00007fe5070ed0a4 execute_native_thread_routine (libstdc++.so.6 + 0xed0a4)
#12 0x00007fe506c972e3 start_thread (libc.so.6 + 0x972e3)
#13 0x00007fe506d1b2fc __clone3 (libc.so.6 + 0x11b2fc)

Stack trace of thread 2286721:
#0  0x00007fe506c9381e __futex_abstimed_wait_common (libc.so.6 + 0x9381e)
#1  0x00007fe506c96490 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x96490)
#2  0x00007fe507a018fd nix::Pool<nix::RemoteStore::Connection>::get() (libnixstore.so + 0x2018fd)
#3  0x00007fe507a02261 nix::RemoteStore::getConnection() (libnixstore.so + 0x202261)
#4  0x00007fe507a093ad nix::RemoteStore::queryRealisationUncached(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) (libnixstore.so + 0x2093ad)
#5  0x00007fe507a303fc nix::Store::queryRealisation(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) (libnixstore.so + 0x2303fc)
#6  0x00007fe507a30a87 nix::Store::queryRealisation(nix::DrvOutput const&) (libnixstore.so + 0x230a87)
#7  0x00007fe5079e1268 std::_Function_handler<void (nix::DerivedPath), nix::Store::queryMissing(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, unsigned long&, unsigned long&)::{lambda(nix::DerivedPath const&)#1}>::_M_invoke(std::_Any_data const&, nix::DerivedPath&&) [clone .lto_priv.0] (libnixstore.so + 0x1e1268)
#8  0x00007fe5079d91ce std::_Function_handler<void (), std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPathBuilt)> >::_M_invoke(std::_Any_data const&) (libnixstore.so + 0x1d91ce)
#9  0x00007fe507dd735b nix::ThreadPool::doWork(bool) (libnixutil.so + 0xe735b)
#10 0x00007fe5070ed0a4 execute_native_thread_routine (libstdc++.so.6 + 0xed0a4)
#11 0x00007fe506c972e3 start_thread (libc.so.6 + 0x972e3)
#12 0x00007fe506d1b2fc __clone3 (libc.so.6 + 0x11b2fc)

Stack trace of thread 2286482:
#0  0x00007fe506c9381e __futex_abstimed_wait_common (libc.so.6 + 0x9381e)
#1  0x00007fe506c98ed3 __pthread_clockjoin_ex (libc.so.6 + 0x98ed3)
#2  0x00007fe5070ed11b std::thread::join() (libstdc++.so.6 + 0xed11b)
#3  0x00007fe507dd3798 nix::ThreadPool::shutdown() (libnixutil.so + 0xe3798)
#4  0x00007fe507d4e171 nix::ThreadPool::process() [clone .cold] (libnixutil.so + 0x5e171)
#5  0x00007fe5079dc351 nix::Store::queryMissing(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, unsigned long&, unsigned long&) (libnixstore.so + 0x1dc351)
#6  0x00007fe507959f6e nix::daemon::performOp(nix::daemon::TunnelLogger*, nix::ref<nix::Store>, nix::TrustedFlag, nix::daemon::RecursiveFlag, nix::WorkerProto::BasicServerConnection&, nix::WorkerProto::Op) (libnixstore.so + 0x159f6e)
#7  0x00007fe50795e9a5 nix::daemon::processConnection(nix::ref<nix::Store>, nix::FdSource&&, nix::FdSink&&, nix::TrustedFlag, nix::daemon::RecursiveFlag) (libnixstore.so + 0x15e9a5)
#8  0x000000000053cf8e n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64

Steps To Reproduce

Unfortunately I don't have a reproducer right now. It happens for me after I:

1. Enable extra-experimental-features = ca-derivations
2. Try to build an internal package with importing nixpkgs with config.contentAddressedByDefault = true;.

Though, looking at the stack trace, CA-derivations might be unrelated to this crash.

Expected behavior

No crash.

Metadata

nix-env (Nix) 2.26.3

Additional context

I noticed a similar crash was reported in an unrelated comment: #3377 (comment)

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[the-sun-will-rise-tomorrow]

I don't know how reproducible it is for others, but I noticed that this also happens if I run nix --extra-experimental-features 'nix-command flakes' develop in a checkout of this repository (NixOS/nix) with extra-experimental-features = ca-derivations enabled.

[edolstra]

nix::Store::queryRealisation() is related to CA derivations.

[the-sun-will-rise-tomorrow]

Does this patch make sense? It doesn't fix the underlying problem, but seems to fix the assertion failure.

diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc
index 8f7514273..e069ce4ec 100644
--- a/src/libstore/daemon.cc
+++ b/src/libstore/daemon.cc
@@ -105,12 +105,20 @@ struct TunnelLogger : public Logger
         auto state(state_.lock());
         state->canSendStderr = true;
 
-        for (auto & msg : state->pendingMsgs)
-            to(msg);
-
-        state->pendingMsgs.clear();
-
-        to.flush();
+        try {
+            for (auto & msg : state->pendingMsgs)
+                to(msg);
+
+            state->pendingMsgs.clear();
+
+            to.flush();
+        } catch (...) {
+            /* Write failed; that means that the other side is
+               gone. */
+            state->canSendStderr = false;
+            state->pendingMsgs.clear();
+            throw;
+        }
     }
 
     /* stopWork() means that we're done; stop sending stderr to the

Underlying problem is likely closer to:

nix-daemon: src/libstore/sqlite.cc:135: nix::SQLiteStmt::Use::Use(nix::SQLiteStmt&): Assertion `stmt.stmt' failed.

which was on another machine (accessed by SSH).

[the-sun-will-rise-tomorrow]

Underlying problem is likely closer to:

nix-daemon: src/libstore/sqlite.cc:135: nix::SQLiteStmt::Use::Use(nix::SQLiteStmt&): Assertion `stmt.stmt' failed.

I am not sure but I think this may help with that:

diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc
index f1cf466a6..e86666e91 100644
--- a/src/libstore/local-store.cc
+++ b/src/libstore/local-store.cc
@@ -1641,6 +1641,7 @@ std::optional<const Realisation> LocalStore::queryRealisation_(
 void LocalStore::queryRealisationUncached(const DrvOutput & id,
         Callback<std::shared_ptr<const Realisation>> callback) noexcept
 {
+    experimentalFeatureSettings.require(Xp::CaDerivations);
     try {
         auto maybeRealisation
             = retrySQLite<std::optional<const Realisation>>([&]() {

I think the crash happens because Nix is trying to use a SQLite prepared statement that is only initialized when content-addressed derivations are enabled, but on a path that doesn't check if they're actually enabled. Looking at the code, this seems to be the only remaining path that doesn't have a check on it.

[Ericson2314]

Good catch, thanks!