Skip to content

Commit

Permalink
Move pseudoterminal slave open to child
Browse files Browse the repository at this point in the history
Hopefully this fixes "unexpected EOF" failures on macOS
(NixOS#3137, NixOS#3605, NixOS#7242, NixOS#7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.

(cherry picked from commit c536e00)
  • Loading branch information
edolstra authored and Ericson2314 committed Oct 31, 2023
1 parent 791af78 commit d2f4b8d
Showing 1 changed file with 27 additions and 25 deletions.
52 changes: 27 additions & 25 deletions src/libstore/build.cc
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,10 @@ class DerivationGoal : public Goal
/* Pipe for the builder's standard output/error. */
Pipe builderOut;

/* Slave side of the pseudoterminal used for the builder's
standard output/error. */
Path slaveName;

/* Pipe for synchronising updates to the builder user namespace. */
Pipe userNamespaceSync;

Expand Down Expand Up @@ -2228,14 +2232,12 @@ void DerivationGoal::startBuilder()
/* Create the log file. */
Path logFile = openLogFile();

/* Create a pipe to get the output of the builder. */
//builderOut.create();

/* Create a pseudoterminal to get the output of the builder. */
builderOut.readSide = posix_openpt(O_RDWR | O_NOCTTY);
if (!builderOut.readSide)
throw SysError("opening pseudoterminal master");

std::string slaveName(ptsname(builderOut.readSide.get()));
slaveName = ptsname(builderOut.readSide.get());

if (buildUser) {
if (chmod(slaveName.c_str(), 0600))
Expand All @@ -2248,30 +2250,9 @@ void DerivationGoal::startBuilder()
throw SysError("granting access to pseudoterminal slave");
}

#if 0
// Mount the pt in the sandbox so that the "tty" command works.
// FIXME: this doesn't work with the new devpts in the sandbox.
if (useChroot)
dirsInChroot[slaveName] = {slaveName, false};
#endif

if (unlockpt(builderOut.readSide.get()))
throw SysError("unlocking pseudoterminal");

builderOut.writeSide = open(slaveName.c_str(), O_RDWR | O_NOCTTY);
if (!builderOut.writeSide)
throw SysError("opening pseudoterminal slave");

// Put the pt into raw mode to prevent \n -> \r\n translation.
struct termios term;
if (tcgetattr(builderOut.writeSide.get(), &term))
throw SysError("getting pseudoterminal attributes");

cfmakeraw(&term);

if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
throw SysError("putting pseudoterminal into raw mode");

result.startTime = time(0);

/* Fork a child to build the package. */
Expand Down Expand Up @@ -2320,7 +2301,11 @@ void DerivationGoal::startBuilder()

options.allowVfork = false;

Pipe sendPid;
sendPid.create();

Pid helper = startProcess([&]() {
sendPid.readSide.close();

/* Drop additional groups here because we can't do it
after we've created the new user namespace. FIXME:
Expand Down Expand Up @@ -2368,6 +2353,8 @@ void DerivationGoal::startBuilder()
_exit(0);
}, options);

sendPid.writeSide.close();

int res = helper.wait();
if (res != 0 && settings.sandboxFallback) {
useChroot = false;
Expand Down Expand Up @@ -2724,6 +2711,21 @@ void DerivationGoal::runChild()

try { /* child */

/* Open the slave side of the pseudoterminal. */
builderOut.writeSide = open(slaveName.c_str(), O_RDWR | O_NOCTTY);
if (!builderOut.writeSide)
throw SysError("opening pseudoterminal slave");

// Put the pt into raw mode to prevent \n -> \r\n translation.
struct termios term;
if (tcgetattr(builderOut.writeSide.get(), &term))
throw SysError("getting pseudoterminal attributes");

cfmakeraw(&term);

if (tcsetattr(builderOut.writeSide.get(), TCSANOW, &term))
throw SysError("putting pseudoterminal into raw mode");

commonChildInit(builderOut);

try {
Expand Down

0 comments on commit d2f4b8d

Please sign in to comment.