Python may abort if unable to completely write a .pyc file in one go

[stefanor]

Bug report

Bug description:

#126606 fixed a bug, in that writing incomplete .pyc files is not going to end well for the system.
But the solution was just to crash the file write, not attempt any retries.

It's not completely unexpected to receive a signal or some other interrupt that results in write() not writing all the provided data. This can be handled.

We've observed incomplete .pyc files in the past in Debian systems, and I just ran into this in a CI job:

https://people.debian.org/~stefanor/upstream/debusine-server_0.13.1.log

  Setting up python3-debusine-server (0.13.1) ...
  os.write() didn't write the full pyc filedpkg: error processing package python3-debusine-server (--configure):
   installed python3-debusine-server package post-installation script subprocess returned error exit status 1

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-141930: Retry writing .pyc files if incomplete #141931

[cmaloney]

EINTR in specific should already be retried when possible (see: PEP-475 and _Py_write which does the underlying I/O).

If this needs a full well-tested retry loop I'd strongly prefer moving this to use BufferedIO rather than FileIO directly. Yes, it has slightly more overheads but re-implementing "handle all buffering and retry cases" here doesn't seem worth the complexity.

[stefanor]

Agreed that writing another retry loop is nasty.

I had missed that retry bit in _Py_write (and misled by the outdated comments that #129011 aims to fix). If it's not EINTR, I wonder why this case was failing... We'll probably find out in time if we implement something better, because the real error will bubble up, rather than the generic os.write() didn't write the full pyc file that masks the error.

[cmaloney]

Is the process running out of quota/space? It looks like the host is using a new systemd and in that there's a recent change to do quotas on tmp space. My thought here is gh-126606 the underlying cause of the partial write was a file size limit imposed by ulimit so maybe similar.

[stefanor]

Writing pycache shouldn't go to /tmp, but I'm going to check with the admin of that service if they saw anything.

Munin graphs for the machine that ran the task don't show any disk issues at that time.

[stefanor]

Note that EINTR only applies when 0 bytes were written, if the kernel is interrupted after writing some bytes, it may just return the number of bytes written. signal(7) says a local disk is not a "slow" device that could exhibit this behaviour... but I/O stacks are complex, and this is obviously a massive simplification.

[stefanor]

I'm going to check with the admin of that service if they saw anything.

They found another instance of this in the log pile: https://piuparts.debian.org/oldstable22testing/fail/tsung_None.log

Filesystem in question is tmpfs. The admin doesn't think ENOSPC is likely (or they'd see it more often), but I don't think we can rule it out.

Just getting Python to print the errno error here would be a win for debugging these problems :)

[brettcannon]

Is this that important to succeed at the write in the end? .pyc files are an optimization, so if they don't end up being written then it shouldn't matter that much. Plus the incomplete file should be deleted, so there's shouldn't be any trash left over.

[stefanor]

Yeah, at least the write is atomic, so we shouldn't be left with broken .pyc files, as we were in the past. (I used to see numerous bug reports from broken systems due to corrupt .pyc files).

If the intention is that failing to write .pyc files is OK, then maybe py_compile should raise a warning and exit 0, when one of these is hit?