"File not found" after disk full

[jtyers]

Bug Report

I run concourse via the quick-start method under docker-compose, on a single machine. The machine ran out of disk space on its docker volume, and jobs started failing with no space left on device. I freed up space on the device and restarted concourse (via docker restart <concourse container id>).

Now, whenever I restart any job, I get:

initializing
file not found
errored

The job previously ran without this error, and the job has not been changed since it worked. I noticed that the job begins with a git step, which works without issue, and the next step is a run, which is where file not found occurs.

Steps to Reproduce

• existing pipeline using git resource followed by run
• fill space up on Docker host, until new jobs start failing with no space left on device
• free up space and restart concourse
• new jobs fail with file not found

Expected Results

Any corrupt state caused by the disk filling up should be corrected by concourse transparently so that workers/jobs can execute normally after recovering from this error.

Actual Results

New jobs fail with file not found.

Additional Context

• Concourse 5.0.0

Version Info

• Concourse version: 5.0.0
• Deployment type Docker
• Infrastructure/IaaS: Arch Linux rolling
• Did this used to work? I hit a similar thing under 4.x; clearing the database and re-initialising concourse (adding users and pipelines from scratch) was the only way to clear out corrupt state and get things working again.

[marco-m]

I freed up space on the device

This might be the problem. How exactly ?

[jtyers]

Hmm, hopefully not. I ran docker volume prune while concourse was still running. Note that the "file not found" errors were occurring before I did this; the error message didn't change before/after I ran the prune.

[stale]

Beep boop! This issue has been idle for long enough that it's time to check
in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a
PR or
continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be
exterminated closed, in accordance with our stale issue
process.