[bug] ENOENT: no such file or directory, action hangs forever

[kachkaev]

What happened?

I maintain a public repo which uses hosted GitHub runners and actions/upload-artifact@v3. The action freezes time and again, thus extending job duration to six hours unless someone has cancelled it.

Example 1 (manually cancelled after about 40 minutes):

With the provided path, there will be 15 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "site-integration-tests-var-firefox" successfully created. Starting upload of file(s)
Error: ENOENT: no such file or directory, stat '/home/runner/work/blockprotocol/blockprotocol/var/site/s3/.minio.sys/buckets/.usage-cache.bin/67f9c76c-1cf5-47f9-a43d-401f7702133a/part.1'
Total file count: 15 ---- Processed file #14 (93.3%)
Total file count: 15 ---- Processed file #14 (93.3%)
Total file count: 15 ---- Processed file #14 (93.3%)
....................................................
Total file count: 15 ---- Processed file #14 (93.3%)
Total file count: 15 ---- Processed file #14 (93.3%)
Total file count: 15 ---- Processed file #14 (93.3%)
Error: The operation was canceled.

Example 2 (timeout in 360 minutes):

With the provided path, there will be 49 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "site-integration-tests-var-chrome" successfully created. Starting upload of file(s)
Error: ENOENT: no such file or directory, stat '/home/runner/work/blockprotocol/blockprotocol/var/site/s3/.minio.sys/buckets/.usage-cache.bin/a3b2a0fd-8b64-42f7-803d-7fa690aa8804/part.1'
Total file count: 49 ---- Processed file #48 (97.9%)
Total file count: 49 ---- Processed file #48 (97.9%)
Total file count: 49 ---- Processed file #48 (97.9%)
....................................................
Total file count: 49 ---- Processed file #48 (97.9%)
Total file count: 49 ---- Processed file #48 (97.9%)
Total file count: 49 ---- Processed file #48 (97.9%)
Error: The operation was canceled.

Example 3 (timeout in 360 minutes):

Run actions/upload-artifact@v3
With the provided path, there will be 50 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "site-integration-tests-var-chrome" successfully created. Starting upload of file(s)
Error: ENOENT: no such file or directory, stat '/home/runner/work/blockprotocol/blockprotocol/var/site/s3/.minio.sys/tmp/.trash/051a4f7f-22bf-44f9-b177-4efa70ac4f55'
Total file count: 50 ---- Processed file #13 (26.0%)
Total file count: 50 ---- Processed file #13 (26.0%)
Total file count: 50 ---- Processed file #13 (26.0%)
.......................................................
Total file count: 50 ---- Processed file #13 (26.0%)
Total file count: 50 ---- Processed file #13 (26.0%)
Total file count: 50 ---- Processed file #13 (26.0%)
Error: The operation was canceled.

What did you expect to happen?

I expected action code to track stuck upload request and retry them. If the upload is impossible I’d expect the action to fail within a reasonable amount of time.

How can we reproduce it?

Run actions/upload-artifact@v3 enough times to see it stuck. Examples above link to publicly available runs.

Anything else we need to know?

Glad that this time is not billed for OSS. Folks with private repos have to pay a fortune because of this bug 😥

What version of the action are you using?

v3 (v3.1.0, v3.1.1)

What are your runner environments?

linux

Are you on GitHub Enterprise Server? If so, what version?

No response

[vilkinsons]

In case wiped, dumping the log archives from the above 3x example runs here:

• logs_29659.zip
• logs_29852.zip
• logs_30800.zip

Specifically of interest are the Site integration tests (chrome) runs.

[laeubi]

Could it be related to:

• ENOENT: no such file or directory while attempting to upload #240

[kachkaev]

👋 @laeubi! #240 seems to be related to long file paths on Windows, which is not the case here. I believe that the underlying cause of this bug is a background process that occasionally deletes files during artifact upload. I've stopped this background process as a workaround, but I still believe that the task should not hang for hours if some of the uploaded files go missing.