Abort repo when out of disk space message found in artifact update error #16536

rarkins · 2022-07-12T05:58:43Z

What would you like Renovate to be able to do?

Catch yarn out of disk space errors during branch updates

If you have any ideas on how this should be implemented, please tell us here.

Here's an example "Error updating branch" error:

{
message: 
error: unable to write file .yarn/cache/got-npm-11.8.2-c1eb105458-41ad7bf74f.zip
fatal: sha1 file '/mnt/renovate/gh/org/repo/.git/index.lock' write error. Out of diskspace
, 
stack: 
Error: error: unable to write file .yarn/cache/got-npm-11.8.2-c1eb105458-41ad7bf74f.zip
fatal: sha1 file '/mnt/renovate/gh/org/repo/.git/index.lock' write error. Out of diskspace

    at Object.action (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1194:25)
    at PluginStore.exec (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1229:29)
    at /home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1590:43
    at new Promise (<anonymous>)
    at GitExecutorChain.handleTaskData (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1588:16)
    at GitExecutorChain.<anonymous> (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1572:44)
    at Generator.next (<anonymous>)
    at fulfilled (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:55:24)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
, 
task: {
commands: [Array], 
format: utf-8, 
parser: [function]
}
}

I think if we look for the string write error. Out of diskspace in an artifact error then we should then abort if we find it.

Is this a feature you are interested in implementing yourself?

No

The text was updated successfully, but these errors were encountered:

rarkins · 2022-07-12T06:01:08Z

BTW this will be rather hard to reproduce, so we will need to just code review it instead.

github-actions · 2022-07-12T11:13:07Z

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

PhilipAbed · 2022-07-12T11:14:12Z

our team will try to reproduce in order to get more context

rarkins · 2022-07-12T11:37:12Z

I think a simple and quicker fix based on the log should be fine. Otherwise you would also be contributing to the app's load. It takes a multi GB project to use up the disk

PhilipAbed · 2022-07-12T11:50:05Z

we thought we might be able to reproduce using docker while limiting it's space on self hosted, then we can have better context about what is the Actual vs the Expected results

what i understand for the expected result is : log the error and stop the run?
please Confirm

rarkins · 2022-07-12T11:57:14Z

Yes warn and stop the repo run, instead of continuing to every branch and getting the same error

MaronHatoum · 2022-08-07T14:16:43Z

After trying to reproduce this issue using VM with disk space limitation,
this is the Logs files we got :

DEBUG: lock file error (repository=MaronHatoum/renovate, branch=renovate/jamiemagee-osv-offline-1.x)
       "err": {
         "name": "ExecError",
         "cmd": "/bin/sh -c yarn install --ignore-engines --ignore-platform --network-timeout 100000 --ignore-scripts",
         "stderr": "error https://registry.yarnpkg.com/tunnel/-/tunnel-0.0.6.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: \"ENOSPC: no space left on device, mkdir '/tmp/renovate/cache/others/yarn/v6/npm-tunnel-0.0.6-72f1314b34a5b192db012324df2cc587ca47f92c-integrity/node_modules/tunnel/lib'\"\n",
         "stdout": "yarn install v1.22.19\n[1/4] Resolving packages...\n[2/4] Fetching packages...\ninfo Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/MaronHatoum/renovate",
           "encoding": "utf-8",
           "env": {
             "NPM_CONFIG_CACHE": "/tmp/renovate/cache/others/npm",
             "npm_config_store": "/tmp/renovate/cache/others/pnpm",
             "CI": "true",
             "YARN_CACHE_FOLDER": "/tmp/renovate/cache/others/yarn",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.10.5/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "message": "Process exited with exit code \"1\"",
         "stack": "ExecError: Process exited with exit code \"1\"\n    at ChildProcess.<anonymous> (/usr/src/app/node_modules/renovate/lib/util/exec/common.ts:95:16)\n    at ChildProcess.emit (events.js:412:35)\n    at ChildProcess.emit (domain.js:475:12)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:285:12)"
       },
       "type": "yarn"
DEBUG: Passing disk-space error up (repository=MaronHatoum/renovate, branch=renovate/jamiemagee-osv-offline-1.x)
ERROR: Disk space error - skipping (repository=MaronHatoum/renovate)
DEBUG: Repository result: disk-space, status: onboarded, enabled: true, onboarded: true (repository=MaronHatoum/renovate)

reproduction steps:

create a new ubuntu VM with disk space limitation (20GB)
install docker
pull renovate/renovate:latest image
identify the available space on the machine using df -h command
use the command head -c 2GB /dev/urandom > 2gb.txt to utilize the remaining space keeping ~200MB (in this example we used 2GB extra)
run Renovate docker to scan your forked Renovate repository (which needs 300MB+ space, cloning, and running yarn install)
once the disk space gets full Renovate will print the logs above and exit

rarkins · 2022-08-07T14:39:51Z

It's similar but not the same error

MaronHatoum · 2022-08-08T07:32:59Z

Tried to make it fail during git clone, and I got these Logs:

DEBUG: Performing blobless clone (repository=MaronHatoum/renovate)
DEBUG: Git function thrown (repository=MaronHatoum/renovate)
       "err": {
         "task": {
           "commands": [
             "clone",
             "--filter=blob:none",
             "https://**redacted**@github.com/MaronHatoum/renovate.git",
             "."
           ],
           "format": "utf-8",
           "parser": "[function]"
         },
         "message": "Cloning into '.'...",
         "stack": "Error: Cloning into '.'..."
       }
DEBUG: git clone error (repository=MaronHatoum/renovate)
       "err": {
         "task": {
           "commands": [
             "clone",
             "--filter=blob:none",
             "https://**redacted**@github.com/MaronHatoum/renovate.git",
             "."
           ],
           "format": "utf-8",
           "parser": "[function]"
         },
         "message": "Cloning into '.'...",
         "stack": "Error: Cloning into '.'..."
       }
ERROR: Disk space error - skipping (repository=MaronHatoum/renovate)
DEBUG: Unknown res (repository=MaronHatoum/renovate)
       "res": "disk-space"
DEBUG: Repository result: disk-space, status: unknown, enabled: undefined, onboarded: undefined (repository=MaronHatoum/renovate)
DEBUG: Repository timing splits (milliseconds) (repository=MaronHatoum/renovate)
       "splits": {},
       "total": 12631
 INFO: Repository finished (repository=MaronHatoum/renovate)
       "durationMs": 12631
DEBUG: Renovate exiting
 INFO: Renovate is exiting with a non-zero code due to the following logged errors
       "loggerErrors": [
         {
           "name": "renovate",
           "level": 50,
           "logContext": "json",
           "repository": "MaronHatoum/renovate",
           "msg": "Disk space error - skipping"
         }
       ]

Can you please provide more log context?
should we handle the error you reported and the error I reported in the previous comment or just the error you reported in this issue?

Closes #16536

renovate-release · 2022-08-09T06:48:33Z

🎉 This issue has been resolved in version 32.153.0 🎉

The release is available on:

GitHub release
32.153.0

Your semantic-release bot 📦🚀

rarkins added type:feature Feature (new functionality) priority-2-high Bugs impacting wide number of users or very important features status:ready labels Jul 12, 2022

PhilipAbed added the auto:reproduction A minimal reproduction is necessary to proceed label Jul 12, 2022

nabeelsaabna assigned MaronHatoum Jul 19, 2022

rarkins self-assigned this Aug 8, 2022

rarkins added status:in-progress Someone is working on implementation and removed status:ready labels Aug 8, 2022

rarkins added a commit that referenced this issue Aug 8, 2022

fix(yarn): check stdout for disk space error

0a64fdd

Closes #16536

rarkins mentioned this issue Aug 8, 2022

fix(yarn): check stdout for disk space error #17056

Merged

6 tasks

rarkins closed this as completed in #17056 Aug 9, 2022

rarkins added a commit that referenced this issue Aug 9, 2022

fix(yarn): check stdout for disk space error (#17056)

157c07a

Closes #16536

github-actions bot locked as resolved and limited conversation to collaborators Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Abort repo when out of disk space message found in artifact update error #16536

Abort repo when out of disk space message found in artifact update error #16536

rarkins commented Jul 12, 2022

rarkins commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

PhilipAbed commented Jul 12, 2022

rarkins commented Jul 12, 2022

PhilipAbed commented Jul 12, 2022

rarkins commented Jul 12, 2022

MaronHatoum commented Aug 7, 2022

rarkins commented Aug 7, 2022

MaronHatoum commented Aug 8, 2022

renovate-release commented Aug 9, 2022

Abort repo when out of disk space message found in artifact update error #16536

Abort repo when out of disk space message found in artifact update error #16536

Comments

rarkins commented Jul 12, 2022

What would you like Renovate to be able to do?

If you have any ideas on how this should be implemented, please tell us here.

Is this a feature you are interested in implementing yourself?

rarkins commented Jul 12, 2022

github-actions bot commented Jul 12, 2022

PhilipAbed commented Jul 12, 2022

rarkins commented Jul 12, 2022

PhilipAbed commented Jul 12, 2022

rarkins commented Jul 12, 2022

MaronHatoum commented Aug 7, 2022

rarkins commented Aug 7, 2022

MaronHatoum commented Aug 8, 2022

renovate-release commented Aug 9, 2022