abortOnError:false not working

[bprachi29]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

latest

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Describe the bug

I have added "abortOnError: false" for npm host type, but the renovate bot is still exiting with the error External host error causing abort - skipping

Relevant debug logs

Logs

WARN: Host error (repository=prachi/private_repo)
        "hostType": "npm",
        "lookupName": "grunt-contrib-less",
        "err": {
          "name": "RequestError",
          "code": "ECONNRESET",
          "timings": {
            "start": 1641889113437,
            "socket": 1641889113437,
            "lookup": 1641889113532,
            "connect": 1641889113567,
            "error": 1641889113595,
            "phases": {"wait": 0, "dns": 95, "tcp": 35, "total": 158}
          },
          "message": "read ECONNRESET",
          "stack": "RequestError: read ECONNRESET\n    at ClientRequest.<anonymous> (/usr/src/app/node_modules/got/dist/source/core/index.js:962:111)\n    at Object.onceWrapper (events.js:520:26)\n    at ClientRequest.emit (events.js:412:35)\n    at ClientRequest.emit (domain.js:475:12)\n    at ClientRequest.origin.emit (/usr/src/app/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at TLSSocket.socketErrorListener (_http_client.js:475:9)\n    at TLSSocket.emit (events.js:400:28)\n    at TLSSocket.emit (domain.js:475:12)\n    at emitErrorNT (internal/streams/destroy.js:106:8)\n    at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21)\n    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)",
          "options": {
            "headers": {
              "user-agent": "RenovateBot/31.14.0 (https://github.com/renovatebot/renovate)",
              "accept": "application/json",
              "accept-encoding": "gzip, deflate, br"
            },
            "url": "https://registry.npmjs.org/grunt-contrib-less",
            "hostType": "npm",
            "username": "",
            "password": "",
            "method": "GET",
            "http2": false
          }
        }
  INFO: External host error causing abort - skipping (repository=prachi/private_repo)
  INFO: Repository finished (repository=prachi/private_repo)
        "durationMs": 277328

Have you created a minimal reproduction repository?

Repo link (https://github.com/bprachi29/testRepo1)

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[bprachi29]

Created the minimal reproduction repo - https://github.com/bprachi29/testRepo1

[rarkins]

Thanks, could you edit the readme there to describe what it is aimed to demonstrate, e.g. expected behavior vs observed behavior? Also attach relevant debug logs

[bprachi29]

Updated the readme file

[rarkins]

Please reduce the repo to minimal (I assume that you don't need 10 or more dependencies to demonstrate it?) plus format the logs in the readme so that they're readable.

[bprachi29]

Please reduce the repo to minimal (I assume that you don't need 10 or more dependencies to demonstrate it?) plus format the logs in the readme so that they're readable.

Hey.. I have removed most of the dependencies now.. but we are facing issues with the grunt libraries, so kept all of them. Formatted the logs as well.

[jord1e]

Also happening for me, when a Docker image is not present everything fails (even when abortOnError: false is set):

 WARN: docker registry failure: unauthorized (repository=xxx/xxx)
       "registryHost": "https://index.docker.io",
       "dockerRepository": "library/unknown_image",
       "err": {
         "name": "HTTPError",
         "code": "ERR_NON_2XX_3XX_RESPONSE",
         "timings": {
           "start": 1643393222490,
           "socket": 1643393222490,
           "lookup": 1643393222491,
           "connect": 1643393222516,
           "secureConnect": 1643393222613,
           "upload": 1643393222614,
           "response": 1643393222642,
           "end": 1643393222642,
           "phases": {
             "wait": 0,
             "dns": 1,
             "tcp": 25,
             "tls": 97,
             "request": 1,
             "firstByte": 28,
             "download": 0,
             "total": 152
           }
         },
         "message": "Response code 401 (Unauthorized)",
         "stack": "HTTPError: Response code 401 (Unauthorized)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/31.49.0 (https://github.com/renovatebot/renovate)",
             "authorization": "***********",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://index.docker.io/v2/library/unknown_image/tags/list?n=10000",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         },
         "response": {
           "statusCode": 401,
           "statusMessage": "Unauthorized",
           "body": {
             "errors": [
               {
                 "code": "UNAUTHORIZED",
                 "message": "authentication required",
                 "detail": [
                   {
                     "Type": "repository",
                     "Class": "",
                     "Name": "library/unknown_image",
                     "Action": "pull"
                   }
                 ]
               }
             ]
           },
           "headers": {
             "content-type": "application/json",
             "docker-distribution-api-version": "registry/2.0",
             "www-authenticate": "Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/unknown_image:pull\",error=\"insufficient_scope\"",
             "date": "Fri, 28 Jan 2022 18:07:02 GMT",
             "content-length": "184",
             "strict-transport-security": "max-age=31536000",
             "connection": "close"
           },
           "httpVersion": "1.1"
         }
       }
 WARN: Host error (repository=xxx/xxx)
       "hostType": "docker",
       "lookupName": "unknown_image",
       "err": {
         "name": "HTTPError",
         "code": "ERR_NON_2XX_3XX_RESPONSE",
         "timings": {
           "start": 1643393222490,
           "socket": 1643393222490,
           "lookup": 1643393222491,
           "connect": 1643393222516,
           "secureConnect": 1643393222613,
           "upload": 1643393222614,
           "response": 1643393222642,
           "end": 1643393222642,
           "phases": {
             "wait": 0,
             "dns": 1,
             "tcp": 25,
             "tls": 97,
             "request": 1,
             "firstByte": 28,
             "download": 0,
             "total": 152
           }
         },
         "message": "Response code 401 (Unauthorized)",
         "stack": "HTTPError: Response code 401 (Unauthorized)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/31.49.0 (https://github.com/renovatebot/renovate)",
             "authorization": "***********",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://index.docker.io/v2/library/unknown_image/tags/list?n=10000",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         },
         "response": {
           "statusCode": 401,
           "statusMessage": "Unauthorized",
           "body": {
             "errors": [
               {
                 "code": "UNAUTHORIZED",
                 "message": "authentication required",
                 "detail": [
                   {
                     "Type": "repository",
                     "Class": "",
                     "Name": "library/unknown_image",
                     "Action": "pull"
                   }
                 ]
               }
             ]
           },
           "headers": {
             "content-type": "application/json",
             "docker-distribution-api-version": "registry/2.0",
             "www-authenticate": "Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/unknown_image:pull\",error=\"insufficient_scope\"",
             "date": "Fri, 28 Jan 2022 18:07:02 GMT",
             "content-length": "184",
             "strict-transport-security": "max-age=31536000",
             "connection": "close"
           },
           "httpVersion": "1.1"
         }
       }
 INFO: External host error causing abort - skipping (repository=xxx/xxx)
 INFO: Repository finished (repository=xxx/xxx)
       "durationMs": 4676

[MaronHatoum]

Hi,
would you assign this bug to me, please?

[MaronHatoum]

Hi
after investigating the issue here I noticed a few things:

1. the flag abortOnError is implemented inside the HTTP calling method.
2. the docker/npm and others create a new ExternalHostError which is not returned by the HTTP call.

Because as I see in the code, it doesn't handle the abortOnError flag when throwing the exception instead the code throws new exception
without handling the flag, and since it doesn't come from http, the flag isn't working correctly.
should I move the handling to the general HandleError somehow with a refactor? or should I just add a check for the flag to these managers?
there might be other managers with the same issue.

The code parts that I'm talking about has been attached below.

Docker:

renovate/lib/datasource/docker/index.ts

Lines 171 to 179 in 98b20ad

	if (err.name === 'RequestError' && isDockerHost(registryHost)) {
	throw new ExternalHostError(err);
	}
	if (err.statusCode === 429 && isDockerHost(registryHost)) {
	throw new ExternalHostError(err);
	}
	if (err.statusCode >= 500 && err.statusCode < 600) {
	throw new ExternalHostError(err);
	}

Npm :

renovate/lib/datasource/npm/get.ts

Lines 199 to 207 in 98b20ad

	if (uri.host === 'registry.npmjs.org') {
	// istanbul ignore if
	if (err.name === 'ParseError' && err.body) {
	err.body = 'err.body deleted by Renovate';
	}
	throw new ExternalHostError(err);
	}
	return null;
	}

Thanks

[rarkins]

Thanks. What I think this means it that for Docker and npm, we are not aborting on every error and instead specific ones. Therefore it's not the HTTP layer (which understands abortOnError) but instead the datasource itself, which doesn't understand abortOnError. I'm not sure of the best solution for this, because we also don't want to embed Docker/npm awareness into our http layer.

[edude03]

+1, having this issue with docker. Maybe for my particular issue a fix could be ignoring "dockerRepository": "library/null" since there is no package to update anyway.

[gberche-orange]

Crosslinking discussion with same problem observed with docker #17794

[rarkins]

FYI the workaround for this is to add the affected libraries to ignoreDeps - or create a package rule which disables them - so that they're not looked up in the first place.

[gberche-orange]

FYI the workaround for this is to add the affected libraries to ignoreDeps - or create a package rule which disables them - so that they're not looked up in the first place.

@rarkins I encountered again this issue due to the fact that the dockerhub image prometheus/prometheus has migrated to prom/prometheus and is now erroring. This suddently broke the renovate builds for our project without us noticing immediately. As a result, we were not receiving any renovate PRs or rebases from the renovate SaaS instance.

As a workaround for this issue, is there a known approach to receive notifications for failed SaaS renovate builds for more than a given duration ? Could a prometheus metrics endpoint exposing metrics per project allow users to handle their own monitoring and alerting ?

    INFO: External host error causing abort - skipping
    DEBUG: Repository result: external-host-error, status: onboarded, enabled: true, onboarded: true
     "url": "https://index.docker.io/v2/prometheus/prometheus/tags/list?n=10000",
    "Response code 401 (Unauthorized)
    "hostType": "docker",  "packageName": "prometheus/prometheus",

https://hub.docker.com/r/prometheus/prometheus 404
https://hub.docker.com/r/prom/prometheus 200 OK

[gberche-orange]

Could a prometheus metrics endpoint exposing metrics per project allow users to handle their own monitoring and alerting ?

Sorry, already tracked into #17661

[renovate-release]

🎉 This issue has been resolved in version 37.267.1 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)
• 37.267.1

Your semantic-release bot 📦🚀