repology timeout error

[wrighbr]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

bitbucket

Please tell us more about your question or problem

After upgrading to Renovate v38, I'm encountering a timeout error for the Repology host. I previously used the workarounds:reduceRepologyServerLoad preset, which helped mitigate rate limiting in version 37 of Renovate. However, I see that this preset has been removed in v38.

Logs (if relevant)

Logs

{
 "name": "RequestError",
 "code": "ETIMEDOUT",
 "timings": {
  "start": 1728594158359,
  "socket": 1728594158359,
  "lookup": 1728594158361,
  "error": 1728594158617,
  "phases": {
   "wait": 0,
   "dns": 2,
   "total": 258
  }
 },
 "stack": "RequestError: \n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/got@11.8.6/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:634:26)\n    at ClientRequest.emit (node:events:531:35)\n    at ClientRequest.emit (node:domain:488:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@szmarczak+http-timer@4.0.6/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at emitErrorEvent (node:_http_client:101:11)\n    at TLSSocket.socketErrorListener (node:_http_client:504:5)\n    at TLSSocket.emit (node:events:519:28)\n    at TLSSocket.emit (node:domain:488:12)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)AggregateError: \n    at internalConnectMultiple (node:net:1118:18)\n    at internalConnectMultiple (node:net:1186:5)\n    at Timeout.internalConnectMultipleTimeout (node:net:1712:5)\n    at listOnTimeout (node:internal/timers:583:11)\n    at processTimers (node:internal/timers:519:7)",
 "options": {
  "headers": {
   "user-agent": "RenovateBot/38.110.2 (https://github.com/renovatebot/renovate)",
   "accept": "application/json",
   "accept-encoding": "gzip, deflate, br"
  },
  "url": "https://repology.org/tools/project-by?repo=debian_12&name_type=binname&target_page=api_v1_project&noautoresolve=on&name=chromium",
  "hostType": "repology",
  "username": "",
  "password": "",
  "method": "GET",
  "http2": false
 }
}

[gnuletik]

I'm also getting these errors using the SAAS version of renovate.

{
  "hostType": "repology"
  "packageName": "debian_unstable/gvm"
  "err": {
    "name": "RequestError",
    "code": "ETIMEDOUT",
    "timings": {
      "start": 1735829603744,
      "socket": 1735829603745,
      "lookup": 1735829604291,
      "error": 1735829608732,
      "phases": {
        "wait": 1,
        "dns": 546,
        "total": 4988
      }
    },
    "stack": "RequestError: \n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/got@11.8.6/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:633:26)\n    at ClientRequest.emit (node:events:530:35)\n    at ClientRequest.emit (node:domain:489:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@szmarczak+http-timer@4.0.6/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at emitErrorEvent (node:_http_client:103:11)\n    at TLSSocket.socketErrorListener (node:_http_client:506:5)\n    at TLSSocket.emit (node:events:518:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:170:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n    at processTicksAndRejections (node:internal/process/task_queues:90:21)\n    at runNextTicks (node:internal/process/task_queues:69:3)\n    at processTimers (node:internal/timers:526:9)AggregateError: \n    at internalConnectMultiple (node:net:1121:18)\n    at internalConnectMultiple (node:net:1189:5)\n    at Timeout.internalConnectMultipleTimeout (node:net:1715:5)\n    at listOnTimeout (node:internal/timers:596:11)\n    at processTimers (node:internal/timers:529:7)",
    "options": {
      "headers": {
        "user-agent": "RenovateBot/39.85.0 (https://github.com/renovatebot/renovate)",
        "accept": "application/json",
        "accept-encoding": "gzip, deflate, br"
      },
      "url": "https://repology.org/tools/project-by?repo=debian_unstable&name_type=binname&target_page=api_v1_project&noautoresolve=on&name=gvm",
      "hostType": "repology",
      "username": "",
      "password": "",
      "method": "GET",
      "http2": false
    }
  }
}

I've tried extending the timeout value to 60 seconds without success.
What's quite surprising is that the request times out after 5 seconds according to logs.

I've also found a similar issue on repology's repo: repology/repology-rs#143
Repology's author, @AMDmi3 stated that there was connectivity issues from GCP.

As far as I know, renovate SAAS is hosted on AWS.
Maybe there's a similar issue on AWS?

[gnuletik]

I gave it another try and got the same result:

DEBUG: GET https://repology.org/tools/project-by?repo=debian_unstable&name_type=binname&target_page=api_v1_project&noautoresolve=on&name=gvm = (code=ETIMEDOUT, statusCode=-1 retryCount=2, duration=3537)
WARN: Repology lookup failed with unexpected error
{
  "packageName": "debian_unstable/gvm"
  "err": {
    "name": "RequestError",
    "code": "ETIMEDOUT",
    "timings": {
      "start": 1736957713204,
      "socket": 1736957713205,
      "lookup": 1736957713231,
      "error": 1736957716741,
      "phases": {
        "wait": 1,
        "dns": 26,
        "total": 3537
      }
    },
    "stack": "RequestError: \n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/got@11.8.6/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:639:26)\n    at ClientRequest.emit (node:events:536:35)\n    at ClientRequest.emit (node:domain:489:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@szmarczak+http-timer@4.0.6/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at emitErrorEvent (node:_http_client:104:11)\n    at TLSSocket.socketErrorListener (node:_http_client:518:5)\n    at TLSSocket.emit (node:events:524:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:170:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n    at processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError: \n    at internalConnectMultiple (node:net:1139:18)\n    at internalConnectMultiple (node:net:1215:5)\n    at Timeout.internalConnectMultipleTimeout (node:net:1739:5)\n    at listOnTimeout (node:internal/timers:596:11)\n    at processTimers (node:internal/timers:529:7)",
    "options": {
      "headers": {
        "user-agent": "RenovateBot/39.107.0 (https://github.com/renovatebot/renovate)",
        "accept": "application/json",
        "accept-encoding": "gzip, deflate, br"
      },
      "url": "https://repology.org/tools/project-by?repo=debian_unstable&name_type=binname&target_page=api_v1_project&noautoresolve=on&name=gvm",
      "hostType": "repology",
      "username": "",
      "password": "",
      "method": "GET",
      "http2": false
    }
  }
}

Job ID is 40705668-0b8a-4b74-8900-57924ea457e7.

Do you have a clue on how I can help fixing this @rarkins?
Also, do you know why this issue makes renovate fail in a way that it stops processing other dependencies?

[rarkins]

Your logs are generated from this location:

renovate/lib/modules/datasource/repology/index.ts

Lines 223 to 230 in e9bc921

	} else {
	logger.warn(
	{ packageName, err },
	'Repology lookup failed with unexpected error',
	);
	}
	throw new ExternalHostError(err);

The repology datasource throws an external host error to intentionally abort a job from continuing. This is because repology was a datasource which "goes down" frequently and if Renovate continued, it could cause a lot of noise by autoclosing and/or reopening PRs due to temporary errors.

An ETIMEDOUT error is most likely due to blocking, so I'm not sure what we can do about it. The only thing you can do in the meantime is to disable all repology datasources, or switch to self-hosting if you're not blocked directly (yet).

[gnuletik]

I've created an issue here to investigate the issue with Repology's author:
repology/repology-rs#151

Thanks for the fast feedback!

[gnuletik]

Is there a way to find the public IP address used by each renovate run?

Repology's author needs this info in order to help debugging:
repology/repology-rs#151 (comment)

[rarkins]

No, jobs run on an inner VM and don't know their external IP

[github-actions[bot]]

Hi there,

This is intended as a polite, automated request that users avoid @ mentioning repository maintainers like @rarkins or @viceice. Doing so causes annoying mobile notifications and makes it harder to maintain this repository.

We know it might be common elsewhere but we participate in hundreds of discussions a week and would need to turn off GitHub mobile notifications if we were mentioned in every one.

As a general rule, we will read and respond to all discussions in this repository, so there is no need to mention us.

Thanks, the Renovate team