File match scan got slow since version #19937

mszalbach · 2023-01-20T09:45:03Z

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.106.0

If you're self-hosting Renovate, select which platform you are using.

Renovate full image on Linux Docker against a Bitbucket Server

If you're self-hosting Renovate, tell us what version of the platform you run.

Renovate image: renovate/renovate:34.106.0
Docker version 20.10.23, build 7155243
BitbucketServer 7.13.7

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

Renovate got slow while scanning for file match. This change happend somewhere between version 34.100.2(trying to pinpoint the excact version) and 34.102.7. It is still slow with the version 34.106.0.

From Tests with github platform against a glassfish fork I would say the problem starts occuring with renovate/renovate:34.102.6

For a single project the time for all file match checks increased from around 30 seconds to 3 minutes.

Relevant debug logs

Logs

[2023-01-20T09:23:25.257Z] DEBUG: Using RE2 as regex engine
[2023-01-20T09:23:29.453Z] DEBUG: Parsing configs
[2023-01-20T09:23:29.453Z] DEBUG: Checking for config file in /usr/src/app/config.js
[2023-01-20T09:23:29.453Z] DEBUG: File config
[2023-01-20T09:23:29.453Z]        "config": {
[2023-01-20T09:23:29.453Z]          "platform": "bitbucket-server",
[2023-01-20T09:23:29.453Z]          "endpoint": "https://git.ivu-ag.com",
[2023-01-20T09:23:29.453Z]          "onboardingConfig": {"extends": ["config:base"]},
[2023-01-20T09:23:29.453Z]          "onboarding": true,
[2023-01-20T09:23:29.453Z]          "allowedPostUpgradeCommands": [".*"],
[2023-01-20T09:23:29.453Z]          "regexManagers": [
[2023-01-20T09:23:29.453Z]            {
[2023-01-20T09:23:29.453Z]              "fileMatch": ["^\\.copier-version$"],
[2023-01-20T09:23:29.453Z]              "matchStrings": [
[2023-01-20T09:23:29.453Z]                "\\s+(?<currentDigest>\\S+)\\s*#\\s*datasource=(?<datasource>\\S+)\\s+template=(?<template>\\S+)\\s*\\n"
[2023-01-20T09:23:29.453Z]              ],
[2023-01-20T09:23:29.453Z]              "depNameTemplate": "https://git.ivu-ag.com/scm/{{{template}}}.git"
[2023-01-20T09:23:29.453Z]            }
[2023-01-20T09:23:29.453Z]          ],
[2023-01-20T09:23:29.453Z]          "packageRules": [
[2023-01-20T09:23:29.453Z]            {
[2023-01-20T09:23:29.453Z]              "matchDatasources": ["maven"],
[2023-01-20T09:23:29.453Z]              "registryUrls": ["http://ac-nexus.ivu-ag.com/content/groups/public/"]
[2023-01-20T09:23:29.453Z]            },
[2023-01-20T09:23:29.453Z]            {
[2023-01-20T09:23:29.453Z]              "matchDatasources": ["docker"],
[2023-01-20T09:23:29.453Z]              "registryUrls": ["https://dockerhub.nexus3.ivu.de"]
[2023-01-20T09:23:29.453Z]            },
[2023-01-20T09:23:29.453Z]            {
[2023-01-20T09:23:29.453Z]              "matchFiles": [".copier-version"],
[2023-01-20T09:23:29.453Z]              "postUpgradeTasks": {
[2023-01-20T09:23:29.453Z]                "commands": [
[2023-01-20T09:23:29.453Z]                  "pip3 install copier",
[2023-01-20T09:23:29.453Z]                  "git stash",
[2023-01-20T09:23:29.453Z]                  "GIT_SSL_NO_VERIFY=true copier --force update",
[2023-01-20T09:23:29.453Z]                  "rm -rf *.rej TODO.md"
[2023-01-20T09:23:29.453Z]                ],
[2023-01-20T09:23:29.453Z]                "fileFilters": ["**/.*", "**/*"],
[2023-01-20T09:23:29.453Z]                "executionMode": "update"
[2023-01-20T09:23:29.453Z]              }
[2023-01-20T09:23:29.453Z]            }
[2023-01-20T09:23:29.453Z]          ],
[2023-01-20T09:23:29.453Z]          "optimizeForDisabled": true,
[2023-01-20T09:23:29.453Z]          "fetchReleaseNotes": false,
[2023-01-20T09:23:29.453Z]          "persistRepoData": true,
[2023-01-20T09:23:29.453Z]          "productLinks": {
[2023-01-20T09:23:29.453Z]            "documentation": "https://docs.renovatebot.com/",
[2023-01-20T09:23:29.453Z]            "help": "https://teams.microsoft.com/l/channel/19%3ac55a93924e964f6a8583c37cf45db527%40thread.skype/IVU%2520Buildgruppe%2520(PTO)?groupId=d3b18564-d928-4a22-83c6-7fa6dff17953&tenantId=0161bba8-a41c-41d1-a282-2a1de8f1e082",
[2023-01-20T09:23:29.454Z]            "homepage": "https://confluence.ivu.de/display/BUILD/Renovate"
[2023-01-20T09:23:29.454Z]          }
[2023-01-20T09:23:29.454Z]        }
[2023-01-20T09:23:29.454Z] DEBUG: CLI config
[2023-01-20T09:23:29.454Z]        "config": {}
[2023-01-20T09:23:29.454Z] DEBUG: Env config
[2023-01-20T09:23:29.454Z]        "config": {
[2023-01-20T09:23:29.454Z]          "hostRules": [],
[2023-01-20T09:23:29.454Z]          "username": "svc_pto_build",
[2023-01-20T09:23:29.454Z]          "password": "***********",
[2023-01-20T09:23:29.454Z]          "autodiscover": false,
[2023-01-20T09:23:29.454Z]          "repositories": ["DATAHUB/vdv45x"]
[2023-01-20T09:23:29.454Z]        }
[2023-01-20T09:23:29.454Z] DEBUG: Combined config
[2023-01-20T09:23:29.454Z]        "config": {
[2023-01-20T09:23:29.454Z]          "platform": "bitbucket-server",
[2023-01-20T09:23:29.454Z]          "endpoint": "https://git.ivu-ag.com",
[2023-01-20T09:23:29.454Z]          "onboardingConfig": {"extends": ["config:base"]},
[2023-01-20T09:23:29.454Z]          "onboarding": true,
[2023-01-20T09:23:29.454Z]          "allowedPostUpgradeCommands": [".*"],
[2023-01-20T09:23:29.454Z]          "regexManagers": [
[2023-01-20T09:23:29.454Z]            {
[2023-01-20T09:23:29.454Z]              "fileMatch": ["^\\.copier-version$"],
[2023-01-20T09:23:29.454Z]              "matchStrings": [
[2023-01-20T09:23:29.454Z]                "\\s+(?<currentDigest>\\S+)\\s*#\\s*datasource=(?<datasource>\\S+)\\s+template=(?<template>\\S+)\\s*\\n"
[2023-01-20T09:23:29.454Z]              ],
[2023-01-20T09:23:29.454Z]              "depNameTemplate": "https://git.ivu-ag.com/scm/{{{template}}}.git"
[2023-01-20T09:23:29.454Z]            }
[2023-01-20T09:23:29.454Z]          ],
[2023-01-20T09:23:29.454Z]          "packageRules": [
[2023-01-20T09:23:29.454Z]            {
[2023-01-20T09:23:29.454Z]              "matchDatasources": ["maven"],
[2023-01-20T09:23:29.454Z]              "registryUrls": ["http://ac-nexus.ivu-ag.com/content/groups/public/"]
[2023-01-20T09:23:29.454Z]            },
[2023-01-20T09:23:29.454Z]            {
[2023-01-20T09:23:29.454Z]              "matchDatasources": ["docker"],
[2023-01-20T09:23:29.454Z]              "registryUrls": ["https://dockerhub.nexus3.ivu.de"]
[2023-01-20T09:23:29.454Z]            },
[2023-01-20T09:23:29.454Z]            {
[2023-01-20T09:23:29.454Z]              "matchFiles": [".copier-version"],
[2023-01-20T09:23:29.454Z]              "postUpgradeTasks": {
[2023-01-20T09:23:29.454Z]                "commands": [
[2023-01-20T09:23:29.454Z]                  "pip3 install copier",
[2023-01-20T09:23:29.454Z]                  "git stash",
[2023-01-20T09:23:29.454Z]                  "GIT_SSL_NO_VERIFY=true copier --force update",
[2023-01-20T09:23:29.454Z]                  "rm -rf *.rej TODO.md"
[2023-01-20T09:23:29.454Z]                ],
[2023-01-20T09:23:29.454Z]                "fileFilters": ["**/.*", "**/*"],
[2023-01-20T09:23:29.454Z]                "executionMode": "update"
[2023-01-20T09:23:29.454Z]              }
[2023-01-20T09:23:29.454Z]            }
[2023-01-20T09:23:29.454Z]          ],
[2023-01-20T09:23:29.454Z]          "optimizeForDisabled": true,
[2023-01-20T09:23:29.454Z]          "fetchReleaseNotes": false,
[2023-01-20T09:23:29.454Z]          "persistRepoData": true,
[2023-01-20T09:23:29.454Z]          "productLinks": {
[2023-01-20T09:23:29.454Z]            "documentation": "https://docs.renovatebot.com/",
[2023-01-20T09:23:29.454Z]            "help": "https://teams.microsoft.com/l/channel/19%3ac55a93924e964f6a8583c37cf45db527%40thread.skype/IVU%2520Buildgruppe%2520(PTO)?groupId=d3b18564-d928-4a22-83c6-7fa6dff17953&tenantId=0161bba8-a41c-41d1-a282-2a1de8f1e082",
[2023-01-20T09:23:29.454Z]            "homepage": "https://confluence.ivu.de/display/BUILD/Renovate"
[2023-01-20T09:23:29.454Z]          },
[2023-01-20T09:23:29.454Z]          "hostRules": [],
[2023-01-20T09:23:29.454Z]          "username": "svc_pto_build",
[2023-01-20T09:23:29.454Z]          "password": "***********",
[2023-01-20T09:23:29.454Z]          "autodiscover": false,
[2023-01-20T09:23:29.454Z]          "repositories": ["DATAHUB/vdv45x"]
[2023-01-20T09:23:29.455Z]        }
[2023-01-20T09:23:29.455Z] DEBUG: Adding trailing slash to endpoint
[2023-01-20T09:23:29.455Z] DEBUG: Found valid git version: 2.39.0
[2023-01-20T09:23:29.455Z] DEBUG: Adding password authentication for git.ivu-ag.com to hostRules
[2023-01-20T09:23:29.455Z] DEBUG: Using baseDir: /tmp/renovate
[2023-01-20T09:23:29.455Z] DEBUG: Using cacheDir: /tmp/renovate/cache
[2023-01-20T09:23:29.455Z] DEBUG: Using containerbaseDir: /tmp/renovate/cache/containerbase
[2023-01-20T09:23:29.455Z] DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
[2023-01-20T09:23:29.455Z] DEBUG: Commits limit = null
[2023-01-20T09:23:29.455Z] DEBUG: Setting global hostRules
[2023-01-20T09:23:29.455Z] DEBUG: Adding password authentication for git.ivu-ag.com to hostRules
[2023-01-20T09:23:29.455Z] DEBUG: validatePresets()
[2023-01-20T09:23:29.455Z] DEBUG: Reinitializing hostRules for repo
[2023-01-20T09:23:29.455Z] DEBUG: Clearing hostRules
[2023-01-20T09:23:29.455Z] DEBUG: Adding password authentication for git.ivu-ag.com to hostRules
[2023-01-20T09:23:29.455Z]  INFO: Repository started (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.455Z]        "renovateVersion": "34.106.0"
[2023-01-20T09:23:29.455Z] DEBUG: Using localDir: /tmp/renovate/repos/bitbucket-server/DATAHUB/vdv45x (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.455Z] DEBUG: PackageFiles.clear() - Package files deleted (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.713Z] DEBUG: initRepo("{
[2023-01-20T09:23:29.713Z]          "repository": "DATAHUB/vdv45x"
[2023-01-20T09:23:29.713Z]        }") (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.713Z] (node:9) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
[2023-01-20T09:23:29.713Z] (Use `node --trace-warnings ...` to show where the warning was created)
[2023-01-20T09:23:29.714Z] DEBUG: DATAHUB/vdv45x owner = DATAHUB (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.971Z] DEBUG: Resetting npmrc (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.972Z] DEBUG: checkOnboarding() (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.972Z] DEBUG: isOnboarded() (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.972Z] DEBUG: findFile(renovate.json) (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:29.972Z] DEBUG: Initializing git repository into /tmp/renovate/repos/bitbucket-server/DATAHUB/vdv45x (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:30.234Z] DEBUG: resetToBranch(master) (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.176Z] DEBUG: resetToBranch(master) (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.437Z] DEBUG:  (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.437Z]        "existingBranches": ["renovate/major-lucene-version"]
[2023-01-20T09:23:31.698Z]  INFO: git fetch completed (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.698Z]        "durationMs": 1153
[2023-01-20T09:23:31.698Z] DEBUG: latest repository commit (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.698Z]        "latestCommit": {
[2023-01-20T09:23:31.698Z]          "hash": "7dc33c313888211d55b8550e44e4f7b1da3c3253",
[2023-01-20T09:23:31.698Z]          "date": "2023-01-20T09:38:07+01:00",
[2023-01-20T09:23:31.698Z]          "message": "Pull request #245: Update dependency org.xmlunit:xmlunit-assertj to v2.9.1",
[2023-01-20T09:23:31.698Z]          "refs": "HEAD -> master, origin/master, origin/HEAD",
[2023-01-20T09:23:31.698Z]          "body": "Merge in DATAHUB/vdv45x from renovate/xmlunit-assertj.version to master\n\n* commit 'ddfe1584648797a9dd1270139109205047b627be':\n  Update dependency org.xmlunit:xmlunit-assertj to v2.9.1\n",
[2023-01-20T09:23:31.698Z]          "author_name": "BUILD Gruppe PTO Service Account Renovate",
[2023-01-20T09:23:31.698Z]          "author_email": "whatever@ivu.de"
[2023-01-20T09:23:31.698Z]        }
[2023-01-20T09:23:31.698Z] DEBUG: Config file exists, fileName: renovate.json (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.698Z] DEBUG: ensureIssueClosing(Action required: Add a Renovate config) (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.698Z] DEBUG: Repo is onboarded (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.962Z] DEBUG: Found renovate.json config file (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.962Z] DEBUG: Repository config (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.962Z]        "fileName": "renovate.json",
[2023-01-20T09:23:31.962Z]        "config": {
[2023-01-20T09:23:31.962Z]          "$schema": "https://docs.renovatebot.com/renovate-schema.json",
[2023-01-20T09:23:31.963Z]          "extends": ["config:base"]
[2023-01-20T09:23:31.963Z]        }
[2023-01-20T09:23:31.963Z] DEBUG: migrateAndValidate() (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.963Z] DEBUG: No config migration necessary (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.963Z] DEBUG: massaged config (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.963Z]        "config": {
[2023-01-20T09:23:31.963Z]          "$schema": "https://docs.renovatebot.com/renovate-schema.json",
[2023-01-20T09:23:31.963Z]          "extends": ["config:base"]
[2023-01-20T09:23:31.963Z]        }
[2023-01-20T09:23:31.963Z] DEBUG: migrated config (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:31.963Z]        "config": {
[2023-01-20T09:23:31.963Z]          "$schema": "https://docs.renovatebot.com/renovate-schema.json",
[2023-01-20T09:23:31.963Z]          "extends": ["config:base"]
[2023-01-20T09:23:31.963Z]        }
[2023-01-20T09:23:32.892Z] DEBUG: Setting hostRules from config (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z] DEBUG: Found repo ignorePaths (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z]        "ignorePaths": [
[2023-01-20T09:23:32.892Z]          "**/node_modules/**",
[2023-01-20T09:23:32.892Z]          "**/bower_components/**",
[2023-01-20T09:23:32.892Z]          "**/vendor/**",
[2023-01-20T09:23:32.892Z]          "**/examples/**",
[2023-01-20T09:23:32.892Z]          "**/__tests__/**",
[2023-01-20T09:23:32.892Z]          "**/test/**",
[2023-01-20T09:23:32.892Z]          "**/tests/**",
[2023-01-20T09:23:32.892Z]          "**/__fixtures__/**"
[2023-01-20T09:23:32.892Z]        ]
[2023-01-20T09:23:32.892Z] DEBUG: getVulnerabilityAlerts() (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z] DEBUG: No vulnerability alerts found (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z] DEBUG: findIssue(Dependency Dashboard) (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z] DEBUG: No baseBranches (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:32.892Z] DEBUG: extract() (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:33.149Z] DEBUG: Setting current branch to master (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:33.149Z] DEBUG: latest commit (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:33.149Z]        "branchName": "master",
[2023-01-20T09:23:33.149Z]        "latestCommitDate": "2023-01-20T09:38:07+01:00"
[2023-01-20T09:23:36.464Z] DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:38.994Z] DEBUG: Using file match: (^|/)requirements\.ya?ml$ for manager ansible-galaxy (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:38.995Z] DEBUG: Using file match: (^|/)galaxy\.ya?ml$ for manager ansible-galaxy (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:43.202Z] DEBUG: Using file match: (^|/)\.tool-versions$ for manager asdf (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:45.734Z] DEBUG: Using file match: azure.*pipelines?.*\.ya?ml$ for manager azure-pipelines (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:47.642Z] DEBUG: Using file match: (^|/)batect(-bundle)?\.yml$ for manager batect (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:50.199Z] DEBUG: Using file match: (^|/)batect$ for manager batect-wrapper (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:52.736Z] DEBUG: Using file match: (^|/)WORKSPACE(|\.bazel)$ for manager bazel (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:52.736Z] DEBUG: Using file match: \.bzl$ for manager bazel (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:54.641Z] DEBUG: Using file match: (^|\/)\.bazelversion$ for manager bazelisk (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:57.200Z] DEBUG: Using file match: (^|/)\.?bitbucket-pipelines\.ya?ml$ for manager bitbucket-pipelines (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:59.728Z] DEBUG: Using file match: buildkite\.ya?ml for manager buildkite (repository=DATAHUB/vdv45x)
[2023-01-20T09:23:59.728Z] DEBUG: Using file match: \.buildkite/.+\.ya?ml$ for manager buildkite (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:02.255Z] DEBUG: Using file match: (^|/)Gemfile$ for manager bundler (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:04.151Z] DEBUG: Using file match: \.cake$ for manager cake (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:06.738Z] DEBUG: Using file match: (^|/)Cargo\.toml$ for manager cargo (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:12.022Z] DEBUG: Using file match: (^|/)\.circleci/config\.yml$ for manager circleci (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:13.943Z] DEBUG: Using file match: (^|/)cloudbuild\.ya?ml for manager cloudbuild (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:15.863Z] DEBUG: Using file match: (^|/)Podfile$ for manager cocoapods (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:18.406Z] DEBUG: Using file match: (^|/)([\w-]*)composer\.json$ for manager composer (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:20.973Z] DEBUG: Using file match: (^|/)conanfile\.(txt|py)$ for manager conan (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:22.883Z] DEBUG: Using file match: (^|/)(?:deps|bb)\.edn$ for manager deps-edn (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:24.788Z] DEBUG: Using file match: (^|/)(?:docker-)?compose[^/]*\.ya?ml$ for manager docker-compose (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:27.329Z] DEBUG: Using file match: (^|/|\.)Dockerfile$ for manager dockerfile (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:27.329Z] DEBUG: Using file match: (^|/)Dockerfile[^/]*$ for manager dockerfile (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:29.236Z] DEBUG: Using file match: (^|/)\.drone\.yml$ for manager droneci (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:31.804Z] DEBUG: Using file match: (^|/)fleet\.ya?ml for manager fleet (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:33.710Z] DEBUG: Using file match: (^|\/)flux-system\/(?:.+\/)?gotk-components\.yaml$ for manager flux (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:36.255Z] DEBUG: Using file match: (^|\/)\.fvm\/fvm_config\.json$ for manager fvm (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:38.162Z] DEBUG: Using file match: (^|/)\.gitmodules$ for manager git-submodules (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:40.731Z] DEBUG: Using file match: ^(workflow-templates|\.github\/workflows)\/[^/]+\.ya?ml$ for manager github-actions (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:40.731Z] DEBUG: Using file match: (^|\/)action\.ya?ml$ for manager github-actions (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:42.631Z] DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:45.167Z] DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci-include (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:47.079Z] DEBUG: Using file match: (^|/)go\.mod$ for manager gomod (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.613Z] DEBUG: Using file match: \.gradle(\.kts)?$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.614Z] DEBUG: Using file match: (^|\/)gradle\.properties$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.614Z] DEBUG: Using file match: (^|\/)gradle\/.+\.toml$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.614Z] DEBUG: Using file match: \.versions\.toml$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.614Z] DEBUG: Using file match: (^|\/)versions.props$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:49.614Z] DEBUG: Using file match: (^|\/)versions.lock$ for manager gradle (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:51.525Z] DEBUG: Using file match: (^|/)gradle/wrapper/gradle-wrapper\.properties$ for manager gradle-wrapper (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:54.157Z] DEBUG: Using file match: (^|/)requirements\.yaml$ for manager helm-requirements (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:56.688Z] DEBUG: Using file match: (^|/)values\.yaml$ for manager helm-values (repository=DATAHUB/vdv45x)
[2023-01-20T09:24:58.603Z] DEBUG: Using file match: (^|/)helmfile\.yaml$ for manager helmfile (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:03.887Z] DEBUG: Using file match: (^|/)Chart\.yaml$ for manager helmv3 (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:05.803Z] DEBUG: Using file match: (^|/)bin/hermit$ for manager hermit (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:08.373Z] DEBUG: Using file match: ^Formula/[^/]+[.]rb$ for manager homebrew (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:10.929Z] DEBUG: Using file match: \.html?$ for manager html (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:12.857Z] DEBUG: Using file match: (^|/)plugins\.(txt|ya?ml)$ for manager jenkins (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:15.394Z] DEBUG: Using file match: (^|/)jsonnetfile\.json$ for manager jsonnet-bundler (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:17.942Z] DEBUG: Using file match: ^.+\.main\.kts$ for manager kotlin-script (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:22.186Z] DEBUG: Using file match: (^|/)kustomization\.ya?ml$ for manager kustomize (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:24.092Z] DEBUG: Using file match: (^|/)project\.clj$ for manager leiningen (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:26.635Z] DEBUG: Using file match: (^|/|\.)pom\.xml$ for manager maven (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:26.635Z] DEBUG: Using file match: ^(((\.mvn)|(\.m2))/)?settings\.xml$ for manager maven (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:28.551Z] DEBUG: Using file match: (^|/)package\.js$ for manager meteor (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:31.088Z] DEBUG: Using file match: (^|\/)Mintfile$ for manager mint (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:33.017Z] DEBUG: Using file match: (^|/)mix\.exs$ for manager mix (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:35.564Z] DEBUG: Using file match: (^|\/)flake\.nix$ for manager nix (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:38.105Z] DEBUG: Using file match: (^|/)\.node-version$ for manager nodenv (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:40.660Z] DEBUG: Using file match: (^|/)package\.json$ for manager npm (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:42.624Z] DEBUG: Using file match: \.(?:cs|fs|vb)proj$ for manager nuget (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:42.624Z] DEBUG: Using file match: \.(?:props|targets)$ for manager nuget (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:42.625Z] DEBUG: Using file match: (^|\/)dotnet-tools\.json$ for manager nuget (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:42.625Z] DEBUG: Using file match: (^|\/)global\.json$ for manager nuget (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:45.157Z] DEBUG: Using file match: (^|/)\.nvmrc$ for manager nvm (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:47.692Z] DEBUG: Using file match: (^|/)src/main/features/.+\.json$ for manager osgi (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:52.981Z] DEBUG: Using file match: (^|/)([\w-]*)requirements\.(txt|pip)$ for manager pip_requirements (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:54.373Z] DEBUG: Using file match: (^|/)setup\.py$ for manager pip_setup (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:56.911Z] DEBUG: Using file match: (^|/)Pipfile$ for manager pipenv (repository=DATAHUB/vdv45x)
[2023-01-20T09:25:58.823Z] DEBUG: Using file match: (^|/)pyproject\.toml$ for manager poetry (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:01.367Z] DEBUG: Using file match: (^|/)\.pre-commit-config\.yaml$ for manager pre-commit (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:03.299Z] DEBUG: Using file match: (^|/)pubspec\.ya?ml$ for manager pub (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:05.859Z] DEBUG: Using file match: (^|\/)Puppetfile$ for manager puppet (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:07.840Z] DEBUG: Using file match: (^|/)\.python-version$ for manager pyenv (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:10.381Z] DEBUG: Using file match: ^\.copier-version$ for manager regex (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:12.940Z] DEBUG: Using file match: (^|/)\.ruby-version$ for manager ruby-version (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:15.485Z] DEBUG: Using file match: \.sbt$ for manager sbt (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:15.485Z] DEBUG: Using file match: project/[^/]*.scala$ for manager sbt (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:17.407Z] DEBUG: Using file match: (^|/)setup\.cfg$ for manager setup-cfg (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:19.965Z] DEBUG: Using file match: (^|/)Package\.swift for manager swift (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:25.275Z] DEBUG: Using file match: \.tf$ for manager terraform (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:26.676Z] DEBUG: Using file match: (^|/)\.terraform-version$ for manager terraform-version (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:29.220Z] DEBUG: Using file match: (^|/)terragrunt\.hcl$ for manager terragrunt (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:31.132Z] DEBUG: Using file match: (^|/)\.terragrunt-version$ for manager terragrunt-version (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:33.115Z] DEBUG: Using file match: \.tflint\.hcl$ for manager tflint-plugin (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:35.657Z] DEBUG: Using file match: ^\.travis\.yml$ for manager travis (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:37.563Z] DEBUG: Using file match: (^|/)\.vela\.ya?ml$ for manager velaci (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:39.469Z] DEBUG: Using file match: ^\.woodpecker(?:\/[^/]+)?\.ya?ml$ for manager woodpecker (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:39.469Z] DEBUG: Matched 1 file(s) for manager dockerfile: Dockerfile (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:39.469Z] DEBUG: Matched 5 file(s) for manager html: vdv45x-main/src/main/resources/templates/error.html, vdv45x-web/src/main/resources/static/interfaces/ura/journeyAlternativeDebug.html, vdv45x-web/src/main/resources/static/interfaces/ura/journeyDebug.html, vdv45x-web/src/main/resources/static/interfaces/ura/journeyUpdateDebug.html, vdv45x-web/src/main/resources/templates/index.html (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:39.469Z] DEBUG: Matched 13 file(s) for manager maven: pom.xml, vdv45x-command/pom.xml, vdv45x-connector/pom.xml, vdv45x-core/pom.xml, vdv45x-incident/pom.xml, vdv45x-ipc-interface/pom.xml, vdv45x-logging/pom.xml, vdv45x-main/pom.xml, vdv45x-npm/pom.xml, vdv45x-siri/pom.xml, vdv45x-soap-ret-river/pom.xml, vdv45x-trias/pom.xml, vdv45x-web/pom.xml (repository=DATAHUB/vdv45x)
[2023-01-20T09:26:39.736Z] DEBUG: Found dockerfile package files (repository=DATAHUB/vdv45x)

Have you created a minimal reproduction repository?

No reproduction repository

The text was updated successfully, but these errors were encountered:

rarkins · 2023-01-20T11:32:16Z

That seems unlikely based on this diff: https://app.renovatebot.com/package-diff?name=renovate&from=34.102.5&to=34.102.6.

Please:

Fill in the questionnaire above completely
Describe how you run Renovate exactly (e.g. the full fat Docker image, or slim)
To what extent have you tested/debugged, or just looked at timestamps on logs?

mszalbach · 2023-01-20T12:31:21Z

Hi thanks for the reply I added the details which docker image I use and on which platforms I run.

I have tested this by starting Renovate with different versions multiple times and and checked timestamps of logs and the overall runtime of the process, and the "durationMs" of Renovate, which all gets slower.

From the release notes I would agree there is nothing remotely connected to this behavior.
Maybe you have some tips what else I can try or enable to have some more insights.

rarkins · 2023-01-20T13:18:25Z

It looks like you changed your description about affected version too? Are you able to retry until you pinpoint it?

rarkins · 2023-01-20T13:28:36Z

I'm running some quick tests using Docker on my Mac and I can confirm I'm seeing very slow extract too. I'll try to pinpoint which release it is

mszalbach · 2023-01-20T13:30:21Z

I am not sure if I did some error writing it down but my tests has the following results:

Version 34.100.2 fast
Version 34.101.0 fast
Version 34.102.0 fast
Version 34.102.5 fast
Version 34.102.6 slow
Version 34.102.7 slow
Version 34.106.0 slow

mszalbach · 2023-01-20T13:34:51Z

I also created an example on github against a fork of a Glassfish Repository (https://github.com/mszalbach/glassfish).
If you wish I can also upload the docker compose and Renovate config I used.

With 34.106.0 it takes 1 minute+ for the first 12 file match. With the older versions I can scan this in 12 seconds. With smaller repos it is not so clear.

mszalbach · 2023-01-20T13:40:19Z

Did some more tests with the Github platform and I would still say the breaking point was the release renovate/renovate:34.102.6 The image renovate/renovate:34.102.5 is faster.

rarkins · 2023-01-20T14:02:22Z

I still can't believe my eyes, but somehow this seems caused by prettier update from 2.8.1 to 2.8.2 (and not by the type changes related to that update).

I get slowness on the latest commit when running like this:

RENOVATE_REPOSITORY_CACHE=reset yarn start --dry-run=extract renovatebot/renovate

DEBUG: Repository timing splits (milliseconds) (repository=renovatebot/renovate)
       "splits": {"init": 8044, "extract": 14854},
       "total": 23052

If I downgrade prettier, I get:

DEBUG: Repository timing splits (milliseconds) (repository=renovatebot/renovate)
       "splits": {"init": 11294, "extract": 2863},
       "total": 14289

(I suggest ignore init, it's mainly git and may fluctuate a little).

viceice · 2023-01-20T14:04:36Z

confirmed

build(deps): update dependency prettier to v2.8.2 #19843

viceice · 2023-01-20T14:10:41Z

disable prettier import speeds up too

viceice · 2023-01-20T14:10:55Z

here's the changes of prettier:
prettier/prettier@2.8.1...2.8.2

rarkins · 2023-01-20T14:14:15Z

npm diff: https://app.renovatebot.com/package-diff?name=prettier&from=2.8.1&to=2.8.2

mszalbach · 2023-01-20T17:06:12Z

Thanks a lot for the fast replies and the fast fix.

rarkins · 2023-01-20T17:12:39Z

Thanks for reporting! Once it's built, can you confirm it fixes things from your side?

mszalbach · 2023-01-20T17:49:22Z

Of course I will test it and will give you guys feedback, no later than Monday.

renovate-release · 2023-01-20T19:50:03Z

🎉 This issue has been resolved in version 34.108.1 🎉

The release is available on:

GitHub release
34.108.1

Your semantic-release bot 📦🚀

mszalbach · 2023-01-23T07:29:35Z

Tested with Release 34.108.5 and scan time is back to ~30 seconds like expected.
So everything works fine, thanks a lot.

mszalbach added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Jan 20, 2023

viceice added priority-2-high Bugs impacting wide number of users or very important features status:ready and removed status:requirements Full requirements are not yet known, so implementation should not be started priority-5-triage labels Jan 20, 2023

viceice self-assigned this Jan 20, 2023

This was referenced Jan 20, 2023

fix: dynamically load prettier #19948

Merged

Massive app slowdown when loading prettier 2.8.2 into memory prettier/prettier#14209

Open

viceice closed this as completed in #19948 Jan 20, 2023

HonkingGoose added the regression Issue about a regression bug, or the PR caused it label Jan 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Feb 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File match scan got slow since version #19937

File match scan got slow since version #19937

mszalbach commented Jan 20, 2023 •

edited

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

rarkins commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

mszalbach commented Jan 20, 2023

mszalbach commented Jan 20, 2023 •

edited

rarkins commented Jan 20, 2023

viceice commented Jan 20, 2023

viceice commented Jan 20, 2023

viceice commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

renovate-release commented Jan 20, 2023

mszalbach commented Jan 23, 2023

File match scan got slow since version #19937

File match scan got slow since version #19937

Comments

mszalbach commented Jan 20, 2023 • edited

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

If you're self-hosting Renovate, select which platform you are using.

If you're self-hosting Renovate, tell us what version of the platform you run.

Was this something which used to work for you, and then stopped?

Describe the bug

Relevant debug logs

Have you created a minimal reproduction repository?

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

rarkins commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

mszalbach commented Jan 20, 2023

mszalbach commented Jan 20, 2023 • edited

rarkins commented Jan 20, 2023

viceice commented Jan 20, 2023

viceice commented Jan 20, 2023

viceice commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

rarkins commented Jan 20, 2023

mszalbach commented Jan 20, 2023

renovate-release commented Jan 20, 2023

mszalbach commented Jan 23, 2023

mszalbach commented Jan 20, 2023 •

edited

mszalbach commented Jan 20, 2023 •

edited