Preset changes cause unnecessary cache busting

[rarkins]

Describe the proposed change(s).

The hosted app was upgraded between these versions: https://app.renovatebot.com/package-diff?name=renovate&from=32.214.0&to=32.216.0

Clone percentage immediately went up to ~100%

Why?

Random example from public repo https://github.com/larvit/larvitsession which had no commit for 9 days:

Last job log before upgrade:

{"level":20,"msg":"Using app token for git init","time":"2022-10-05T03:09:46.830Z"}
{"level":20,"msg":"Repository cache is restored from revision 13","time":"2022-10-05T03:09:47.238Z"}
{"level":20,"msg":"Resetting npmrc","time":"2022-10-05T03:09:47.238Z"}
{"level":20,"msg":"detectSemanticCommits()","time":"2022-10-05T03:09:47.239Z"}
{"level":20,"msg":"checkOnboarding()","time":"2022-10-05T03:09:47.239Z"}
{"level":20,"msg":"isOnboarded()","time":"2022-10-05T03:09:47.239Z"}
{"level":20,"msg":"Checking cached config file name","time":"2022-10-05T03:09:47.239Z"}
{"level":20,"msg":"Existing config file confirmed","time":"2022-10-05T03:09:47.474Z"}
{"level":20,"fileName":"renovate.json","config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>larvit/standards:renovate-default"]},"msg":"Repository config","time":"2022-10-05T03:09:47.474Z"}
{"level":20,"msg":"Repo is onboarded","time":"2022-10-05T03:09:47.474Z"}
{"level":20,"msg":"migrateAndValidate()","time":"2022-10-05T03:09:47.475Z"}
{"level":20,"msg":"No config migration necessary","time":"2022-10-05T03:09:47.476Z"}
{"level":20,"config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>whitesource/merge-confidence:beta","github>larvit/standards:renovate-default"]},"msg":"massaged config","time":"2022-10-05T03:09:47.476Z"}
{"level":20,"config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>whitesource/merge-confidence:beta","github>larvit/standards:renovate-default"]},"msg":"migrated config","time":"2022-10-05T03:09:47.476Z"}
{"level":20,"msg":"Setting hostRules from config","time":"2022-10-05T03:09:48.721Z"}
{"level":20,"ignorePaths":["**/node_modules/**","**/bower_components/**","**/vendor/**","**/examples/**","**/__tests__/**","**/test/**","**/tests/**","**/__fixtures__/**"],"msg":"Found repo ignorePaths","time":"2022-10-05T03:09:48.768Z"}
{"level":20,"concurrency":10,"host":"api.github.com","msg":"Using queue","time":"2022-10-05T03:09:48.772Z"}
{"level":20,"msg":"No vulnerability alerts found","time":"2022-10-05T03:09:48.990Z"}
{"level":20,"msg":"No vulnerability alerts found","time":"2022-10-05T03:09:48.990Z"}
{"level":20,"msg":"findIssue(Dependency Dashboard)","time":"2022-10-05T03:09:48.991Z"}
{"level":20,"msg":"Retrieving issueList","time":"2022-10-05T03:09:48.991Z"}
{"level":20,"msg":"Retrieved 1 issues","time":"2022-10-05T03:09:49.215Z"}
{"level":20,"msg":"Found issue 49","time":"2022-10-05T03:09:49.215Z"}
{"level":20,"msg":"No baseBranches","time":"2022-10-05T03:09:49.395Z"}
{"level":20,"msg":"extract()","time":"2022-10-05T03:09:49.396Z"}
{"level":20,"baseBranch":"master","baseBranchSha":"b8619f8dc5b55bd65bddfc64f220ef69e09d9c97","msg":"Found cached extract","time":"2022-10-05T03:09:49.397Z"}

After upgrade:

{"level":20,"msg":"Using app token for git init","time":"2022-10-05T05:39:37.737Z"}
{"level":20,"msg":"Repository cache is restored from revision 13","time":"2022-10-05T05:39:38.155Z"}
{"level":20,"msg":"Resetting npmrc","time":"2022-10-05T05:39:38.155Z"}
{"level":20,"msg":"detectSemanticCommits()","time":"2022-10-05T05:39:38.156Z"}
{"level":20,"msg":"checkOnboarding()","time":"2022-10-05T05:39:38.156Z"}
{"level":20,"msg":"isOnboarded()","time":"2022-10-05T05:39:38.156Z"}
{"level":20,"msg":"Checking cached config file name","time":"2022-10-05T05:39:38.156Z"}
{"level":20,"msg":"Existing config file confirmed","time":"2022-10-05T05:39:38.394Z"}
{"level":20,"fileName":"renovate.json","config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>larvit/standards:renovate-default"]},"msg":"Repository config","time":"2022-10-05T05:39:38.395Z"}
{"level":20,"msg":"Repo is onboarded","time":"2022-10-05T05:39:38.395Z"}
{"level":20,"msg":"migrateAndValidate()","time":"2022-10-05T05:39:38.396Z"}
{"level":20,"msg":"No config migration necessary","time":"2022-10-05T05:39:38.397Z"}
{"level":20,"config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>whitesource/merge-confidence:beta","github>larvit/standards:renovate-default"]},"msg":"massaged config","time":"2022-10-05T05:39:38.397Z"}
{"level":20,"config":{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["github>whitesource/merge-confidence:beta","github>larvit/standards:renovate-default"]},"msg":"migrated config","time":"2022-10-05T05:39:38.397Z"}
{"level":20,"msg":"Setting hostRules from config","time":"2022-10-05T05:39:39.829Z"}
{"level":20,"ignorePaths":["**/node_modules/**","**/bower_components/**","**/vendor/**","**/examples/**","**/__tests__/**","**/test/**","**/tests/**","**/__fixtures__/**"],"msg":"Found repo ignorePaths","time":"2022-10-05T05:39:39.874Z"}
{"level":20,"concurrency":10,"host":"api.github.com","msg":"Using queue","time":"2022-10-05T05:39:39.879Z"}
{"level":20,"msg":"No vulnerability alerts found","time":"2022-10-05T05:39:40.099Z"}
{"level":20,"msg":"No vulnerability alerts found","time":"2022-10-05T05:39:40.099Z"}
{"level":20,"msg":"findIssue(Dependency Dashboard)","time":"2022-10-05T05:39:40.100Z"}
{"level":20,"msg":"Retrieving issueList","time":"2022-10-05T05:39:40.100Z"}
{"level":20,"msg":"Retrieved 1 issues","time":"2022-10-05T05:39:40.332Z"}
{"level":20,"msg":"Found issue 49","time":"2022-10-05T05:39:40.332Z"}
{"level":20,"msg":"No baseBranches","time":"2022-10-05T05:39:40.548Z"}
{"level":20,"msg":"extract()","time":"2022-10-05T05:39:40.548Z"}
{"level":20,"msg":"Setting current branch to master","time":"2022-10-05T05:39:40.549Z"}
{"level":20,"msg":"Initializing git repository into /mnt/renovate/gh/larvit/larvitsession","time":"2022-10-05T05:39:40.550Z"}
{"level":20,"msg":"Performing blobless clone","time":"2022-10-05T05:39:40.550Z"}

I don't see why the config change caused by preset addition should have cache-busted the extract.

[RahulGautamSingh]

I think one reason for this could be change in order of config. Maybe some updates were made that caused change in config order between the 2 versions. We use JSON.stringify() which should be replaced with
safe-stable-stringify like we do while generating hash for branchFingerprint

[rarkins]

Not sure it's it, but a good start

[rarkins]

It's definitely not it.

Here's where the fingerprint is calculated:

renovate/lib/workers/repository/process/extract-update.ts

Line 74 in 4c1b3e5

const configHash = hasha(JSON.stringify(config));

I debugged and confirmed it's the full config, not just the extract config. So if you extend config:base, then any change to monorepo presets will invalidate it.

Instead we need to fingerprint based on the extract config, not full

[renovate-release]

🎉 This issue has been resolved in version 32.219.1 🎉

The release is available on:

• GitHub release
• 32.219.1

Your semantic-release bot 📦🚀