feat(cache): Repository cache fingerprinting

[zharinov]

Changes

• New cache repo revision
• New fingerprint parameter
• Gradual migration

Context

• Closes: Unrecoverable repo cache problem when repository reset #17247

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[zharinov]

Blocked by #17364

[rarkins]

Why the need for gradual migration instead of full? Assumption that it's too heavy for the app in production?

[zharinov]

Why the need for gradual migration instead of full? Assumption that it's too heavy for the app in production?

Yes, otherwise it will reset all the caches instantly 🤷‍♂️

[viceice]

🤔 is writing new json files so bad? i don't see a big issue for the hoseted app. I think it's always copying the cache file after each repo anyways. 🤔

[zharinov]

It's not about writing JSON, we have to reset all the cache while migrating from 12 to 13. I forgot to add this "resetting" part.

[viceice]

It's not about writing JSON, we have to reset all the cache while migrating from 12 to 13. I forget to add this "resetting" part.

but didn't we still use / migrate the old cache? so it's irrelevant if we change it afterwards?
or do you fully discard the old cache?

[zharinov]

If we migrate gradually, we have to migrate previous versions to 12, or reset the data and migrate it to 13 for the smaller fraction of repos. If we're okay to reset them all at once, I think we can make "fresh start" for the revisions.

[viceice]

i still don't know why we can't migrate all old caches like we did before?

btw the hosted app is using multiple worker nodes and every node will only process one repo at a time.

[rarkins]

Hard to tell from code inspection alone

[viceice]

i still don't understand why we can't migrate old cache this time

[zharinov]

Because for some repos, we may reuse the cache of previously deleted repository with same name, and there is no easy way to detect repos like these.

[viceice]

Because for some repos, we may reuse the cache of previously deleted repository with same name, and there is no easy way to detect repos like these.

that case is very rare, so those could be manually clean like today?

[rarkins]

How much work would it be to preserve existing caches? i.e. knowing that maybe some of them are invalid, but thinking it's an OK trade-off?

[zharinov]

Well, we may just write and check fingerprints only for repos that don't have any cache (i.e. only for new ones).

[rarkins]

So the choice is between:

• (current) burn the existing cache, but do it gradually to prevent overload, or
• preserve the existing cache, do it immediately: add the fingerprint to all existing caches?

[zharinov]

Not exactly:

• (current) reset all the cache at once and start with newest revision
• (previous) gradually migrate all the caches controlling pace of change via hardcoded variable
• add fingerprint to caches, despite some of them can be wrong

[viceice]

i would simply add the fingerprint if missing as the wrong cases are only a few repos.

[rarkins]

OK, let's go with @viceice's suggestion and I'll keep manually fixing any we find in the app. Sorry for the changing of requirements

[zharinov]

No problem, I'll restore it

[viceice]

otherwise LGTM

[renovate-release]

🎉 This PR is included in version 32.184.0 🎉

The release is available on:

• GitHub release
• 32.184.0

Your semantic-release bot 📦🚀