Adopt keyv to help expand storage/cache options

[setchy]

What would you like Renovate to be able to do?

Raising as a feature request / suggestion. Currently renovate supports a few different storage options (local, redis, S3), each with their own implementation.

Keyv is a simple key-value storage with support for multiple backend adapters (MySQL, PostgreSQL, SQLite, Redis, Mongo, DynamoDB, Firestore, Memcached, and more).

There are two flavors of it

• https://github.com/jaredwray/keyv - original
• https://github.com/microlinkhq/keyv // https://keyvhq.js.org/#/ - fork with a few more storage adapters (ie: S3)

If you have any ideas on how this should be implemented, please tell us here.

By introducing something like keyv, Renovate would be able to support a wider variety of storage options without needing a custom implementation for each specific adapter

Is this a feature you are interested in implementing yourself?

Maybe

[rarkins]

Sounds like a good idea. We can make breaking changes to configuration in a major release if necessary

[rarkins]

Happy for this to go ahead but suggest if you or anyone is implementing it then check here in this issue first with proposed configuration changes, etc. so we can discuss

[rarkins]

Today we have both compression and memoization with our package cache, so it would be interesting if we can replace both those with the keyv equivalents. I'm not sure their definition of memoization is same as I'm thinking. In Renovate's case I mean we prevent doing more than one identical query per package within the same repo run.

We'd also need to support the file-based approach to replace cacache. Maybe need to check if it supports concurrent reads/writes, although I forget exactly what we require these days.

[rarkins]

I would like to support the package cache first, but we should also think about repo cache support for later too. It must be possible to pick different backends for each (e.g. Redis for package cache, S3 for repo cache).

[viceice]

• i like the build-in compress and ttl feature.
• the multi wrapper can help to have a local in memory cahcve and a persistent remote cache
• sqlite adapter can replace the cacache, it supports concurrent access

[viceice]

• the sql adapters don't support ttl
• the s3 adapter stores cache public, so anyone who knows the bucket and key can download the cache value.

[rarkins]

• the sql adapters don't support ttl

Strange, I wonder if it can be added (e.g. a ttl column, and period sweep).

• the s3 adapter stores cache public, so anyone who knows the bucket and key can download the cache value.

We could submit a PR to make it customizable?

[setchy]

@jamietanna @viceice - is this idea suggestion still valid, or worth closing out?