Introduce Redis to handle resources

[rsarky]

After a conversation with @rralf I realised that a lot of the problems and hacky things we do in PaStA is while trying to persist stuff. Currently we are persisting stuff in an ad-hoc way in text files and caches and trying to simulate a database.

What if we actually used a database instead? I dont have a lot of experience with Redis but seems like it could be a good fit for this usecase.

Other database options can be considered too

[rralf]

Indexes are not just for simulating a database. They give a human-readable overview and allow manipulation with standard tooling.

However, I agree that we can use databases for replacing the pickle-hell that we currently have. There might be another student that will evaluate this topic (hopefully soon).

[rsarky]

Indexes are not just for simulating a database. They give a human-readable overview and allow manipulation with standard tooling.

A human readable overview could be a view into the database.

The choice of a database will also depend on whether the data that we need to store can fit in memory

[rralf]

But regarding memory consumption, indexes are not a major issue. It's the pickle files (i.e., commit caches), and the way how python duplicates memory when doing things in parallel. If we want to benefit from databases, then a persistent, non-relational in-memory database (e.g., key-value store) would probably be the best choice. So yes, redis could be a promising candidate.