[politeiad+www] Create single source of truth cache layer

[marcopeereboom]

At this time www caches proposals, comments etc in memory. As this grows this is going to become increasingly hard on memory and the hosting machines. In addition, the caches we use are prone to errors because they are updated in several code paths.

What we need is a cache layer that politeiad has write-only access to and politeaid has read-only access from. The flow of data becomes as follows: www receives command -> www cooks command and sends to d -> d stores new data -> d updates cache -> d completes command to www -> www reads cache (if needed) and replies completion to user.

With such an architecture we can scale www without putting too much strain on politeiad and thus prolonging the usefulness of git as the backend in politeiad. In addition, this will reduce chattiness and reliance on d for all responses. RPC commands will be able to be trimmed down (don't reply with data at all, just status) and solely rely on cache for data answers.

• Determine what to use for cache (memcached, RoachDB, Redis, Mongo, ?)
• Update politeiad to update caches when new props, new comments etc come in.
• Update www to read data from cache.
• Drop data answers from JSON REST protocol and solely rely on cache.

[xaur]

What we need is a cache layer that politeiad has write-only access to and politeaid has read-only access from.

www?

[lukebp]

Here's a summary of some of my thoughts on the database choice.

Problem
Two main issues with current setup that we're trying to solve:

1. We store lot of data in an in-memory cache. This will be increasingly hard on memory and the hosting machines. Not very scalable.
2. Our caches are updated in several code paths making them prone to errors.

In-memory stores
Redis and memcached are both in-memory key-value stores. The primary benefit is speed, but at the cost of memory. We'd be storing a lot of data in-memory (proposals/attachments, comments, comment likes, ticket pool snapshots), so using one of these options would help solve issue 2, but wouldn't do much to address issue 1.

MongoDB
Mature NoSQL distributed database.

Compared to CockroachDB:

• Less development overhead
• More scaling overhead
• Higher latency

CockroachDB
Distributed, strongly-consistent, transactional key-value store, with the external API being standard SQL.
Built to automatically replicate, rebalance, and recover with minimal configuration and operational overhead.
Scales without the manual complexity of sharding.

Compared to MongoDB:

• More development overhead
• Less scaling overhead
• Lower latency

CockroachDB is not as mature of a product as MongoDB. The company that builds CockroachDB was founded in 2015 and there is always the risk that they run out of funding like we've seen with other database projects (ex. RethinkDB). They've raised a total of $53.5M to date with their most recent round being a Series B, so I think there is a good chance that they'll be around for a while.

Conclusion
I would go with either MongoDB or CockroachDB. My personal vote would be to go with CockroachDB for the improved scalability and since, at this point, we know what our data looks like and how we'll be querying it. I think either option would be fine though.

[chappjc]

Be aware that there is much that CockroachDB does not support, although they aim to be compatible with PostgreSQL: https://www.cockroachlabs.com/docs/v2.0/sql-feature-support.html
They only recently added ARRAY and JSONB support. Their JOINs are still not fantastic.

About a year ago I started with CockroachDB and quickly switched to PostgreSQL. Perhaps it is getting closer to prime time for CockroachDB, but why not PostgreSQL?

[lukebp]

@chappjc there was a previous discussion about PostgreSQL that you can check out here:

https://matrix.to/#/!VFRvyndKpzcLrVslQD:decred.org/$153141999337887qcMtI:decred.org?via=decred.org&via=matrix.org&via=zettaport.com