Slow response on batched pi summaries.

[lukebp]

The pi summaries response on the approved tab of the proposals site is taking up to 5 seconds. This is unacceptably slow and needs to be improved.

https://proposals.decred.org/?tab=approved

[lukebp]

I think I know what the root cause of this is and why it only showed up on production. Anytime data is retrieved from a record in the tstore backend, the full tlog tree is fetched. The tlog tree contains a leaf entry for every piece of record data. The entries are light weight, but when you have 30k votes on a proposals, that is going to still be over 30k leaves in the tlog tree. Prior performance testing revealed that when retrieving the tlog tree, every 10k leaves will add roughly 100ms onto the retrieval response.

The pi summary contains the proposal status, which is a status that is determined at runtime based on various proposal metadata. Proposals in the "Approved" politeiagui tab required two different calls be made to the tstore backend in order to determine their status. The first call retrieves record metadata and the second call retrieves billing status metadata. Each of these calls result in the tlog tree being retrieved. When a tree contains 30k leaves, as a result of having roughly that many DCR ticket votes cast on it, then each tlog tree retrieval is going to take ~300ms. Two separate tree retrievals means that it will take a minimum of 600ms to determine the proposal status of an approved proposal. The politieagui list view fetches the pi summaries in batches of 5. These requests are executed serially by the backend. This means that the tlog tree retrievals alone for this batch of calls is going to take ~3000ms (5 summaries per batch * 2 tree retrievals per summary * 300ms per tree retreival).

This was overlooked during development because the tlog trees in our test enviroments and on test-proposals don't have 30k leaves.

There are two things that we can do to help alleviate this performance bottleneck.

1. Have the backend run batched read requests concurrently.
2. Memoize certain parts of the proposal status call. For example, once a proposal vote has finished, its proposal status can never move backwards, i.e. move back to voting or under discussion. We can take advantage of this fact and write the proposal status function to memoize certain results in memory. This memory cache should be lazy loaded and should have an upper limit for the number of proposals results that it memoizes.

[lukebp]

Performance testing was done using a batch of 5 proposals with the proposal status of "active" and where each proposal had ~1000 tlog leaves. The pi summaries call was taking ~650ms to return when tested locally and without any performance improvements. The combined improvements made in 1584 and 1586 brought this response time down to ~90ms. An 86% improvement.

This should bring the production 5 second response time down to around 1 second.