[proxy] Implement compute node info cache

[funbringer]

• Find a decent LRU implementation.
• Implement timed LRU on top of that.
• Cache results of proxy_wake_compute API call.
• Don't invalidate newer cache entries for the same key.
• Add cmdline configuration knobs (requires some refactoring).
• Add failed connection estab metric.
• Refactor auth backends to make things simpler (retries, cache placement, etc).
• Address review comments (add code comments + cleanup).
• Retry /proxy_wake_compute if we couldn't connect to a compute (e.g. stalled cache entry).
• Add high-level description for TimedLru.
• Add cache metrics (hit, spurious hit, miss).
• Synchronize http requests across concurrent per-client tasks ([proxy] Implement compute node info cache #3331 (comment)).
• Cache results of proxy_get_role_secret API call.
• Benchmark and deploy to staging.

[vadim2404]

Closes #3332

[koivunej]

Patch is looking good, also did some digging into surrounding proxy parts, which are looking great. Some comment specific ideas , and question about logging.

I doubt benchmarking would provide too interesting results, because the time to wakeup currently fluctuates and will dominate the time before this patch and with happen every once per ttl with this patch.

[LizardWizzard]

Looks very nice!

AM I right that invalidation happens based on either time or error during compute connection establishment?

[hlinnaka]

If you have a burst of 100 connections to the same database simultaneously, does all the connections race and perform 100 lookups in the console, and then all insert the same value in the cache? It would be better to keep track of which lookups are in progress, so that only one connection performs the lookup, and the others wait for it to finish. That is more efficient.

I don't think that's must-have, just something to consider and maybe add on later.

[funbringer]

I don't think this PR is too big. Most of the changes are straightforward and actually facilitate the review process. For instance, there used to be an ugly auth::backend::console::handle_client which grew completely out of control once I added a few more parameters:

Now everything related to the console API lives in src/console and the rest of the auth backends has been freed from the lower-level boilerplate (http requests and such). We now have a decent console::Api abstraction which has helped me tremendously because it battles the overall complexity.

[LizardWizzard]

Looks good to me. I've had a call with @funbringer discussing the implementation which resulted in a few extra comments in code. The control flow indeed became simpler

I think we can move forward with testing this on staging