Make Cache More Memory Efficient (#77546) (#77648)

No need to keep counters per segment when we only use them in aggregate.
Also, removed a few spots of using needless indirection by making
segments reference the cache itself directly and saved a bunch of empty maps
for idle caches on e.g. rarely used indices.