Operate on hash table entries directly instead of through keys

[florian1345]

Some performance is lost by making hash lookups in situations where it would not be necessary, such as deleting the LRU element, where we already have a pointer.

At the moment this issue is blocked by the lack of any method to convert raw pointers to hash table buckets. This issue serves as a reminder to check whether any such method is available.

As an alternative, it may be possible to switch from using raw pointers to using buckets.