Local response cache implementation

[SachinVarghese]

This PR introduces basic local response caching as per the issue #1023
The implementation is inspired by the triton equivalent local cache https://github.com/triton-inference-server/local_cache

[SachinVarghese]

PTAL @adriangonz 🙂

[adriangonz]

Nice one @SachinVarghese! It's great to see you back contributing some good stuff to MLServer!

I've added a few comments, but would be good to hear your thoughts on those. Besides that, it would also be great if you could add some tests around the caching behaviour.

[SachinVarghese]

I have made most of the suggested changes now, and all the CI tests should be passing. Please review.
Also, I'm not fully sure how to write the tests for something like this. Any ideas would be good.

[adriangonz]

I think for this one we probably want to add two types of tests:

• Unit tests for the cache itself, which could go under tests/cache/.... These can test the LocalCache implementation, i.e. does it rotate values correctly when it's full, what happens when key is not present, does it save the values, etc.
• Integration tests for the DataPlane class, which can go under the tests/handlers/.... These can test that the DataPlane class create the cache correctly, picks up on cached values, caches new responses, ignores the cache when disabled, etc.

[SachinVarghese]

I have added a few tests for the local cache and data plane component now, please review.

[adriangonz]

Nice one @SachinVarghese ! Thanks for adding those changes in.

This should be good to go from my side. 👍

[SachinVarghese]

Awesome and thanks for reviewing/ almost co-authoring 😄 Will wait for other reviews.