Cache repeated parameter datatype queries for the same model

[isidentical]

What does this PR do?

Fixes #4975. Adds a generic LRU cache for the expensive parameter datatype computation function with an optional fallback to the uncached implemnetation in cases where the underlying module is not hashable (since this can be any torch module subclass, it's possible that it might have overriden the __hash__ or added properties which make it unsafe to hash). From my local testing, the perf increase is very noticable (2X to 3X) more benchmarks are below.

Benchmarks

For the following script which loads the LoRA state dict into memory and then runs load_lora_weights/unload_lora_weights cycle 5 times, the results are as following:

	min	max	mean	median	mean speed-up
baseline (main)	5.44s	6.46s	6.02s	6.31s	1.0x
This PR	1.84s	2.05s	1.96s	1.94s	3.2x

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[patrickvonplaten]

I'm sadly not seeing a big speed-up anymore (after having merged: #4994). Could you maybe double check once quickly if you still see the same speed-up with the current version of main?

[isidentical]

Yeah, my own benchmarks also show no noticable speed up with this PR on (I think this was because of non-meta devices and my sequence of optimizations also reduced the need for this). Closing this PR as its currently very good as is!