adding litellm integration into toolkit ENG-1086

[r0ymanesco]

This PR adds our litellm integration PR into the ND python SDK. I implemented this in the most brute forced way by copying the main.py file in litellm directly into our toolkit module. This is because litellm is implemented purely modularly without any classes so I couldn't just inherit a class and modify some methods. The up side is that this is a complete replacement for litellm. A user can simply use it as they are using the regular litellm with the added benefit of supporting notdiamond.

[acompa]

I generally worry about bundling external code with an SDK wholesale in this manner. Let's work together during next sprint to find a lightweight path to introducing litellm support without bringing over their main.py or __init__.py files.

[acompa]

I missed this in Devansh's PR, but we should probably use @pytest.mark.parametrize here to separate these tests (and optionally run them in parallel via pytest -n, see details).

[r0ymanesco]

I generally worry about bundling external code with an SDK wholesale in this manner. Let's work together during next sprint to find a lightweight path to introducing litellm support without bringing over their main.py or __init__.py files.

I think one way to reduce copied code is to only copy completion and acompletion from litellm.main and modify it to support notdiamond. The rest can be "copied" by from litellm.main import *

[acompa]

@r0ymanesco Do we still need the enormous main.py file? I do see a star import from main, but would hope to avoid that. Could we instead import * directly from litellm.main, since the library is a dependency?

[r0ymanesco]

@r0ymanesco Do we still need the enormous main.py file? I do see a star import from main, but would hope to avoid that. Could we instead import * directly from litellm.main, since the library is a dependency?

This is the minimum amount of main.py that we need to keep. It's just 3 methods but those 3 methods are over 2000 lines long

[acompa]

@r0ymanesco Do we still need the enormous main.py file? I do see a star import from main, but would hope to avoid that. Could we instead import * directly from litellm.main, since the library is a dependency?

This is the minimum amount of main.py that we need to keep. It's just 3 methods but those 3 methods are over 2000 lines long

Yep, I actually spent part of my flight up north reviewing this further. I agree. Approving to unblock you, and we can chat with the LiteLLM team about this contribution experience.