Using LiteLLM?

[krrishdholakia]

Hi @derp-dev,

Saw you had litellm in your requirements.txt - that's awesome!

I'm the maintainer of LiteLLM. How can I make it better for you?

[derp-dev]

It's really good! It works flawlessly with my cuda12 RTX3080 in wsl2! I use all the models that support cublas and nvidia, they are great. Thanks!

If you want to do anything, go check out AGiXT! I think they could use LiteLLM to a great extent. As-a-matter-of-fact, this repo is only-ever going to be an extension to the logic of that app! I think that artificial generative intelligence (AGI, get it?) is already possible via the 'agent' concept popularized by BabyAGI. My repo is aimed at providing maximum computational throughput for consumers who want to use AGI but don't want to use credit cards and apis, because its really not a good deal as a consumer - 'the more you buy the more you save' as Jensen from Nvidia says.

I think that the fairly insurmountable costs of API usage in the for-profit space scares most people away from this 'dumb' artificial generative intelligence. A concept most impactfully described by extolling that, interestingly, human-thought is actually made-up of a significant amount of hallucination, too... That is to say; hallucination of large foundational models is not 'bad' once you consider 'agentic cognition' expressed in the time dimension - a hallucination is simply another hypothesis to test.

Couple of things I'm researching with my repos:

1. file-system-access with a focus on sharing space (server view of the agent's learned-experience, client view of the users learned experience). Focus on shared-toolkits and shared-problem-solving-sets (and how to define what 'shared' means in any-given scenario and how can the separation of roles play-out over-time?).

2. gnu parallelizing pipelines of 'cognition' - how to juggle dozens of 'conversations' with dozens of services, particularly dealing with the issue of relying on consumer-grade free tools (lots of errors, outages, and just plain-old slow). I'm not a huge fan of asynch or promises so I've been researching any and every possibility.

3. 'bread-crumb-trail' (From 'Hansel And Gretel') for people even worse-off than me getting-into "AGI" because the amount of different software and frameworks and whatnot is super intimidating.

4. rlhf training regimens and context-limited post-training-prompt-training (more of that meta-cognition stuff like "my goal is to yield 'x' from a chat with agent/situation/parameters 'y' so what methodologies 'abcqz' can I use to yield that return?" via things like reverse-prompting, parallelized multi-model piping, prompt-tuning, hyperparameters, parameter-space, argument vectors, coherence and salience in the realm of cognition (for example I love to try to get different models to the same place, creating a convergence, and I love to get many levels-deep in conversation then maintain coherence by meta-analyzing what it would take to get back to a given coherence level).

I think that 'modeling' synthetic/artificial 'cognition' after what we know about human 'cognition' will aid-us in understanding eachother - ourselves, and our creations. For example; I think that "behavior" is an extremely rich area for AGI (generative) to self-analyze, just like ourselves. Another is the concept of subconscious and conscious thought, and even sleeping (dreaming) and cyclic behavior.

Anyways, I hope you check out that repo, it's so, so cool! And nice work on LiteLLM, I'll defiantly tell people about it.

[derp-dev]

ChatAll would be another project that you could really make an impact, on! Its a great app but could use LiteLLM very effectively, I think. It's actually one of my favorite ways to use LLM (single query multiple model).

I'm just a 2nd year, self-taught video game enjoyer, hehe. My repos are really only ever for education.

[krrishdholakia]

Woah @derp-dev you used LiteLLM locally? What for?

[derp-dev]

@krrishdholakia Just LocalAI (api) in a docker container with local networking only, so far, so nothing production ready or particularly interesting, haha. LocalAI (this one, there are two: https://github.com/go-skynet/LocalAI) seems to work well with LiteLLM.

I haven't gotten oogabooga textgen-webui api functionality to work too-well (just in-general, I haven't tried with LiteLLM), but that would probably be my next experiment.

I'll let you know if I have any quandary, thanks! I hope to upload an 'everything' local-docker/kubernetes api which queries all possible options that a consumer has access to. Scope is extremally limited, so thanks to your app I am now diving into the browserless and headless tech I am going to need (inspiration being ChatALL) to wrap into a rest api for all of the free 'webapp' chat endpoints (+litellm to handle local models api and consumer apis).

[krrishdholakia]

looks like local ai itself is a replacement for openai, how're you using litellm with it? (code snippet would be great) would be an awesome tutorial to put out