LiteLLM Rust Discussion

[narmstrong-berri]

Hey All! This discussion will be used as a place to have community discussion and questions around the Rust migration. I am the lead on the Rust migration, and I will be putting updates here regularly that are much more detailed than the updates in the blog post.

You can follow the progress on the docs. I will also post status updates and plans for the week here every Monday which are a lot more detailed or rough than what would live in a blog post. The target audience is developers interested in following along.

The most recent post is here, and we will have a very fast experience for /v1/messages available by December 1st, with Rust serving the entirety of those requests.

[narmstrong-berri]

Week of Oct 26

Notes

Over the last week, we decided on a different order of operations. We've been scoping out what exactly needs to happen in order to get messages end-to-end in Rust, and have reworked the order of things in order to support that happening as fast as possible. We are tackling the bigger ones first to prove the approach works to migrate a module even without working from the edges of the DAG that constitutes the dependency chain of our codebase.

Our rough approach going forwards will be to migrate modules necessary for the end-to-end path, following this approach:

1. Define a public API for the rest of the codebase to use
2. Remove all references from the rest of the codebase to this module, making everything pass through the public API
3. Create a Rust crate that implements that functionality
4. Rigorously test that the Rust implementation has exactly behavioral parity with the Python implementation.
5. Create a Python shim on top of the Rust implementation which exists in parallel to the Python implementation.
6. Delete the old implementation

Goals

By the start of next week, we plan to have the following things centralized with the business logic in Rust:

• The model catalog
• Redis access
• DB access pattern

We also plan to have the following meaningfully refactored:

• The core router