Serverless LLM Deployment Examples

Welcome to the repository containing a set of hackable examples for serverless deployement of Large Language Models (LLMs). Here, we explore and analyze three services: Modal Labs, Beam Cloud, and Runpod, each abstracting out the deployment process at different levels.

Service	Blogpost	Implementation
Modal Labs	Tutorial Blogpost	Modal Labs Deployment
Beam Cloud	Tutorial Blogpost	Beam Cloud Deployment
RunPod	Tutorial Blogpost	RunPod Deployment

We provide blog posts for each service, as well as dedicated repositories containing full code examples and instructions on how to run and test them.

Test Deployed Model

If you've followed our tutorials and deployed your models using any of the mentioned services, you can test the deployments from here. Please note that testing is currently available only for streaming. However, if you want to make changes, feel free to do so. Before getting started, please install the requirements from here.

pip install -r requirements.txt

Now, assuming you deployed your model using either of the services, you can run test.py as shown below:

For Modal and Beam Cloud:

python3 test.py modal --url <YOUR-DEPLOYED-MODEL/BEAM-URL> --prompt "hello"

For RunPod, you also need to provide the service ID:

python3 test.py modal \
    --url <YOUR-DEPLOYED-RUNPOD-URL> \
    --prompt "hello" \
    --runpod_id <RUNPOD-ID>

Replace <RUNPOD-ID> with a value that looks like this: 80r0eh3jel99f8 (this is an example ID).