LLaMbda.cpp 🦙🔥

Overview

Learn how to run LLaMA.cpp at scale on AWS Lambda using function URLs and response streaming.

Prerequisites

Make sure the following tools are installed and properly configured:

• Docker 🐋: docker info

• AWS SAM CLI 🐿️: sam --version

• jq (Optional): jq --help

Instructions

0. Set up AWS credentials.

   💡 For more information on how to do this, please refer to the AWS Boto3 documentation (Developer Guide > Credentials).

   # Option 1: (recommended) AWS CLI
   aws configure
   
   # Option 2: environment variables
   export AWS_DEFAULT_REGION=
   export AWS_ACCESS_KEY_ID=...
   export AWS_SECRET_ACCESS_KEY=...
   export AWS_SESSION_TOKEN=...

1. Build and deploy the application.

   # 🏗️ Build
   sam build --use-container
   
   # 🚀 Deploy
   sam deploy --guided
   
   # ❗ Don't forget to note down the function URL
   export FUNCTION_URL=`sam list stack-outputs --stack-name llambda-cpp --output json | jq -r '.[] | select(.OutputKey == "LLaMbdaCppFunctionUrl") | .OutputValue'`

2. Test it out!

   SAM

   sam remote invoke --stack-name llambda-cpp --event '{"body": "{\"message\": \"Explain the theory of relativity.\"}"}'

   cURL

   curl --no-buffer \
        --silent \
        --aws-sigv4 "aws:amz:$AWS_DEFAULT_REGION:lambda" \
        --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
        -H "x-amz-security-token: $AWS_SESSION_TOKEN" \
        -H "content-type: application/json" \
        -d '{"message": "Explain the theory of relativity."}' \
        $FUNCTION_URL