Deploy Large Language Models on MSE

Blog post associated with this repository: link

Example of a simple MSE application to perform confidential inference on LLM.

The MSE app is composed of the following files:

├── mse_src                   # Code folder to encrypt and deploy in the enclave
│   ├── app.py                # Flask application
│   ├── ggml-model-q4_0.bin   # EleutherAI/pythia-1b model weights
│   └── requirements.txt      # Python packages to install during deployment
└── mse.toml                  # MSE config file

The example mse.toml is using the free hardware provided by Cosmian. More information about config file here.

Here are the steps to follow to deploy your own confidential AI app!

1 - Convert a model from HuggingFace

See convert_model for instructions.

One can also use a custom fine-tuned model converted to GGML format.

Finally, you should copy the resulting model file to ./mse_src:

mse_src/
├── app.py
├── ggml-model-q4_0.bin
└── requirements.txt

2 - Deploy

• Install mse-cli on your computer.

• Test locally

$ mse cloud localtest
...
test_app.py ...

============================================ 3 passed in 3.05s
Tests successful

• Create an account on console.cosmian.com.

• Deploy on MSE

$ mse cloud deploy
...
Deploying your app 'demo-mse-gpt' with 4096M memory and 3.00 CPU cores...
...
💡 You can now test your application: 

     curl https://$APP_DOMAIN_NAME/health --cacert $CERT_PATH

Keep the url and certificate path to perform requests to the MSE app.

• Simple text generation test

curl https://$APP_DOMAIN_NAME/generate --cacert $CERT_PATH
     -H 'Content-Type: application/json' \
     -d '{"query":"User data protection is important for AI applications since"}'

3 - Interact with your application

More ways to interact with the MSE app are shown in clients_example