Skip to content

mithril-security/blind_llama

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
August 10, 2023 18:59
September 29, 2023 19:27
August 14, 2023 12:17
August 10, 2023 11:39


Making AI Confidential & Transparent

Table of Contents
  1. About the project
  2. Use cases
  3. Advanced security
  4. Vision and roadmap
  5. About us
  6. Contributing
  7. Contact

๐Ÿ“œ About the project

Introduction

๐Ÿ› ๏ธ BlindLlama makes it easy to use open-source LLMs by using Confidential & transparent AI APIs that abstract all the complexity of model deployment while ensuring usersโ€™ data is never exposed to us thanks to end-to-end protection with secure hardware.

๐Ÿ” To provide guarantees to developers that data sent to our managed infrastructure is not exposed, we have developed a Confidential & transparent architecture to serve AI models.

We currently serve Llama2 but will be making more open-source models available in the near future!

Our backend has two key properties:

  • Confidentiality: Your data is never accessible to us. We serve AI models inside hardened environments that do not expose data even to our admins. All points of access, such as SSH, logs, networks, etc., are blocked to ensure the isolation of data.

  • Transparency: We provide you with verifiable cryptographic proof that these controls are in place, thanks to the use of Trusted Platform Modules (TPMs).

Warning
BlindLlama is still under development and does have the full security features.

Do not test our APIs with confidential information... yet!

You can follow our progress towards the next beta and 1.0 versions of BlindLLama on our roadmap.

We welcome contributions to our project from the community! Don't hesitate to raise issues on GitHub, reach out to us or see our guide on how to audit BlindLlama (coming soon!).

(back to top)

๐Ÿš€ Getting started

  • Check out our Quick tour, which will enable you to play with an example using the Llama 2 model while ensuring your data remains private and without the hassle of provisioning!
  • Find out more about How we protect your data
  • Refer to our Concepts guide for more information on key concepts
  • Learn more about BlindLlama's design here guide

(back to top)

Architecture

BlindLlama is composed of two main parts:

  • An open-source client-side Python SDK that verifies the remote AI models we serve are indeed guaranteeing data sent is not exposed to us.
  • An open-source server that serves models without exposing any user data to the AI provider (Mithril security). This is achieved by hardening the server components and removing any potential leakage channels from network to logs. We provide cryptographic proof those privacy controls are indeed in place using TPMs.

The client performs two main tasks:

  • Verifying that the server it communicates with is the expected hardened AI server using attestation.
  • Securely sending data to be analyzed by a remote AI model using attested TLS to ensure data is not exposed.

The server has two main tasks:

  • It loads a hardened AI server which is inspected to ensure no data is exposed to the outside.
  • It serves models using the hardened AI server that can be remotely verified using attestation.

Note that there are three key components behind what we call the "server" here. You can find out more about each of these components and how they interact in our docs.

Trust model

On this page, we will explain more precisely what components/parties have to be trusted when using BlindLlama.

To understand better which components and parties are trusted with BlindLlama, letโ€™s start by examining what is trusted with regular AI services.

To do so, we will use the concept of a Trusted Computing Base (TCB), which refers to the set of all hardware, firmware, and software components that are critical to a system's security.

Trusted Computing Base with regular AI providers

In the case of an AI provider serving an AI API to end users on a Cloud infrastructure, the parties to be trusted are:

  • The AI provider: they provide the software application that is in charge of applying AI models to usersโ€™ data. Examples of AI providers in the industry include Hugging Face, OpenAI, Cohere, etc.

  • The Cloud provider: they provide the infrastructure, Hypervisor, VMs and OS, to the AI provider. Examples of Cloud providers in the industry include Azure, GCP, AWS, etc.

  • The hardware providers: they provide the physical components, CPU, GPU, TPMs, etc. to the Cloud provider. Examples of hardware providers in the industry include Intel, AMD, Nvidia, etc.

The higher the party in the stack, the closer they are to the data. Thus, the AI provider if malicious or negligent represents the biggest security risk for the user of the API.

In most scenarios today, there is often blind trust in the AI provider, aka we send data to them without any technical guarantees regarding how they will handle or use our data. For instance, the AI provider could say they just do inference on data, while they could actually train models on usersโ€™ data. And even if most AI providers are honest, there is no way to know if their security practices are strong enough to protect your data.

For privacy-demanding users that require more technical guarantees, they often choose simply not to use AI APIs as they cannot trust AI providers with their confidential data.

Trusted parties with BlindLlama

With BlindLlama, we remove the AI provider (Mithril Security) from the list of trusted parties. When models are served with BlindLlama, our admins cannot see user data because we use a Confidential & transparent AI infrastructure, removing the need for users to blindly trust us.

We can prove such controls are in place using TPM-based attestation.

trust-model-dark trust-model-light

See our section on BlindLlama's Trusted Computing Base (TCB) to see which components we trust or verify in our stack!

๐Ÿ‘ฉ๐Ÿปโ€๐Ÿ’ป Use cases

BlindLlama is meant to help developers working with sensitive data to easily get started with LLMs by using managed AI APIs that abstract the hardware and software complexity of model deployment while ensuring their data remains unexposed.

Several scenarios can be answered by using BlindLlama, such as:

  • Benchmarking the best open-source LLMs against oneโ€™s private data to find out which one is the most relevant without having to do any provisioning
  • Structuring medical documents
  • Analysis or auto-completion of a confidential code base

โœ… When should you use BlindLlama?

  • You want to get started with LLMs that are complex to deploy, such as Llama 2 70B
  • You donโ€™t want to manage that infrastructure as it requires too much time, expertise and/or budget
  • You donโ€™t want to expose your data to a third party AI provider that manages the infrastructure for you due to privacy/compliance issues

โŒ What is not covered by BlindLlama?

  • BlindLlama is simply a drop-in replacement to query a remotely hosted model instead of having to go through complex local deployment. We do not cover training from scratch, but we will cover fine-tuning soon.
  • BlindLlama allows you to quickly and securely leverage models which are open-source, such as Llama 2, StarCoder, etc. Proprietary models from OpenAI, Anthropic, and Cohere are not supported yet as we would require them to modify their backend to offer a Confidential & transparent AI infrastructure like ours.
  • BlindLlamaโ€™s trust model implies some level of trust in Cloud providers and hardware providers since we leverage secure hardware available and managed by Cloud providers (see our trust model section for more details).

BlindLlama virtually provides the same level of security, privacy, and control as solutions provided by Cloud providers like Azure OpenAI Services.

(back to top)

๐Ÿ“š Advanced security

We created the BlindLlama whitepaper to cover the architecture and security features behind BlindLLama in greater detail.

The whitepaper is intended for an audience with security expertise.

You can read or download the whitepaper here!

๐ŸŽฏ Roadmap

There are three key milestones planned for the BlindLlama project.

BlindLlama Alpha launch (not attestable):

The alpha launch of BlindLlama provides a regular API for the Llama2-70b model which you can query with our python SDK.

Users can test out and query our API but should not yet send any confidential data to the API as it is does not yet have full implementation of security features.

The server-side code already includes the backbones for our attestation feature (which will enable us to be able to prove the server is deploying the expected code to end users) but this feature will be fully launched in the following beta phase.

Expected launch date: week ending 08/09/2023

BlindLlama Beta launch (with attestation):

The beta version adds the full implementation of TPM-based attestation, meaning our APIs can be fully verified remotely. This version will not yet have full hardening of server-side environment or audit and thus is not yet recommended in production!

Provisional launch date: week ending 06/10/2023

BlindLlama 1.0 launch (audit-ready):

A fully-secure version of BlindLlama ready for audit, with a fully hardened server environment.

Provisional launch date: week ending 08/12/2023

You can check out more details about these stages and our progress to achieveing these milestones on our official roadmap.

(back to top)

๐Ÿฆ™ Who made BlindLlama?

BlindLlama is developed by Mithril Security, a startup focused on democratizing privacy-friendly AI using secure hardware solutions.

We have already had our first project, BlindAI, an open-source Rust inference server that deploys ONNX models on Intel SGX secure enclaves, audited by Quarkslab.

BlindLlama builds on the foundations of BlindAI but provides much faster performance and focuses on serving managed models directly to developers instead of helping AI engineers to deploy models.

(back to top)

๐Ÿค Contributing

Hereโ€™s how you can help us make AI confidential:

๐Ÿ› ๏ธ Code contribution

You can contribute our code by forking our project on GitHub and creating a new pull request. Make sure to detail the modifications you are suggesting in your pull request description.

๐ŸŒŽ Spread the word

Share our project on social media!

share-on-twitter share-on-fb share-on-reddit share-on-linkedin

(back to top)

๐Ÿ“‡ Get in touch

We would love to hear your feedback or suggestions, here are the ways you can reach us:

Want to hear more about our work on privacy in the field AI?

  • Check out our blog
  • Subscribe to our newsletter here

Thank you for your support!

(back to top)