[RFC] Generic Key Broker Service (KBS) & Attestation Service high level architecture proposal

[hdxia]

Authors

• Liang Zhou

• Jia Zhang

• Haidong Xia

Motivation

CC enables container confidential computing with image at-rest protection using encryption and signing, and runtime protection with hardware-based trust execution environment (TEE).

One of the important tasks in the CC architecture is for the attestation agent to fetch image decryption key (DEK) from a key broker service (KBS) to decrypt the container image. It is also expected that the KBS only releases the DEK to the attestation agent after the attestation of the workload execution environment. Therefore, key broker service and attestation service are fundamental to the end to end (E2E) flow of CC. However, both components are not part of CC yet.

The CC's Attestation-Agent has implemented the modular KBC design that is compatible with any vendor-specific attestation service. Generally, a vendor can implement a completely customized attestation service according to their special requirements. But the CC community still lacks a complete reference design of the attestation system that includes all the RATS defined roles of the Attester, Verifier, and Relaying Party presently. The lack of reference design causes confusion and the inconvenience for potential CC users to use this confidential container solution.

The Attestation-Agent has already implemented the role of Attester, so this proposal focuses on Verifier and Relaying Party high level architecture design, per RATs architecture specification.

Background - RATS Architecture

IETF has defined RATS architecture defining different entities involved in attestation and use of the attestation result. It is best to understand it first to avoid any confusion and misuse of terminologies.

The following is the reference architecture from RATS[1, Architectural Overview]:

The definition of each entity are as follows, in the context of CC.

• Attester: Fetches HW-TEE's Evidence and sends it to Verifier.
• Verifier: Executes the following steps to verify the Evidence and send the corresponding Attestation Results to the Relying Party:
  • Invoke Endorser to verify the Evidence's identity.
  • Invoke both Verifier Owner and Reference Value Provider to verify the TCB status inside Evidence.
• Relying Party: Parses and checks the Attestation Results and provides some services.

Goal

The goal of this proposal is to

• Provide a generic KBS and attestation service with high level architecture to be aligned with standard RATS reference architecture.
• Decouple the Attestation Service from the hardware agnostic KBS for modularity and flexibility.
• Define industry standard interaction model/protocol between Attestation-Agent and KBS, and KBS and Attestation Service.

High Level Architecture of KBS & Attestation Service

Per RATs architecture specification, we propose to have a KBS and attestation service. KBS implements the Relying Party, and Attestation Service implements the Verifier in RATs architecture. They are decoupled as two separate services/modules to make KBS a hardware agnostic component for key policy and key management. The whole architecture aligns with RATS Background-Check Model[1, Topological Patterns].

The interaction procedures should be:

• Attestation-Agent sends a request to KBS then KBS will decide whether an attestation is needed.
• If the attestation is needed, KBS will reply a Challenge to Attestation-Agent.
• Then Attestation-Agent will send the Evidence to KBS.
• KBS will relay the Evidence to Attestation Service for verification.
• Attestation Service will reply the Attestation Results to KBS after verification is finished.
• If the Attestation Results are verified by KBS is passed, it will execute the corresponding resource fetch operation (such as Get KEK) and send it to Attestation-Agent.

TODO: A separate proposal to introduce these interaction message format in details.

KBS (Relying Party)

KBS is hardware agnostically and it includes the following features:

• Relay Evidence to Attestation Service for verification.
• Apply appraisal policy for the returned Attestation Results to assess the trustworthiness of the Attestation-Agent.
• Return responses, such as Get KEK, Get Resources, Inject secret, and so on, to Attestation-Agent.
• Support modularization of KMS.

A reference architecture and implementation of KBS is illustrated in figure below. Detailed discussion of KBS design will be in its own key broker service repo.

TODO: A separate proposal to discuss the KBS architecture in details.

Attestation Service (Verifier)

Attestation Service includes the following features:

• Support multiple HW-TEEs with modularization design.
• Validate the HW-TEE Evidence identity.
• Apply Appraisal Policy for Evidence to assess the trustworthiness of the Attestation-Agent.
• Return Attestation Results to KBS.

A high architecture of Attestation service is illustrated in the Figure below. Note the detailed design discussion of attestation service will be in its repo.

TODO: A separate proposal to discuss the Attestation Service architecture in details.

References

• [1] [https://www.ietf.org/archive/id/draft-ietf-rats-architecture-15.txt]

[peterzcst]

This is awesome

[jiazhang0]

@larrydewey @fitzthum

[fitzthum]

This is a really high quality overview. Thanks for posting.

Like I said in the meeting, I think this is on the right track. My main questions are about the best way to engineer a solution that is robust and well-integrated.

I also have some thoughts about policies. In the diagrams above you are evaluating policies in two different locations. First, there is the policy engine in the KBS that determines whether a key should be released. A simple example here would be to release a key if the attestation service approves the evidence. There is also policy evaluation in the attestation service, which is used to determine whether the evidence is valid. For example, the attestation service might only approve evidence that includes a particular launch measurement.

The question is whether it makes sense to separate these two things. What if I want to deploy a workload only to pods that have a particular configuration of the Kata Agent? The agent config file is part of the launch measurement, so this could be evaluated by a policy in the Attestation Service, but really we want this to be taken into account by the key policy of the KBS. In some ways I think splitting the policy evaluation in two is wise. The policy of the attestation service is platform specific and it might be tough to map platform-specific characteristics directly to keys. On the other hand this might limit some of the policies we can enforce. Maybe a compromise would be to have the AS policy return more than just approved or rejected. It's possible that each guest owner could specify multiple levels of AS policy for different platforms and then define KBS policies that depend on these levels.

I am also curious about plans for signature support, which isn't really described here. In some ways the public keys for signature validation are just like any other secret. On the other hand, we've talked about having the signature policy file be dynamically generated by the KBS. Where would something like that fit in?

Finally, we have often overlooked what happens when the guest owner provisions images. While it isn't too complex, it might be nice to add some information about how and when keys are populated. It also might good to have a tool that the guest owner can use as a keyprovider when they encrypt their images, which would automatically populate the KBS. Currently it looks like setup would involve uploading keys to a KMS, policies to the KBS, and AS, and some configuration to tie everything together. How can we make this easy to use?

[liangzhou121]

The question is whether it makes sense to separate these two things.

The reason to separate the evaluating policies in two different locations is the KBS and AS will be two separate services/modules. We expected they will have two separate repos.

What if I want to deploy a workload only to pods that have a particular configuration of the Kata Agent? The agent config file is part of the launch measurement, so this could be evaluated by a policy in the Attestation Service, but really we want this to be taken into account by the key policy of the KBS.

For this scenario, the AS's policy engine can evaluate the launch measurement firstly, then AS can response a Attestation Result to KBS that also includes this launch measurement. KBS can execute some evaluation base on the received Attestation Result also.
The Attestation Result should be a JSON string and includes multiple messages (not just approved or rejected), such as the TCB info inside the Evidence. We plan to define both Attestation Result and Evidence messages' details inside AS's specific arch design proposal.

It's possible that each guest owner could specify multiple levels of AS policy for different platforms and then define KBS policies that depend on these levels.

This is the reason why we choose the OPA as policy engine because it supports load policy files dynamically. It can select different policy.rego file to execute the evaluation accoding to the TEE's type inside Evidence. As example: use the tdxpolicy.rego to evalute the TDX Evidence, use sevpolicy.rego to evaluate SEV Evidence.

[hdxia]

Good questions @fitzthum and I think @liangzhou121 addressed most. I just want to add two points.

1. Separation of KBS and attestation service is for the flexibility. KBS is a relying party, and attestation service is the verifier in RATS architecture. They are not 1:1 mapping. so once you have an attestation service, there could be many relying parties (not just KBS) can take advantages of it. so the implementation here will be more generic for other purpose, not limited to the features listed here. In addition, we try to avoid a monolithic architecture.

2. regarding the management of key, and policies, we can discuss the details in its own repo. but in short, the KBS can provide an API to create or import an key (into the backend KMS) and add a key policy in itself.

[larrydewey]

Thank you @liangzhou121, @jiazhang0, and @hdxia for your presentation. You presented some very solid arguments and resources. For the majority of the higher-level discussion, I am in agreement; especially in regard to using OPA. However, I have a number of questions/reservations around adopting ISecL and verdictd.

A few of these include:

• Current there is no support for SEV(-ES, -SNP), which would break our demo and the existing test framework, as support is missing.
• This would introduce a large number of external dependencies, all or most of which are not written in Rust. Instead of bringing us closer to a community goal of one unified language, this would pull us much farther away.
• What does the long-term maintenance road-map look like once these projects have been forked? Are there plans for including upstream/external improvements and bug fixes?

[hdxia]

Thanks @larrydewey for the comments

• Current there is no support for SEV(-ES, -SNP), which would break our demo and the existing test framework, as support is missing.

the idea of the generic attestation service is to have modules to support different HW technologies. in case it is missing there, we would expect the community to develop there. before that, current demo and test framework for SEV should not change. KBS is HW independent.

• This would introduce a large number of external dependencies, all or most of which are not written in Rust. Instead of bringing us closer to a community goal of one unified language, this would pull us much farther away.

Programming language is debatable. My personal opinion is the node components have rust, but services outside of the node should be not constrained. We can start with what we have to streamline the process. however, there is no constraint to create another repo in rust if there is a need later on.

• What does the long-term maintenance road-map look like once these projects have been forked? Are there plans for including upstream/external improvements and bug fixes?

the details on roadmap and long term can be discussed in their respective repo. for KBS, I think we can fork, and maintain it here from long term perspective.

[liangzhou121]

Thanks @larrydewey @hdxia comments, and I have some other comments from Attestation Service's side.

Current there is no support for SEV(-ES, -SNP), which would break our demo and the existing test framework, as support is missing.

As haidong mentioned, the Attestation Service supported HW-TEEs can be extended conveniently by its modularization design. And we also have plan to add the SEV(-ES, -SNP) supporting to the general AS after TDX and SGX related development finished.

This would introduce a large number of external dependencies, all or most of which are not written in Rust. Instead of bringing us closer to a community goal of one unified language, this would pull us much farther away.

From AS side, it won't introduce lots of external dependencies. We plan to donate the the inclavared-container's verdictd project into CC Attestation Service repo. And the verdictd is developed by RUST.

What does the long-term maintenance road-map look like once these projects have been forked? Are there plans for including upstream/external improvements and bug fixes?

From AS side, as we plan donate verdictd to CC entirely and will use it as the master branch, so we will keep development in this repo to support potential new types of HW-TEE and fix bugs.

[larrydewey]

the idea of the generic attestation service is to have modules to support different HW technologies. in case it is missing there, we would expect the community to develop there. before that, current demo and test framework for SEV should not change. KBS is HW independent.

The idea of having a HW agnostic, generic attestation services definitely makes the most sense. However, it would be good to discuss a path forward for make the required changes to our existing examples to have the Generic KBS work. So I think additional research here is definitely warranted.

Programming language is debatable. My personal opinion is the node components have rust, but services outside of the node should be not constrained. We can start with what we have to streamline the process. however, there is no constraint to create another repo in rust if there is a need later on.

I don't know that I agree the language should be debatable. There are more implications to consider than resource consumption, though that is an important point, too. Making a new repository may be easy, correct, but the refactoring of an entire project is not a small undertaking.

the details on roadmap and long term can be discussed in their respective repo. for KBS, I think we can fork, and maintain it here from long term perspective.

If the suggestion here is that CC should retain full ownership and maintenance of a fork (or forks), wouldn't that place a significant maintenance load on the community?

[larrydewey]

As haidong mentioned, the Attestation Service supported HW-TEEs can be extended conveniently by its modularization design. And we also have plan to add the SEV(-ES, -SNP) supporting to the general AS after TDX and SGX related development finished.

Where there is already support for SEV and SEV-ES in the repository, this proposal would be missing support for an already supported architecture, this is what I mean by breaking the existing functionality and test cases. It introduces multiple hard regressions.

From AS side, it won't introduce lots of external dependencies. We plan to donate the the inclavared-container's verdictd project into CC Attestation Service repo. And the verdictd is developed by RUST.

From the Attestation Service side of things, yes, verdictd is written in Rust, but the rest of ISecL is not. This in particular was what I was referring to by introducing a large number of external dependencies.

From AS side, as we plan donate verdictd to CC entirely and will use it as the master branch, so we will keep development in this repo to support potential new types of HW-TEE and fix bugs.

I would echo my question from my last response:

If the suggestion here is that CC should retain full ownership and maintenance of a fork (or forks), wouldn't that place a significant maintenance load on the community?

[sphrasavath]

Agree with @fitzthum here, further discussion is needed here.
I get wanting to decouple of KBS from the attestation functionality, but there are many unknown still.
My understanding is that the KBS service should managed just keys whether that is a plugin to some third-party KMS or user defined values such as expected measurements.
As for the KBS in this proposal:
• So where would user defined (expected measurements) resides, inside the Key policy engine or would that be treated as a plugin?
• It would be useful if the API layer of the KBS is a RESTful so that it can be initiated not only by the attestation-agent by the agent owner as well. I don’t have experience with RATS arch so I don’t know if REST could be supported here?
• What is the attestation client, just a proxy to connect to the attestation service interface? Is there any logic or decision-making pertaining to keys in that component?
As for attestation service:
Please clarify, based on the Attestation Service diagram in issue#30, is this service running on the host system or could it be anywhere (cloud or some on-prem server)?
I am assuming it’s running on TEE host b/c the diagram has it talking to the various HW encryption engine. If yes, isn’t that replicating the job of the attestation agent to communicate with the underlying HW to get the attestation report.
If the Attestation Service is something completely independent to HW env where the TEE runs, then I see a different picture of how currently working TEE env “could fit” into this framework. Btw, that’s not an endorsement b/c we still need to talk through this. 😊
That new understanding is that the Simple KBS, @fitzthum and @larrydewey please correct me here, could be treated as a verification driver in the attestation service. The expected measurement now resides in the Attestation service and not the KBS.
Which means, the KBS could be treated as generic?
If this proposal is for a generic KBS & Attestation Service then I think we need to demonstrate that with something that is currently working, like the simple KBS. For instance, in your KBS proposal we should be able to turn OFF all the items we don’t care about for now, such as the plugins. And for the AS, we should be able to turn off the proxy if we don’t need it. How do you plug the simple KBS and treat is as verification driver? What’s the endorser and why is it needed?
Also there are additional questions here other than architectural, such as languages (go or no go?) and interfaces?

[fitzthum]

btw can you clarify the relationship between verdictd and rats-tls? Is rats-tls part of this proposal?

[jiazhang0]

btw can you clarify the relationship between verdictd and rats-tls? Is rats-tls part of this proposal?

rats-tls is not part of this general AS. The general AS will employ a new subsystem with so-called verifier drivers to support various TEEs.

[jiazhang0]

the idea of the generic attestation service is to have modules to support different HW technologies. in case it is missing there, we would expect the community to develop there. before that, current demo and test framework for SEV should not change. KBS is HW independent.

The idea of having a HW agnostic, generic attestation services definitely makes the most sense. However, it would be good to discuss a path forward for make the required changes to our existing examples to have the Generic KBS work. So I think additional research here is definitely warranted.

Programming language is debatable. My personal opinion is the node components have rust, but services outside of the node should be not constrained. We can start with what we have to streamline the process. however, there is no constraint to create another repo in rust if there is a need later on.

I don't know that I agree the language should be debatable. There are more implications to consider than resource consumption, though that is an important point, too. Making a new repository may be easy, correct, but the refactoring of an entire project is not a small undertaking.

the details on roadmap and long term can be discussed in their respective repo. for KBS, I think we can fork, and maintain it here from long term perspective.

If the suggestion here is that CC should retain full ownership and maintenance of a fork (or forks), wouldn't that place a significant maintenance load on the community?

Please let me clarify this. I think Liang's meaning is the general AS as part of CC's project is inspired from verdictd. Verdictd and general AS will be separate. Verdictd is designed as a combination of KBS and AS, and it is connected to AA through EAA KBC. This is the initial implementation for CCv0. So no external dependency and fork efforts are maintained by CC community.

[jiazhang0]

As haidong mentioned, the Attestation Service supported HW-TEEs can be extended conveniently by its modularization design. And we also have plan to add the SEV(-ES, -SNP) supporting to the general AS after TDX and SGX related development finished.

Where there is already support for SEV and SEV-ES in the repository, this proposal would be missing support for an already supported architecture, this is what I mean by breaking the existing functionality and test cases. It introduces multiple hard regressions.

Do you mean the support for SEV and SEV-ES occurs in rats-tls repo?

From AS side, it won't introduce lots of external dependencies. We plan to donate the the inclavared-container's verdictd project into CC Attestation Service repo. And the verdictd is developed by RUST.

From the Attestation Service side of things, yes, verdictd is written in Rust, but the rest of ISecL is not. This in particular was what I was referring to by introducing a large number of external dependencies.

I think we could leave this question to @hdxia . I think this problem doesn't affect the progress of general AS right?

[liangzhou121]

Hi @sphrasavath , Thank you for your comments.

Please clarify, based on the Attestation Service diagram in issue#30, is this service running on the host system or could it be anywhere (cloud or some on-prem server)?

The Attesattion Service can be run in anywhere. It's a HW-TEE environment independent service but it needs to recognize different types HW-TEE's Evidence.

If this proposal is for a generic KBS & Attestation Service then I think we need to demonstrate that with something that is currently working, like the simple KBS.

From Attestation Service side, as @jiazhang0 mentioned, it's inspired from the existing verdictd project.

What’s the endorser and why is it needed?

This role is defined by RATS reference architecture that used to verify the HW-TEE evidence's identity. Such as the Intel PCS service.

And for the AS, we should be able to turn off the proxy if we don’t need it. How do you plug the simple KBS and treat is as verification driver?
Also there are additional questions here other than architectural, such as languages (go or no go?) and interfaces?

These are design details and It's better to discuss them in the Attestation Service's dedicated proposal. But I think it shouldn't impact the progress of Attestation Service.

[hdxia]

From the Attestation Service side of things, yes, verdictd is written in Rust, but the rest of ISecL is not. This in particular was what I was referring to by introducing a large number of external dependencies.

I think we could leave this question to @hdxia . I think this problem doesn't affect the progress of general AS right?

I think there may be misunderstanding of KBS vs ISecL. ISecL itself supports many use cases and has many components. KBS is one of the components as a solution that ISecL provides. so KBS != ISecL. for CC project, we propose to only move or fork KBS component from ISecL to CC, and we will remove other dependencies to ISecL. So KBS itself will be standalone in CC.

[fitzthum]

@hdxia

I think the most important question is about the nature of the KBS fork. If we fork the KBS, will there still be a KBS in the ISecL repo? Are we going to try to keep them in sync? Do you expect any developers from ISecL to contribute to the fork?

[hdxia]

@hdxia

I think the most important question is about the nature of the KBS fork. If we fork the KBS, will there still be a KBS in the ISecL repo? Are we going to try to keep them in sync? Do you expect any developers from ISecL to contribute to the fork?

@fitzthum I am sure developers from ISecL will contribute and help maintain/sync the KBS in CC. the 1st step is to fork KBS from ISecL and make it self-contained, and contribute to CC, and we will maintain it.

[surajssd]

This image shows that Attester talks to Verifier and similarly Relying Party / KBS also talks to Verifier, does that also imply that Attester and Relying Party / KBS never communicate?

---

If they don't communicate directly then this diagram shows that Attester directly communicates with the Relying Party / KBS. What is really going on? So which one is accurate?

[liangzhou121]

Hi @surajssd ,
The RATS arch has two models:

• passport model: The attester communicates with Verifier.
• background model: The attester communicates with Relying Party.

And Attestation Service's design chooses the background model.

[surajssd]

@liangzhou121 thanks for the clarification!

[fitzthum]

Are you still planning to use the ISecL KBS? We're starting to make progress on the AS thanks to @liangzhou121 but I haven't seen much happening with the KBS. @hdxia @jiazhang0

[ariel-adam]

@hdxia is this issue still relevant or can be closed?
If it's still relevant to what release do you think we should map it to (mid-November, end-December, mid-February etc...)?