Agent: Add grpctls proxy API endpoints

This commit is an initial step towards mitigating the risk of untrustworthy host systems when running Kata Containers in the context of confidential computing. It serves to safeguard against malicious, privileged users gaining access to the vulnerable Kata control plane. In scenarios where a malicious cloud service provider or administrator might intercept or compromise commands from the Kata control plane, tamper with container configuration files, execute processes within the container, retrieve workload statistics, or obtain sensitive container workload information, this protective measure becomes crucial.

This commit addresses the following open issues: [Securing the Kata Control Plane](confidential-containers/confidential-containers#53) and RFC: Separate trust realms for tenant and host kata-containers#1834. A detail history can be found in: https://github.com/ray-valdez/kata-containers/tree/split-api-feature.

The commit introduces a new split API mode in the kata-agent, which partitions the kata-agent’s API endpoints between h**ost-side** and **owner-side** controllers. When this mode is enabled, the host-side controller is restricted to manage resource allocation during startup and resource recycling at termination. In contrasts, the owner-side controller allows workload owners to directly manage theIR deployment pod and containers. This partitioning implicitly labels kata-agent’s endpoint APIs as _host-exclusive_, _owner-exclusive_, or _shared_.

Host-exclusive and owner-exclusive APIs are assigned specifically to either the host-side or owner-side. For instance, `CreateSandbox` and `DestroySandbox` are examples of host-exclusive APIs, while `CopyFile` and `ExecProcess` are examples of owner-exclusive APIs. Shared APIs include those that must be shared to some extent between the control planes, such as `GetOOMEvent` and `GetGuestDetails`.

This commit focuses on providing a secure channel for the owner-side to access owner-exlusive  and shared APIs. Future commit(s) will restrict the host-side access to owner-exclusive APIs when split mode is enabled on the kata-agent and will address the sharing of APIs between host-side and owner-side.

This commit implements the following changes:
- Introduces the split mode to the kata-agent.
- Integrates a gRPC TLS server to handle API requests from the owner-side.  We refer to this as the kata-agent’s API proxy server which ensures that workload owners can establish a secure end-to-end communication channel with the kata-agent tor invoking API endpoint commands.
- Utilizes the Key Broker Service (KBS) to provision secrets, i.e., cryptographic public and private  key pairs, These secrets are crucial for establishing a secure communication channel between the owner-side and the API proxy server.

To enable split mode functionality, the following steps are required:

1. Configuration: Modify the `kernel_params` option in the Kata's configuration.toml file to enable split mode and specify the IP address of the KBS.
- Add following settings to the `kernel_params` option:  `agent.split_api=true`  and `agent.aa_kbc_params=cc_kbc::http://[IP_ADDRESS]:[PORT]`.

2. Dependency on KBS: The kata-agent relies on the KBS to provision cryptographic keys to the split API proxy server, facilitating the establishment of a secure channel.

- Generate TLS keys and certificates for  kata-agent’s API proxy server and client (owner-side)

```
$  KATA_DIR=”<PATH to cloned repo>”
$ pushd ${KATA_DIR}/src/agent/grpc_tls_keys
$ ./gen_key_cert.sh
```

 - Create a zip file named 'tls-keys.zip' containing the CA public key and the server’s public and private key pair

` $ zip tls-keys.zip server.pem server.key ca.pem`

  - Place this zip file in the KBS resource path '/default/tenant-keys/'. During sandbox creation, the kata-agent retrieves this file using the KBS 'get resource' API.  It's important to note that the KBS conducts a background check on the key request, verifying evidence provided by the Trusted Execution Environment (TEE). Future extensions to the KBS will automate the creation of the server’s public and private key pair for each sandbox.

` $ popd`

To  exercise the API proxy server, we provide the Kata Containers agent TLS control tool (kata-agent-tls-ctl), derived from the kata-agent-ctl tool in another commit. This tool communicates over a gRPC TLS channel with the kata-agent.  Similar to the kata-agent-ctl, this is a low level tool that is intended for advanced users. Future commit(s) will introduce a more user-friendly tool that maintains state, designed to function as a kubectl plugin for managing owners’ workloads.

Examples of creating and starting a container using kata-agent-tls-ctl:

Setup environment

```
$ export guest_addr=10.89.0.28 # IP address associated with the confidential VM
$ export guest_port=50090         # API proxy server’s port (listens on)
$ export ctl=./target/x86_64-unknown-linux-musl/release/kata-agent-tls-ctl
$ export key_dir=${KATA_DIR}/src/agent/grpc_tls_key
```

Display the status of containers in the sandbox environment

```
$ ${ctl} -l trace connect --key-dir "${key_dir}"  --bundle-dir "${bundle_dir}"  \
--server-address  "ipaddr://${guest_addr}:${guest_port}" \
 -c  "ListContainers"
```

Set a container ID and specify an OCI spec:

```
$ container_id=9e3d1d4750e4e20945d22c358e13c85c6b88922513bce2832c0cf403f065dc6
$ OCI_SPEC_CONFIG=${KATA_DIR}/src/tools/agent-tls-ctl/config.json
```

_Note: the next two commands require pull_image support in the guest!_
**Create container request**
```

$ ${ctl} -l trace connect --key-dir "${key_dir}" --bundle-dir "${bundle_dir}"  \
--server-address  "ipaddr://${guest_addr}:${guest_port}" \
 -c "CreateContainer cid=${container_id} spec=file:///${OCI_SPEC_CONFIG}"
```

**Start container request**

```
$ ${ctl} -l trace connect --no-auto-values --key-dir "${key_dir}" --bundle-dir "${bundle_dir}"  \
--server-address "ipaddr://${guest_addr}:${guest_port}" \
-c "StartContainer json://{\"container_id\": \"${container_id}\"}"
```
Fixes: kata-containers#1834

src/agent/Cargo.toml

@@ -23,6 +23,11 @@ regex = "1.5.6"
 serial_test = "0.5.1"
 kata-sys-util = { path = "../libs/kata-sys-util" }
 kata-types = { path = "../libs/kata-types" }
+prost = "0.11"
+tonic = {version="0.8", features = ["tls"]}
+zip = "0.5.4"
+
+kbs_protocol = { git = "https://github.com/confidential-containers/guest-components.git", rev = "42b7c96", default-features = false,  features = [ "background_check", "openssl" ]}
 
 # Async helpers
 async-trait = "0.1.42"
@@ -73,6 +78,9 @@ reqwest = { version = "0.11.14", optional = true }
 # The "vendored" feature for openssl is required for musl build
 openssl = { version = "0.10.54", features = ["vendored"], optional = true }
 
+[build-dependencies]
+tonic-build = "0.8"
+
 [dev-dependencies]
 tempfile = "3.1.0"
 test-utils = { path = "../libs/test-utils" }

src/agent/grpc_tls_keys/.gitignore

@@ -0,0 +1,4 @@
+*.key
+*.pem
+*.csr
+*.srl

src/agent/grpc_tls_keys/cert.ext

@@ -0,0 +1,7 @@
+authorityKeyIdentifier=keyid,issuer
+basicConstraints=CA:FALSE
+keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost

src/agent/grpc_tls_keys/gen_key_cert.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Create rsa private and public keys 
+#
+CA_KEY=ca.key
+CA_PEM=ca.pem
+
+SERVER_KEY=server.key
+SERVER_CSR=server.csr
+SERVER_PEM=server.pem
+
+CLIENT_KEY=client.key
+CLIENT_CSR=client.csr
+CLIENT_PEM=client.pem
+
+KEY_SIZE=2048
+COUNTRY="AA"
+STATE="Default State"
+LOCALITY="Default City"
+ORG="Default Unit"
+ORG_UNIT="Default Unit"
+CERT_EXT=cert.ext
+
+#1. Create CA key
+openssl genrsa -out ${CA_KEY} ${KEY_SIZE} 
+
+#2. Create self-signed cert valid for ten years
+openssl req -x509 -new -nodes -key ${CA_KEY} -sha256 -days 3650 -out ${CA_PEM} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=grpc-tls CA"
+
+#3. Create server key (kata_agent)
+openssl genrsa -out ${SERVER_KEY} ${KEY_SIZE}
+
+#4. Create cert request
+openssl req -new -sha256 -key ${SERVER_KEY} -out ${SERVER_CSR} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=server" \
+
+# 5. Create server cert
+openssl x509 -req -in ${SERVER_CSR} -CA ${CA_PEM} -CAkey ${CA_KEY} -CAcreateserial -out ${SERVER_PEM} -days 3650 -sha256 -extfile ${CERT_EXT} 2> /dev/null
+
+## Repeat steps 3 - 5 for client (tenant)
+
+#3. Create client key
+openssl genrsa -out ${CLIENT_KEY} ${KEY_SIZE} 
+
+#4. Create cert request
+openssl req -new -sha256 -key ${CLIENT_KEY} -out ${CLIENT_CSR} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=client"
+
+#5. Create cert cert
+openssl x509 -req -in ${CLIENT_CSR} -CA ${CA_PEM} -CAkey ${CA_KEY} -CAcreateserial -out ${CLIENT_PEM} -days 3650 -sha256 -extfile ${CERT_EXT} 2> /dev/null

src/agent/src/config.rs

@@ -25,6 +25,8 @@ const LOG_VPORT_OPTION: &str = "agent.log_vport";
 const CONTAINER_PIPE_SIZE_OPTION: &str = "agent.container_pipe_size";
 const UNIFIED_CGROUP_HIERARCHY_OPTION: &str = "agent.unified_cgroup_hierarchy";
 const CONFIG_FILE: &str = "agent.config_file";
+const AA_KBC_PARAMS: &str = "agent.aa_kbc_params";
+const SPLIT_API_FLAG: &str = "agent.split_api";
 
 const DEFAULT_LOG_LEVEL: slog::Level = slog::Level::Info;
 const DEFAULT_HOTPLUG_TIMEOUT: time::Duration = time::Duration::from_secs(3);
@@ -66,6 +68,8 @@ pub struct AgentConfig {
     pub unified_cgroup_hierarchy: bool,
     pub tracing: bool,
     pub supports_seccomp: bool,
+    pub aa_kbc_params: String,
+    pub split_api: bool,
 }
 
 #[derive(Debug, Deserialize)]
@@ -81,6 +85,8 @@ pub struct AgentConfigBuilder {
     pub passfd_listener_port: Option<i32>,
     pub unified_cgroup_hierarchy: Option<bool>,
     pub tracing: Option<bool>,
+    pub aa_kbc_params: Option<String>,
+    pub split_api: Option<bool>,
 }
 
 macro_rules! config_override {
@@ -142,6 +148,8 @@ impl Default for AgentConfig {
             unified_cgroup_hierarchy: false,
             tracing: false,
             supports_seccomp: rpc::have_seccomp(),
+            aa_kbc_params: String::from(""),
+            split_api: false,
         }
     }
 }
@@ -171,6 +179,8 @@ impl FromStr for AgentConfig {
         config_override!(agent_config_builder, agent_config, passfd_listener_port);
         config_override!(agent_config_builder, agent_config, unified_cgroup_hierarchy);
         config_override!(agent_config_builder, agent_config, tracing);
+        config_override!(agent_config_builder, agent_config, aa_kbc_params);
+        config_override!(agent_config_builder, agent_config, split_api);
 
         Ok(agent_config)
     }
@@ -270,6 +280,18 @@ impl AgentConfig {
                 config.unified_cgroup_hierarchy,
                 get_bool_value
             );
+            parse_cmdline_param!(
+                param,
+                AA_KBC_PARAMS,
+                config.aa_kbc_params,
+                get_string_value
+            );
+            parse_cmdline_param!(
+                param,
+                SPLIT_API_FLAG,
+                config.split_api,
+                get_bool_value
+            );
         }
 
         if let Ok(addr) = env::var(SERVER_ADDR_ENV_VAR) {

src/agent/src/main.rs

@@ -73,6 +73,7 @@ use tokio::{
 };
 
 mod rpc;
+mod secrets;
 mod tracer;
 
 #[cfg(feature = "agent-policy")]
@@ -376,6 +377,31 @@ async fn start_sandbox(
     let mut server = rpc::start(sandbox.clone(), config.server_addr.as_str(), init_mode)?;
     server.start().await?;
 
+    if config.split_api {
+        // downloading and extracting the tls keys for the grpctls server
+        match secrets::retrieve_secrets().await {
+            Ok(_) => println!("main: Success in getting tenant-keys"),
+            Err(e) => {
+                eprintln!("main: Failed to get keys: {:?}", e)
+            }
+        }
+
+        // if the tls keys are downloaded and extracted, then start the grpctls server
+        if secrets::tls_keys_exist() {
+
+            // TBD Remove owner execlusive APIs from the host side
+            // let _ = match config.remove_owner_api() {
+            //     Ok(_) => println!("main: Disable Owner API"),
+            //     Err(e) => { println!("main: Unable to disable Owner API: {:?}", e) }
+            // };
+
+            let gserver =
+                rpc::rpctls::grpcstart(sandbox.clone(), config.server_addr.as_str(), init_mode)
+                    .await?;
+            gserver.await?;
+        }
+    }
+
     rx.await?;
     server.shutdown().await?;
 

src/agent/src/rpc.rs

@@ -92,6 +92,8 @@ use std::path::PathBuf;
 
 use kata_types::k8s;
 
+pub mod rpctls;
+
 pub const CONTAINER_BASE: &str = "/run/kata-containers";
 const MODPROBE_PATH: &str = "/sbin/modprobe";