Agent: Add grpctls proxy API endpoints

This commit is an initial step towards mitigating the risk of
untrustworthy host systems when running Kata Containers in the context
of confidential computing. It serves to safeguard against malicious,
privileged users gaining access to the vulnerable Kata control
plane. This protective measure becomes crucial in scenarios where a
malicious cloud service provider or administrator might intercept or
compromise commands from the Kata control plane, tamper with container
configuration files, execute processes within the container, retrieve
workload statistics, or obtain sensitive container workload
information.

** Problem statement **

This commit addresses the following open issues:
[Securing the Kata Control Plane](confidential-containers/confidential-containers#53)
and RFC: Separate trust realms for tenant and host kata-containers#1834. A detailed
history can be found in:
https://github.com/ray-valdez/kata-containers/tree/split-api-feature.

** Architectural changes **

The commit introduces a new split API mode in the kata-agent, which
partitions the kata-agent’s API endpoints between **host-side**
and **owner-side** controllers. When this mode is enabled, the
host-side controller is restricted to manage resource allocation
during startup and resource recycling at termination. In contrasts,
the owner-side controller allows workload owners to directly manage
theIR deployment pod and containers. This partitioning implicitly
labels kata-agent’s endpoint APIs as _host-exclusive_,
_owner-exclusive_, or _shared_.

Host-exclusive and owner-exclusive APIs are assigned specifically to
either the host-side or owner-side. For instance, `CreateSandbox` and
`DestroySandbox` are examples of host-exclusive APIs, while `CopyFile`
and `ExecProcess` are examples of owner-exclusive APIs. Shared APIs
include those that must be shared to some extent between the control
planes, such as `GetOOMEvent` and `GetGuestDetails`.

** Content of this commit **

This commit focuses on providing a secure channel for the owner-side
to access owner-exlusive and shared APIs. Future commit(s) will
restrict the host-side access to owner-exclusive APIs when split mode
is enabled on the kata-agent and will address the sharing of APIs
between host-side and owner-side.

This commit implements the following changes:
- Introduces the split mode to the kata-agent.
- Integrates a gRPC TLS server to handle API requests from the
  owner-side.  We refer to this as the kata-agent’s API proxy server
  which ensures that workload owners can establish a secure end-to-end
  communication channel with the kata-agent or invoking API endpoint
  commands.
- Utilizes the Key Broker Service (KBS) to provision secrets, i.e.,
  cryptographic public and private key pairs. These secrets are crucial
  for establishing a secure communication channel between the owner-side
  and the API proxy server.

** Testing **

To enable split mode functionality, the following steps are required:

1. Configuration: Modify the `kernel_params` option in the Kata's
  configuration.toml file to enable split mode and specify the IP
  address of the KBS.
- Add following settings to the `kernel_params` option:
  `agent.split_api=true` and
  `agent.aa_kbc_params=cc_kbc::http://[IP_ADDRESS]:[PORT]`.

2. Dependency on KBS: The kata-agent relies on the KBS to provision
   cryptographic keys to the split API proxy server, facilitating the
   establishment of a secure channel.

- Generate TLS keys and certificates for kata-agent’s API proxy server
  and client (owner-side)

```
$  KATA_DIR=”<PATH to cloned repo>”
$ pushd ${KATA_DIR}/src/agent/grpc_tls_keys
$ ./gen_key_cert.sh
```

 - Create a zip file named 'tls-keys.zip' containing the CA public key
   and the server’s public and private key pair

` $ zip tls-keys.zip server.pem server.key ca.pem`

  - Place this zip file in the KBS resource path
  '/default/tenant-keys/'. During sandbox creation, the kata-agent
  retrieves this file using the KBS 'get resource' API.  It's
  important to note that the KBS conducts a background check on the
  key request, verifying evidence provided by the Trusted Execution
  Environment (TEE). Future extensions to the KBS will automate the
  creation of the server’s public and private key pair for each
  sandbox.

` $ popd`

** External tools required for testing **

To  exercise the API proxy server, we provide the Kata Containers
agent TLS control tool (kata-agent-tls-ctl), derived from the
`kata-agent-ctl` tool in another commit (see `split-api-feature` branch
referenced above).

This tool communicates over a gRPC TLS channel with the kata-agent.
Similar to the kata-agent-ctl, this is a low level tool that is
intended for advanced users. Future commit(s) will introduce a more
user-friendly tool that maintains state, designed to function as a
kubectl plugin for managing owners’ workloads.

Examples of creating and starting a container using
`kata-agent-tls-ctl`:

Setup environment

```
$ export guest_addr=10.89.0.28 # IP address associated with the confidential VM
$ export guest_port=50090         # API proxy server’s port (listens on)
$ export ctl=./target/x86_64-unknown-linux-musl/release/kata-agent-tls-ctl
$ export key_dir=${KATA_DIR}/src/agent/grpc_tls_key
```

Display the status of containers in the sandbox environment

```
$ ${ctl} -l trace connect --key-dir "${key_dir}"  --bundle-dir "${bundle_dir}"  \
--server-address  "ipaddr://${guest_addr}:${guest_port}" \
 -c  "ListContainers"
```

Set a container ID and specify an OCI spec:

```
$ container_id=9e3d1d4750e4e20945d22c358e13c85c6b88922513bce2832c0cf403f065dc6
$ OCI_SPEC_CONFIG=${KATA_DIR}/src/tools/agent-tls-ctl/config.json
```

_Note: the next two commands require pull_image support in the guest!_
**Create container request**
```

$ ${ctl} -l trace connect --key-dir "${key_dir}" --bundle-dir "${bundle_dir}"  \
--server-address  "ipaddr://${guest_addr}:${guest_port}" \
 -c "CreateContainer cid=${container_id} spec=file:///${OCI_SPEC_CONFIG}"
```

**Start container request**

```
$ ${ctl} -l trace connect --no-auto-values --key-dir "${key_dir}" --bundle-dir "${bundle_dir}"  \
--server-address "ipaddr://${guest_addr}:${guest_port}" \
-c "StartContainer json://{\"container_id\": \"${container_id}\"}"
```

Fixes: kata-containers#1834

Signed-off-by: Ray Valdez <rvaldez@us.ibm.com>
Signed-off-by: Salman Ahmed <sahmed@ibm.com>
Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
Signed-off-by: Zhongshu Gu <zgu@us.ibm.com>
Signed-off-by: Pau-Chen Cheng <pau@us.ibm.com>

src/agent/Cargo.toml

@@ -26,6 +26,11 @@ url = "2.5.0"
 kata-sys-util = { path = "../libs/kata-sys-util" }
 kata-types = { path = "../libs/kata-types" }
 safe-path = { path = "../libs/safe-path" }
+prost = "0.11"
+tonic = {version="0.8", features = ["tls"]}
+zip = "0.5.4"
+
+kbs_protocol = { git = "https://github.com/confidential-containers/guest-components.git", rev = "42b7c96", default-features = false,  features = [ "background_check", "openssl" ]}
 
 # Async helpers
 async-trait = "0.1.42"
@@ -79,6 +84,9 @@ openssl = { version = "0.10.54", features = ["vendored"], optional = true }
 # Image pull/decrypt
 image-rs = { git = "https://github.com/confidential-containers/guest-components", rev = "ca6b438", default-features = true, optional = true }
 
+[build-dependencies]
+tonic-build = "0.8"
+
 [dev-dependencies]
 tempfile = "3.1.0"
 test-utils = { path = "../libs/test-utils" }

src/agent/grpc_tls_keys/.gitignore

@@ -0,0 +1,4 @@
+*.key
+*.pem
+*.csr
+*.srl

src/agent/grpc_tls_keys/cert.ext

@@ -0,0 +1,7 @@
+authorityKeyIdentifier=keyid,issuer
+basicConstraints=CA:FALSE
+keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost

src/agent/grpc_tls_keys/gen_key_cert.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Create rsa private and public keys 
+#
+CA_KEY=ca.key
+CA_PEM=ca.pem
+
+SERVER_KEY=server.key
+SERVER_CSR=server.csr
+SERVER_PEM=server.pem
+
+CLIENT_KEY=client.key
+CLIENT_CSR=client.csr
+CLIENT_PEM=client.pem
+
+KEY_SIZE=2048
+COUNTRY="AA"
+STATE="Default State"
+LOCALITY="Default City"
+ORG="Default Unit"
+ORG_UNIT="Default Unit"
+CERT_EXT=cert.ext
+
+#1. Create CA key
+openssl genrsa -out ${CA_KEY} ${KEY_SIZE} 
+
+#2. Create self-signed cert valid for ten years
+openssl req -x509 -new -nodes -key ${CA_KEY} -sha256 -days 3650 -out ${CA_PEM} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=grpc-tls CA"
+
+#3. Create server key (kata_agent)
+openssl genrsa -out ${SERVER_KEY} ${KEY_SIZE}
+
+#4. Create cert request
+openssl req -new -sha256 -key ${SERVER_KEY} -out ${SERVER_CSR} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=server" \
+
+# 5. Create server cert
+openssl x509 -req -in ${SERVER_CSR} -CA ${CA_PEM} -CAkey ${CA_KEY} -CAcreateserial -out ${SERVER_PEM} -days 3650 -sha256 -extfile ${CERT_EXT} 2> /dev/null
+
+## Repeat steps 3 - 5 for client (tenant)
+
+#3. Create client key
+openssl genrsa -out ${CLIENT_KEY} ${KEY_SIZE} 
+
+#4. Create cert request
+openssl req -new -sha256 -key ${CLIENT_KEY} -out ${CLIENT_CSR} \
+	 -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCALITY}/O=${ORG}/OU=${ORG_UNIT}/CN=client"
+
+#5. Create cert cert
+openssl x509 -req -in ${CLIENT_CSR} -CA ${CA_PEM} -CAkey ${CA_KEY} -CAcreateserial -out ${CLIENT_PEM} -days 3650 -sha256 -extfile ${CERT_EXT} 2> /dev/null

src/agent/src/config.rs

@@ -26,6 +26,8 @@ const LOG_VPORT_OPTION: &str = "agent.log_vport";
 const CONTAINER_PIPE_SIZE_OPTION: &str = "agent.container_pipe_size";
 const UNIFIED_CGROUP_HIERARCHY_OPTION: &str = "agent.unified_cgroup_hierarchy";
 const CONFIG_FILE: &str = "agent.config_file";
+const AA_KBC_PARAMS: &str = "agent.aa_kbc_params";
+const SPLIT_API_FLAG: &str = "agent.split_api";
 
 // Configure the proxy settings for HTTPS requests in the guest,
 // to solve the problem of not being able to access the specified image in some cases.
@@ -74,6 +76,8 @@ pub struct AgentConfig {
     pub supports_seccomp: bool,
     pub https_proxy: String,
     pub no_proxy: String,
+    pub aa_kbc_params: String,
+    pub split_api: bool,
 }
 
 #[derive(Debug, Deserialize)]
@@ -91,6 +95,8 @@ pub struct AgentConfigBuilder {
     pub tracing: Option<bool>,
     pub https_proxy: Option<String>,
     pub no_proxy: Option<String>,
+    pub aa_kbc_params: Option<String>,
+    pub split_api: Option<bool>,
 }
 
 macro_rules! config_override {
@@ -154,6 +160,8 @@ impl Default for AgentConfig {
             supports_seccomp: rpc::have_seccomp(),
             https_proxy: String::from(""),
             no_proxy: String::from(""),
+            aa_kbc_params: String::from(""),
+            split_api: false,
         }
     }
 }
@@ -185,6 +193,8 @@ impl FromStr for AgentConfig {
         config_override!(agent_config_builder, agent_config, tracing);
         config_override!(agent_config_builder, agent_config, https_proxy);
         config_override!(agent_config_builder, agent_config, no_proxy);
+        config_override!(agent_config_builder, agent_config, aa_kbc_params);
+        config_override!(agent_config_builder, agent_config, split_api);
 
         Ok(agent_config)
     }
@@ -286,6 +296,8 @@ impl AgentConfig {
             );
             parse_cmdline_param!(param, HTTPS_PROXY, config.https_proxy, get_url_value);
             parse_cmdline_param!(param, NO_PROXY, config.no_proxy, get_string_value);
+            parse_cmdline_param!(param, AA_KBC_PARAMS, config.aa_kbc_params, get_string_value);
+            parse_cmdline_param!(param, SPLIT_API_FLAG, config.split_api, get_bool_value);
         }
 
         if let Ok(addr) = env::var(SERVER_ADDR_ENV_VAR) {

src/agent/src/main.rs

@@ -77,6 +77,7 @@ use tokio::{
 mod image;
 
 mod rpc;
+mod secrets;
 mod tracer;
 
 #[cfg(feature = "agent-policy")]
@@ -388,6 +389,30 @@ async fn start_sandbox(
     let mut server = rpc::start(sandbox.clone(), config.server_addr.as_str(), init_mode).await?;
     server.start().await?;
 
+    if config.split_api {
+        // downloading and extracting the tls keys for the grpctls server
+        match secrets::retrieve_secrets().await {
+            Ok(_) => println!("main: Success in getting tenant-keys"),
+            Err(e) => {
+                eprintln!("main: Failed to get keys: {:?}", e)
+            }
+        }
+
+        // if the tls keys are downloaded and extracted, then start the grpctls server
+        if secrets::tls_keys_exist() {
+            // TBD Remove owner execlusive APIs from the host side
+            // let _ = match config.remove_owner_api() {
+            //     Ok(_) => println!("main: Disable Owner API"),
+            //     Err(e) => { println!("main: Unable to disable Owner API: {:?}", e) }
+            // };
+
+            let gserver =
+                rpc::rpctls::grpcstart(sandbox.clone(), config.server_addr.as_str(), init_mode)
+                    .await?;
+            gserver.await?;
+        }
+    }
+
     rx.await?;
     server.shutdown().await?;
 

src/agent/src/rpc.rs

@@ -96,6 +96,8 @@ use std::path::PathBuf;
 
 use kata_types::k8s;
 
+pub mod rpctls;
+
 pub const CONTAINER_BASE: &str = "/run/kata-containers";
 const MODPROBE_PATH: &str = "/sbin/modprobe";