diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 07de687103..803fff70ed 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -393,6 +393,7 @@ openshell logs | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | +| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login `, gateway auth logs | diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 96e54ad448..1356e2d932 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -36,6 +36,16 @@ This is a stopgap persistence model. It preserves user files across pod rescheduling but duplicates the base workspace and does not automatically apply image updates to existing PVCs. Future snapshotting should replace it. +The workspace PVC size defaults to `workspace_default_storage_size`. Set +`workspace_storage_class` to pin the PVC to a specific `StorageClass`; an empty +value omits `storageClassName` so the cluster's default `StorageClass` applies. +Clusters with no default `StorageClass` must set this, otherwise the PVC stays +`Pending` and the sandbox never starts. Both fields can also be supplied at +runtime via `OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE` and +`OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS`. Both apply only to the workspace PVC +that OpenShell provisions automatically; they have no effect when a `driver_config` +mount attaches an existing PVC under `/sandbox`, which skips the default PVC. + ## Credentials, TLS, and Relay The driver injects gateway callback configuration, sandbox identity, TLS client diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 1eeaac8396..fb471180a9 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -267,6 +267,12 @@ pub struct KubernetesComputeConfig { )] pub app_armor_profile: Option, pub workspace_default_storage_size: String, + /// Kubernetes `StorageClass` name for the default workspace PVC. + /// Empty string (default) = omit `storageClassName`, using the cluster's + /// default `StorageClass`. Set this on clusters with no default + /// `StorageClass`, otherwise the workspace PVC stays `Pending` and the + /// sandbox never starts. + pub workspace_storage_class: String, /// Default Kubernetes `runtimeClassName` for sandbox pods. /// Applied when a `CreateSandbox` request does not specify one. /// Empty string (default) = omit the field, using the cluster default. @@ -347,6 +353,7 @@ impl Default for KubernetesComputeConfig { enable_user_namespaces: false, app_armor_profile: None, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), + workspace_storage_class: String::new(), default_runtime_class_name: String::new(), sa_token_ttl_secs: 3600, provider_spiffe_workload_api_socket_path: String::new(), @@ -514,6 +521,12 @@ mod tests { ); } + #[test] + fn default_workspace_storage_class_is_empty() { + let cfg = KubernetesComputeConfig::default(); + assert!(cfg.workspace_storage_class.is_empty()); + } + #[test] fn default_topology_is_combined() { let cfg = KubernetesComputeConfig::default(); @@ -658,6 +671,15 @@ mod tests { assert_eq!(cfg.workspace_default_storage_size, "10Gi"); } + #[test] + fn serde_override_workspace_storage_class() { + let json = serde_json::json!({ + "workspace_storage_class": "fast-ssd" + }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.workspace_storage_class, "fast-ssd"); + } + #[test] fn serde_override_service_account_name() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index c784f10db9..2203ad9bd5 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -848,6 +848,7 @@ impl KubernetesComputeDriver { enable_user_namespaces: self.config.enable_user_namespaces, app_armor_profile: self.config.app_armor_profile.as_ref(), workspace_default_storage_size: &self.config.workspace_default_storage_size, + workspace_storage_class: &self.config.workspace_storage_class, default_runtime_class_name: &self.config.default_runtime_class_name, sa_token_ttl_secs: self.config.effective_sa_token_ttl_secs(), provider_spiffe_enabled: self.config.provider_spiffe_enabled(), @@ -2155,24 +2156,36 @@ fn apply_workspace_persistence( /// /// Provides a single PVC named "workspace" that backs the `/sandbox` /// directory. The init container seeds it from the image on first use. -fn default_workspace_volume_claim_templates(storage_size: &str) -> serde_json::Value { +/// +/// When `storage_class` is non-empty, it is written to the PVC's +/// `storageClassName`. An empty value omits the field so the cluster's +/// default `StorageClass` applies. Clusters with no default `StorageClass` +/// must set this to prevent the PVC from staying `Pending`. +fn default_workspace_volume_claim_templates( + storage_size: &str, + storage_class: &str, +) -> serde_json::Value { let size = if storage_size.is_empty() { DEFAULT_WORKSPACE_STORAGE_SIZE } else { storage_size }; + let mut spec = serde_json::json!({ + "accessModes": ["ReadWriteOnce"], + "resources": { + "requests": { + "storage": size + } + } + }); + if !storage_class.is_empty() { + spec["storageClassName"] = serde_json::json!(storage_class); + } serde_json::json!([{ "metadata": { "name": WORKSPACE_VOLUME_NAME }, - "spec": { - "accessModes": ["ReadWriteOnce"], - "resources": { - "requests": { - "storage": size - } - } - } + "spec": spec }]) } @@ -2197,6 +2210,7 @@ struct SandboxPodParams<'a> { enable_user_namespaces: bool, app_armor_profile: Option<&'a AppArmorProfile>, workspace_default_storage_size: &'a str, + workspace_storage_class: &'a str, default_runtime_class_name: &'a str, /// Lifetime (seconds) of the projected `ServiceAccount` token used /// for the bootstrap `IssueSandboxToken` exchange. @@ -2231,6 +2245,7 @@ impl Default for SandboxPodParams<'_> { enable_user_namespaces: false, app_armor_profile: None, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE, + workspace_storage_class: "", default_runtime_class_name: "", sa_token_ttl_secs: 3600, provider_spiffe_enabled: false, @@ -2328,7 +2343,10 @@ fn sandbox_to_k8s_spec( if inject_workspace { root.insert( "volumeClaimTemplates".to_string(), - default_workspace_volume_claim_templates(params.workspace_default_storage_size), + default_workspace_volume_claim_templates( + params.workspace_default_storage_size, + params.workspace_storage_class, + ), ); } @@ -5714,14 +5732,14 @@ mod tests { #[test] fn default_workspace_vct_uses_provided_storage_size() { - let vct = default_workspace_volume_claim_templates("5Gi"); + let vct = default_workspace_volume_claim_templates("5Gi", ""); let storage = &vct[0]["spec"]["resources"]["requests"]["storage"]; assert_eq!(storage, "5Gi"); } #[test] fn default_workspace_vct_falls_back_to_const_when_empty() { - let vct = default_workspace_volume_claim_templates(""); + let vct = default_workspace_volume_claim_templates("", ""); let storage = &vct[0]["spec"]["resources"]["requests"]["storage"]; assert_eq!(storage, DEFAULT_WORKSPACE_STORAGE_SIZE); } @@ -5921,4 +5939,42 @@ mod tests { }; assert!(sandbox_id_from_object(&obj).is_err()); } + + #[test] + fn default_workspace_vct_sets_storage_class_when_provided() { + let vct = default_workspace_volume_claim_templates("5Gi", "fast-ssd"); + assert_eq!(vct[0]["spec"]["storageClassName"], "fast-ssd"); + } + + #[test] + fn default_workspace_vct_omits_storage_class_when_empty() { + let vct = default_workspace_volume_claim_templates("5Gi", ""); + assert!(vct[0]["spec"].get("storageClassName").is_none()); + } + + #[test] + fn workspace_storage_class_propagates_to_generated_cr_spec() { + let params = SandboxPodParams { + workspace_storage_class: "fast-ssd", + ..SandboxPodParams::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&SandboxSpec::default()), ¶ms); + assert_eq!( + cr["spec"]["volumeClaimTemplates"][0]["spec"]["storageClassName"], + "fast-ssd" + ); + } + + #[test] + fn workspace_storage_class_omitted_from_cr_spec_when_empty() { + let cr = sandbox_to_k8s_spec_for_test( + Some(&SandboxSpec::default()), + &SandboxPodParams::default(), + ); + assert!( + cr["spec"]["volumeClaimTemplates"][0]["spec"] + .get("storageClassName") + .is_none() + ); + } } diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index c733b8a45b..c7b0939888 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -160,6 +160,8 @@ async fn main() -> Result<()> { .unwrap_or_else(|_| { openshell_driver_kubernetes::DEFAULT_WORKSPACE_STORAGE_SIZE.to_string() }), + workspace_storage_class: std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") + .unwrap_or_default(), default_runtime_class_name: std::env::var("OPENSHELL_K8S_DEFAULT_RUNTIME_CLASS_NAME") .unwrap_or_default(), sa_token_ttl_secs: args.sa_token_ttl_secs, diff --git a/crates/openshell-server/src/compute/driver_config.rs b/crates/openshell-server/src/compute/driver_config.rs index 59fed439bc..f56d233f2f 100644 --- a/crates/openshell-server/src/compute/driver_config.rs +++ b/crates/openshell-server/src/compute/driver_config.rs @@ -145,6 +145,9 @@ fn apply_kubernetes_runtime_defaults(k8s: &mut KubernetesComputeConfig) { if let Ok(size) = std::env::var("OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE") { k8s.workspace_default_storage_size = size; } + if let Ok(storage_class) = std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") { + k8s.workspace_storage_class = storage_class; + } } fn apply_podman_runtime_defaults( diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 8723535d1a..d29873c6fc 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -229,6 +229,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | +| server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. | | service.healthPort | int | `8081` | Gateway health service port. | | service.metricsPort | int | `9090` | Gateway metrics service port. | | service.port | int | `8080` | Gateway gRPC/HTTP service port. | diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 36a579250b..41cadfbad4 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -135,6 +135,9 @@ data: {{- if .Values.server.workspaceDefaultStorageSize }} workspace_default_storage_size = {{ .Values.server.workspaceDefaultStorageSize | quote }} {{- end }} + {{- if .Values.server.workspaceStorageClass }} + workspace_storage_class = {{ .Values.server.workspaceStorageClass | quote }} + {{- end }} {{- if .Values.server.defaultRuntimeClassName }} default_runtime_class_name = {{ .Values.server.defaultRuntimeClassName | quote }} {{- end }} diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index e89a234912..8abd3dea71 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -187,6 +187,11 @@ server: # Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). # Empty = built-in default (2Gi). workspaceDefaultStorageSize: "" + # -- Kubernetes StorageClass for the workspace PVC in sandbox pods. + # Empty (default) = omit storageClassName, using the cluster's default + # StorageClass. Set this on clusters with no default StorageClass, otherwise + # the workspace PVC stays Pending and the sandbox never starts. + workspaceStorageClass: "" # -- Default Kubernetes runtimeClassName for sandbox pods. # Applied when a CreateSandbox request does not specify one. # Empty (default) = omit the field, using the cluster's default RuntimeClass. diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 56f5d15348..13d06813b3 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -277,6 +277,10 @@ host_gateway_ip = "10.0.0.1" enable_user_namespaces = false app_armor_profile = "Unconfined" workspace_default_storage_size = "10Gi" +# Kubernetes StorageClass for the workspace PVC. Empty (default) omits the +# field, using the cluster's default StorageClass. Set this on clusters with no +# default StorageClass, otherwise the workspace PVC stays Pending. +# workspace_storage_class = "fast-ssd" # Kubernetes RuntimeClass applied to sandbox pods when the API request does # not specify one. Empty (default) = omit the field, using the cluster default. # default_runtime_class_name = "kata-containers" diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 280849b5f6..ecc2b92384 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -311,6 +311,7 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `sidecar.process_binary_aware_network_policy` | `supervisor.sidecar.processBinaryAwareNetworkPolicy` | Keep process/binary-aware network policy enabled in `sidecar` topology. The default runs the sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`. Set false to run as `proxy_uid`, drop both capabilities, and enforce endpoint/L7 policy without matching `policy.binaries`. | | `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | +| `workspace_storage_class` | `server.workspaceStorageClass` | Set the `StorageClass` for the workspace PVC. Empty (default) omits `storageClassName` and uses the cluster's default `StorageClass`. Set this on clusters with no default `StorageClass`, otherwise the workspace PVC stays `Pending` and the sandbox never starts. | | `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. | In `combined` topology, the agent container carries the Linux capabilities