Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions deploy/rustfs-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,57 @@ To install in a specific namespace:
helm install rustfs-operator deploy/rustfs-operator/ --namespace rustfs-system --create-namespace
```

### OpenShift Installation

Enable the OpenShift profile so the chart omits the fixed Pod and container
security contexts from the Operator, Console, and optional Console frontend
Deployments. OpenShift SecurityContextConstraints (SCC) then assigns values
valid for the installation namespace, matching the MinIO Operator installation
contract:

```bash
helm upgrade --install rustfs-operator deploy/rustfs-operator/ \
--namespace rustfs-system \
--create-namespace \
--set openshift.enabled=true
```

For Tenant workloads, use explicit empty Pool security contexts as shown in
`examples/openshift-tenant.yaml`:

```yaml
spec:
pools:
- name: pool-0
securityContext: {}
containerSecurityContext: {}
```

The empty objects delegate UID, GID, FSGroup, and container security settings
to the namespace SCC. They are an OpenShift-specific contract; generic
Kubernetes Pod Security admission validates fields but does not assign an
allowed runtime identity. Keep `openshift.enabled=false` and omit the Tenant
fields on generic Kubernetes so the RustFS defaults remain in effect.

This profile provides SCC-compatible manifests but does not by itself imply
OpenShift certification or OperatorHub distribution. Support is currently
limited to `restricted-v2`; the `restricted-v3` requirement to set
`spec.hostUsers: false` is not implemented.

The RustFS server image is an independent prerequisite. It must support an
arbitrary SCC-assigned UID: writable image-layer directories, including
`/data` and `/logs`, must be owned by group `0` and grant the group the same
permissions as the owner. Images that keep those directories as
`10001:10001` with mode `0750` are not compatible even after fixed IDs are
removed from the Pod spec. Use a rebuilt or fixed image before applying the
OpenShift Tenant example; the chart cannot repair image filesystem ownership.

The optional split frontend is disabled by default. Its image must also be
verified for arbitrary-UID execution, writable nginx runtime paths, and
unprivileged port binding before setting `console.frontend.enabled=true` on
OpenShift. Omitting its `securityContext` does not make an incompatible nginx
image OpenShift-ready.

## Uninstalling the Chart

To uninstall/delete the `rustfs-operator` deployment:
Expand Down Expand Up @@ -176,6 +227,7 @@ The generated ClusterRole grants only `get`, `list`, and `watch` for Secrets and

| Parameter | Description | Default |
|-----------|-------------|---------|
| `openshift.enabled` | Omit chart-managed Deployment security contexts and delegate runtime identity to OpenShift SCC | `false` |
| `namespace` | Namespace to deploy to | `""` (uses release namespace) |
| `commonLabels` | Labels to add to all resources | `{}` |
| `commonAnnotations` | Annotations to add to all resources | `{}` |
Expand Down Expand Up @@ -352,6 +404,37 @@ cluster-scoped CRDs first so the API server accepts fields introduced by the
new Operator version. The dedicated field manager deliberately takes ownership
of the chart-managed CRD fields, including CRDs originally created by Helm.

When adopting OpenShift mode on an existing installation, apply the CRDs first,
then upgrade the chart with `openshift.enabled=true`, and wait for the Operator
and Console rollouts before changing Tenant manifests. The chart upgrade rolls
only those Deployments; it does not rewrite Tenant or PVC API objects. The two
empty objects form one explicit delegation signal; a lone empty object retains
the Operator defaults for compatibility with legacy field-based clients.

Inventory existing paired empty objects with the `jq` preflight in the Operator
user guide before upgrading. This release changes such a pair from inheriting
Operator defaults to SCC delegation, so every match is a breaking migration
decision. Changing an existing Pool to `securityContext: {}` and
`containerSecurityContext: {}` changes its StatefulSet Pod template and causes
a Tenant Pod rollout. A changed SCC-assigned FSGroup can also trigger volume
ownership work during first mount; large volumes can start slowly, and CSI or
root-squash permission incompatibilities can prevent mount or write. Verify the
namespace SCC, arbitrary-UID image, and StorageClass with existing data, keep a
recoverable backup, and schedule a maintenance window. A single-replica Tenant
can be unavailable during restart, while a multi-replica Tenant temporarily
runs with reduced capacity.

Do not roll back to an Operator version that interprets explicit empty objects
as a request for the fixed RustFS UID/GID defaults. Such a controller can put
the fixed identity back into the StatefulSet template and OpenShift may reject
the resulting Pods. Recover by rolling forward or restore a complete security
context that is valid for the namespace SCC before downgrading.

Likewise, disabling `openshift.enabled` or rolling the chart back to a version
without this profile reintroduces the chart's fixed Operator/Console identities
and rolls those Deployments. Confirm that the namespace SCC permits those
identities before doing so; otherwise keep the profile enabled and roll forward.

This release adds secure defaults to generated RustFS Pods and containers.
Existing compatible Tenants whose StatefulSet templates do not already contain
those values will roll on their next reconciliation. Schedule the upgrade in a
Expand Down
6 changes: 6 additions & 0 deletions deploy/rustfs-operator/templates/console-deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
{{- $consoleLoginAdmission := default dict .Values.console.loginAdmission -}}
{{- $openShift := default dict .Values.openshift -}}
{{- $openShiftEnabled := default false $openShift.enabled -}}
{{- $reservedConsoleEnv := dict
"CONSOLE_LOGIN_ADMISSION_REQUESTS_PER_SECOND" "console.loginAdmission.requestsPerSecond"
"CONSOLE_LOGIN_ADMISSION_BURST" "console.loginAdmission.burst"
Expand Down Expand Up @@ -44,10 +46,12 @@ spec:
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if not $openShiftEnabled }}
{{- with .Values.console.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
containers:
- name: console
image: "{{ .Values.console.image.repository }}:{{ .Values.console.image.tag | default .Values.operator.image.tag }}"
Expand Down Expand Up @@ -100,10 +104,12 @@ spec:
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if not $openShiftEnabled }}
{{- with .Values.console.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
{{- with .Values.console.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{{- $openShift := default dict .Values.openshift -}}
{{- $openShiftEnabled := default false $openShift.enabled -}}
{{- if and .Values.console.enabled .Values.console.frontend.enabled -}}
apiVersion: apps/v1
kind: Deployment
Expand Down Expand Up @@ -47,8 +49,10 @@ spec:
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if not $openShiftEnabled }}
{{- with .Values.console.frontend.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
{{- end }}
6 changes: 6 additions & 0 deletions deploy/rustfs-operator/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
{{- $livenessProbe := default dict .Values.operator.livenessProbe -}}
{{- $readinessProbe := default dict .Values.operator.readinessProbe -}}
{{- $stsAdmission := default dict .Values.sts.admission -}}
{{- $openShift := default dict .Values.openshift -}}
{{- $openShiftEnabled := default false $openShift.enabled -}}
{{- if and (not .Values.operator.metrics.enabled) (or (hasKey $livenessProbe "httpGet") (hasKey $readinessProbe "httpGet")) -}}
{{- fail "operator.metrics.enabled=false requires overriding operator.livenessProbe and operator.readinessProbe because the chart defaults use the metrics port" -}}
{{- end -}}
Expand Down Expand Up @@ -58,10 +60,12 @@ spec:
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if not $openShiftEnabled }}
{{- with .Values.operator.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
containers:
- name: operator
image: "{{ .Values.operator.image.repository }}:{{ .Values.operator.image.tag }}"
Expand Down Expand Up @@ -150,10 +154,12 @@ spec:
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if not $openShiftEnabled }}
{{- with .Values.operator.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
{{- with .Values.operator.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
12 changes: 12 additions & 0 deletions deploy/rustfs-operator/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,18 @@
],
"description": "Kubernetes cluster DNS domain used for Tenant peer URLs, generated TLS SANs, and operator STS auto TLS."
},
"openshift": {
"type": "object",
"required": [
"enabled"
],
"properties": {
"enabled": {
"type": "boolean",
"description": "Omit chart-managed Deployment security contexts so OpenShift SCC can assign the runtime identity."
}
}
},
"sts": {
"type": "object",
"properties": {
Expand Down
6 changes: 6 additions & 0 deletions deploy/rustfs-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
# Kubernetes cluster DNS domain used for Tenant peer URLs and generated TLS SANs.
clusterDomain: cluster.local

# OpenShift installation compatibility. When enabled, the chart omits Pod and
# container securityContext fields from its own Deployments so the namespace
# SecurityContextConstraints (SCC) can assign an allowed UID and FSGroup.
openshift:
enabled: false

# Operator deployment configuration
operator:
# Number of operator replicas
Expand Down
101 changes: 101 additions & 0 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,35 @@ helm install rustfs-operator deploy/rustfs-operator/ \
--create-namespace
```

On OpenShift, enable SCC-managed runtime identities for the Operator, Console,
and optional split frontend:

```bash
helm upgrade --install rustfs-operator deploy/rustfs-operator/ \
--namespace rustfs-system \
--create-namespace \
--set openshift.enabled=true
```

This follows the MinIO Operator installation behavior: chart-managed
Deployments omit their Pod and container `securityContext`, allowing the
namespace SecurityContextConstraints (SCC) to assign an allowed UID and
FSGroup. This is manifest compatibility, not an OpenShift certification claim.
Generic Kubernetes installations must keep the default
`openshift.enabled=false` behavior. The current target is `restricted-v2`;
`restricted-v3` also requires `spec.hostUsers: false`, which is not yet covered.

SCC-compatible manifests are insufficient when the server image assumes UID
`10001`. Before deploying a Tenant, use an arbitrary-UID-compatible image whose
writable image-layer directories, including `/data` and `/logs`, are owned by
group `0` and give the group the same permissions as the owner. An image with
those directories owned by `10001:10001` and mode `0750` remains incompatible;
the Operator cannot repair image filesystem ownership.

Keep the optional split frontend disabled unless its image is independently
verified for arbitrary-UID nginx runtime paths and unprivileged port binding.
SCC-assigned identity alone cannot repair an incompatible frontend image.

Verify the operator and Console pods:

```bash
Expand Down Expand Up @@ -108,6 +137,56 @@ are cluster-scoped and shared by all Tenant namespaces. The dedicated field
manager deliberately takes ownership of the chart-managed CRD fields so this
upgrade also works for CRDs originally created by Helm.

When enabling OpenShift support on an existing installation, keep this order:
apply both CRDs, upgrade the chart with `openshift.enabled=true`, wait for the
Operator and Console rollouts, and only then update Tenant security contexts.
Before upgrading the controller, inventory every explicit empty pair at Tenant
and Pool scope:

```bash
kubectl get tenants.rustfs.com -A -o json | jq -r '
def empty_object: type == "object" and length == 0;
.items[] as $tenant |
([
(select(($tenant.spec | has("securityContext")) and
($tenant.spec | has("containerSecurityContext")) and
($tenant.spec.securityContext | empty_object) and
($tenant.spec.containerSecurityContext | empty_object)) | "spec"),
($tenant.spec.pools[]? |
select((has("securityContext")) and
(has("containerSecurityContext")) and
(.securityContext | empty_object) and
(.containerSecurityContext | empty_object)) |
"pool:" + .name)
]) as $locations |
select($locations | length > 0) |
[$tenant.metadata.namespace, $tenant.metadata.name, ($locations | join(","))] |
@tsv'
```

This release deliberately changes a paired `{}`/`{}` from "inherit Operator
defaults" to "delegate to platform admission". A lone empty object keeps the
legacy behavior. Treat every reported pair as a breaking migration decision:
remove both fields before upgrading when Operator defaults should remain, or
keep both only after validating the target SCC and image.

The chart upgrade does not rewrite Tenant or PVC API objects. Converting an
existing Pool to the paired empty security contexts changes its StatefulSet Pod
template and rolls that Pool. A changed SCC-assigned FSGroup can also make
kubelet or the CSI driver update volume ownership on first mount; large volumes
can start slowly, and storage with incompatible `fsGroupPolicy`, root-squash, or
permission behavior can fail to mount or write. Test the StorageClass with
existing data, keep a recoverable backup, and schedule a maintenance window. A
single-replica Tenant can be unavailable during restart and a multi-replica
Tenant temporarily runs with reduced capacity.

Do not downgrade to a controller that restores fixed UID/GID defaults for the
empty pair, because OpenShift SCC may reject that rollback. Roll forward, or
restore an SCC-valid complete security context before downgrading. Disabling
`openshift.enabled` or rolling back to a chart without the profile also
reintroduces fixed Operator/Console identities and rolls those Deployments; do
that only if the namespace SCC allows the fixed IDs.

Existing manifests that omit `users[].credsSecret` remain compatible. Wait for
the new Operator rollout to complete before relying on an explicit user Secret
reference; older binaries continue using the same-name Secret convention.
Expand Down Expand Up @@ -170,6 +249,7 @@ Common chart sections:

| Section | Purpose |
|---------|---------|
| `openshift` | SCC-compatible rendering for the Operator, Console, and optional frontend Deployments. Disabled by default. |
| `operator` | Operator Deployment replicas, image, resources, probes, metrics, scheduling, leader election, and tenant monitoring. |
| `sts` | Operator STS endpoint, service port, TokenReview audience, and TLS handling. |
| `serviceAccount` / `rbac` | Operator ServiceAccount and RBAC creation. |
Expand Down Expand Up @@ -467,6 +547,27 @@ be rejected by cluster admission policy. For legacy compatibility, an explicit
`runAsUser: 0` without an explicit `runAsNonRoot` derives `runAsNonRoot: false`;
that configuration cannot run in a `restricted` namespace.

On OpenShift, use explicit empty objects at Pool level to delegate the runtime
identity and container security settings to the namespace SCC, following the
MinIO Operator contract:

```yaml
spec:
pools:
- name: pool-0
securityContext: {}
containerSecurityContext: {}
```

See `examples/openshift-tenant.yaml`. The distinction between omission and an
explicit empty pair is intentional: omission requests the RustFS defaults;
the paired `{}`/`{}` requests SCC ownership. A lone empty object retains the
defaults for compatibility. Do not use the pair on generic Kubernetes unless
another admission controller supplies equivalent settings. Updating an
existing Pool to this form rolls its StatefulSet Pods. The example uses a
placeholder image deliberately; replace it only with a verified
arbitrary-UID-compatible RustFS image.

`RuntimeDefault` also requires a RustFS image that can run under the runtime's
default seccomp profile. `rustfs/rustfs:1.0.0-beta.8` is not compatible because
its Tokio runtime enables io_uring; use a build containing
Expand Down
Loading
Loading