Skip to content

OLS-3697 Add RHOKP standalone operand spec and update related specs #1876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
blublinsky wants to merge 1 commit into
base: main
Choose a base branch
from
Open

OLS-3697 Add RHOKP standalone operand spec and update related specs #1876

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions .ai/spec/what/app-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The App Server is the backend deployment for OpenShift Lightspeed. It runs the l
2. The primary container (lightspeed-service-api) runs the OLS service, listening on HTTPS.
3. The data collector sidecar (lightspeed-to-dataverse-exporter) is added when data collection is enabled AND the telemetry pull secret exists in the openshift-config namespace with a cloud.openshift.com auth entry.
4. The OpenShift MCP server runs as a standalone HTTPS Deployment/Service (`ocpmcp` package) when `spec.ols.introspectionEnabled` is true. The app-server connects via `https://openshift-mcp-server.<ns>.svc:8443/mcp` and trusts `openshift-mcp-server-ca` (`service-ca.crt`). See `ocpmcp.md`.
5. OKP (Offline Knowledge Portal) / Solr hybrid RAG is operator-managed (no CR toggle besides `byokRAGOnly`). When OKP is enabled, the RHOKP sidecar serves Solr HTTP on localhost:9080 for the `search_openshift_documentation` tool path. It requires ~75 GiB ephemeral storage. OKP is on by default; set `spec.ols.byokRAGOnly` to true to skip the RHOKP sidecar, `solr_hybrid` config, and OCP documentation retrieval via Solr.
5. [PLANNED: OLS-3697] OKP (Offline Knowledge Portal) / Solr hybrid RAG is operator-managed (no CR toggle besides `byokRAGOnly`). When OKP is enabled, the RHOKP standalone Deployment serves Solr via HTTPS at `https://lightspeed-rhokp.<ns>.svc:8443`. The app-server connects as a client, trusting the RHOKP service-ca cert via `extra_ca`. OKP is on by default; set `spec.ols.byokRAGOnly` to true to skip the RHOKP standalone operand, `solr_hybrid` config, and OCP documentation retrieval via Solr. See `rhokp.md`.

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Remove the stale RHOKP sidecar description.

The deployment composition still says the app-server has a RHOKP sidecar, contradicting the standalone Deployment described here and in rhokp.md. Change the sidecar count/list so the app-server contract cannot be interpreted as deploying RHOKP locally.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ai/spec/what/app-server.md at line 12, Update the app-server deployment
composition described in this document to remove RHOKP from the sidecar count
and sidecar list. Keep RHOKP represented only as the standalone operator-managed
Deployment referenced by the OKP configuration, so the app-server contract no
longer implies local RHOKP deployment.

6. A PostgreSQL wait init container always runs before the main containers to ensure database readiness.
7. When `spec.ols.rag` is configured, additional init containers copy BYOK RAG data from container images into a shared volume.

Expand All @@ -20,7 +20,7 @@ The App Server is the backend deployment for OpenShift Lightspeed. It runs the l
11. PostgreSQL connection settings are hardcoded to point to the operator-managed PostgreSQL service within the same namespace.
12. If `spec.ols.querySystemPrompt` is set, the custom prompt is written as a second key in the config ConfigMap and referenced by file path in the config.
13. BYOK reference content indexes from `spec.ols.rag` are written to `reference_content.indexes` when present. OCP product documentation is served exclusively via `solr_hybrid` (OKP); the operator does not emit a built-in OCP FAISS index.
14. Unless `byokRAGOnly` is true, the operator generates a `solr_hybrid` config section in `olsconfig.yaml` pointing to `http://localhost:9080` with default hybrid retrieval tuning parameters.
14. [PLANNED: OLS-3697] Unless `byokRAGOnly` is true, the operator generates a `solr_hybrid` config section in `olsconfig.yaml` pointing to `https://lightspeed-rhokp.<ns>.svc:8443` with default hybrid retrieval tuning parameters.

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Document RHOKP CA changes in app-server restart detection.

The new RHOKP HTTPS endpoint requires the RHOKP CA, but the app-server change-detection contract only names the MCP CA. Align it with rhokp.md and tls.md by documenting RHOKP CA-content changes and certificate rotation as restart triggers.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ai/spec/what/app-server.md at line 23, Update the app-server
restart-detection contract near the solr_hybrid RHOKP endpoint description to
include RHOKP CA-content changes and RHOKP certificate rotation as restart
triggers, matching the terminology and behavior documented in rhokp.md and
tls.md while retaining the existing MCP CA trigger.

15a. Unless `byokRAGOnly` is true, the app-server container receives `OCP_CLUSTER_VERSION` (`<major>.<minor>` from the operator's cluster-version lookup) for Solr `chunk_filter_query` resolution in lightspeed-service.

### ROSA-Aware OKP Retrieval
Expand Down Expand Up @@ -68,13 +68,13 @@ The App Server is the backend deployment for OpenShift Lightspeed. It runs the l
| `spec.ols.deployment.api.nodeSelector` | Node selector constraints |
| `spec.ols.deployment.dataCollector.resources` | Data collector container resources |
| `spec.ols.deployment.mcpServer` | Standalone MCP Deployment settings (`Config`: replicas, resources, tolerations, nodeSelector) |
| `spec.ols.deployment.rhokp.resources` | RHOKP sidecar container resources |
| `spec.ols.deployment.rhokp` | Standalone RHOKP Deployment settings (`Config`: replicas, resources, tolerations, nodeSelector) [PLANNED: OLS-3697] |
| `spec.ols.defaultModel` | Default LLM model name |
| `spec.ols.defaultProvider` | Default LLM provider name |
| `spec.ols.logLevel` | Logging level for all service components |
| `spec.ols.maxIterations` | Maximum agent execution iterations |
| `spec.ols.querySystemPrompt` | Custom system prompt for LLM queries |
| `spec.ols.byokRAGOnly` | Disable OKP: no RHOKP sidecar, no `solr_hybrid` section, no `OCP_CLUSTER_VERSION` env. Only BYOK FAISS indexes from `spec.ols.rag` are used. |
| `spec.ols.byokRAGOnly` | Disable OKP: no RHOKP standalone operand, no `solr_hybrid` section, no `OCP_CLUSTER_VERSION` env. Only BYOK FAISS indexes from `spec.ols.rag` are used. |
| `spec.ols.introspectionEnabled` | Enable standalone OpenShift MCP server operand |
| `spec.ols.userDataCollection.feedbackDisabled` | Disable feedback collection |
| `spec.ols.userDataCollection.transcriptsDisabled` | Disable transcript collection |
Expand All @@ -92,30 +92,32 @@ The App Server is the backend deployment for OpenShift Lightspeed. It runs the l
2. Tool filtering requires MCP servers to be configured (either introspection or user-defined).
3. The service always connects to PostgreSQL via the internal cluster service DNS.
4. RAG init containers run in index order, copying data to subdirectories of the shared RAG volume.
5. The RHOKP sidecar requires approximately 75 GiB of ephemeral storage for Solr data. This must be documented in product infrastructure requirements.
5. [PLANNED: OLS-3697] RHOKP runs as a standalone Deployment (`lightspeed-rhokp`) with its own 75 GiB EmptyDir. The app-server pod no longer requires ephemeral storage for OKP. See `rhokp.md`.

### Resource Conventions [OLS-3397]
30. All operator-managed container defaults follow the [OpenShift resource conventions](https://github.com/openshift/enhancements/blob/master/CONVENTIONS.md#resources-and-limits): defaults declare CPU and memory requests only, and do not set resource limits. This applies to the primary API container, sidecars (data collector, RHOKP), and the standalone MCP Deployment.
30. All operator-managed container defaults follow the [OpenShift resource conventions](https://github.com/openshift/enhancements/blob/master/CONVENTIONS.md#resources-and-limits): defaults declare CPU and memory requests only, and do not set resource limits. This applies to the primary API container, sidecars (data collector), the standalone MCP Deployment, and the standalone RHOKP Deployment.
31. Users may still set limits via the CRD (`spec.ols.deployment.<component>.resources`, including `spec.ols.deployment.rhokp.resources`) if their environment requires it. The CRD uses standard `corev1.ResourceRequirements` which accepts both requests and limits.
32. The RHOKP sidecar's ~75 GiB ephemeral storage requirement is unchanged by this convention — it applies only to CPU and memory.
32. [PLANNED: OLS-3697] The RHOKP standalone Deployment's ~75 GiB EmptyDir sizeLimit is unchanged by this convention — it applies only to CPU and memory.

### RHOKP Image
33. The RHOKP sidecar image is set via the operator `--rhokp-image` startup flag. Default comes from `related_images.json` entry `rhokp` (`utils.RHOOKPImageDefault` / `imageDefaultOr`). The OLM bundle lists it in CSV `spec.relatedImages` and passes the image via `--rhokp-image` on the manager deployment.
33. [PLANNED: OLS-3697] The RHOKP standalone Deployment image is set via the operator `--rhokp-image` startup flag. Default comes from `related_images.json` entry `rhokp` (`utils.RHOOKPImageDefault` / `imageDefaultOr`). The OLM bundle lists it in CSV `spec.relatedImages` and passes the image via `--rhokp-image` on the manager deployment. See `rhokp.md`.

### Agentic Sandbox Configuration Handoff [PLANNED: OLS-3572]

34. The operator creates and maintains a `lightspeed-sandbox-config` ConfigMap in the operator namespace, providing the agentic operator with a ready-to-use base pod spec for sandbox pods. The ConfigMap contains:
- `sandbox-pod-spec`: JSON-serialized `corev1.PodSpec` with the sandbox container image (from `--agentic-sandbox-image` flag / related-images), OTEL endpoint env var (when templog collector is deployed), MCP CA cert volumes + volume mounts (when ocp-mcp is deployed as standalone HTTPS service), and resource defaults. CA certificates are mounted directly in the PodSpec — no separate CA bundle key.
- `sandbox-pod-spec`: JSON-serialized `corev1.PodSpec` with the sandbox container image (from `--agentic-sandbox-image` flag / related-images), OTEL endpoint env var (when templog collector is deployed), MCP CA cert volumes + volume mounts (when ocp-mcp is deployed as standalone HTTPS service), RHOKP CA cert volumes + volume mounts (when OKP is enabled), and resource defaults. CA certificates are mounted directly in the PodSpec — no separate CA bundle key.
- `sandbox-mode`: `bare-pod` or `sandbox-claim` from `OLSConfig.spec.agenticOLS.sandboxMode`.
- `mcp-endpoint`: MCP server endpoint URL (when ocp-mcp is deployed as standalone HTTPS service).
- `otel-endpoint`: OTEL collector gRPC endpoint (when templog collector is deployed).
- `rhokp-endpoint`: RHOKP Solr HTTPS endpoint URL (when OKP is enabled, i.e., `!byokRAGOnly`). [PLANNED: OLS-3697]

35. The ConfigMap is always created during reconciliation. Keys are absent when the corresponding feature is not enabled. The `sandbox-pod-spec` key is always present.

36. The ConfigMap is reconciled whenever relevant config changes: sandbox image, cert rotation, OTEL collector deployment, MCP server deployment, or `spec.agenticOLS` field changes.
36. The ConfigMap is reconciled whenever relevant config changes: sandbox image, cert rotation, OTEL collector deployment, MCP server deployment, RHOKP deployment, or `spec.agenticOLS` field changes.

## Planned Changes

- [PLANNED: OLS-3221] Liveness probe now checks PostgreSQL health via the service's background health-check loop status. Probe configuration (failureThreshold, periodSeconds) added to deployment generation. See Rules 24–25.
- Agentic/sandbox reuse of the standalone MCP Service and CA is tracked under [OLS-3572](https://redhat.atlassian.net/browse/OLS-3572); optional agentic auto-injection is separately deferred ([OLS-3594](https://redhat.atlassian.net/browse/OLS-3594)).
- [PLANNED: OLS-3572] Agentic sandbox configuration handoff — classic operator builds base PodSpec and writes `lightspeed-sandbox-config` ConfigMap for the agentic operator. See Rules 34–36.
- [PLANNED: OLS-3697] RHOKP standalone HTTPS cutover — sidecar removed from app-server, standalone `lightspeed-rhokp` Deployment/Service created. `solr_hybrid.solr_http_base` changes to HTTPS cluster DNS. RHOKP CA added to `extra_ca`. `spec.ols.deployment.rhokp` type changes from `ContainerConfig` to `Config`. Handoff ConfigMap gains `rhokp-endpoint`. See `rhokp.md` and design spec `docs/superpowers/specs/2026-07-27-rhokp-standalone.md`.
18 changes: 11 additions & 7 deletions .ai/spec/what/crd-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Field path (relative to `spec.ols.deployment`) | JSON key | Go type | Notes
`api` | `api` | `Config` | API container. Replicas configurable (default 1, min 0)
`dataCollector` | `dataCollector` | `ContainerConfig` | Data collector container. Resources only
`mcpServer` | `mcpServer` | `Config` | Standalone OpenShift MCP server Deployment (replicas, resources, tolerations, nodeSelector)
`rhokp` | `rhokp` | `ContainerConfig` | RHOKP sidecar container (Solr / OKP). Resources only
`rhokp` | `rhokp` | `Config` | [PLANNED: OLS-3697] Standalone RHOKP Deployment (Solr / OKP). Replicas (forced to 1), resources, tolerations, nodeSelector
`console` | `console` | `Config` | Console container. Has replicas field but operator forces 1
`database` | `database` | `Config` | Database container. Has replicas field but operator forces 1
`alertsAdapter` | `alertsAdapter` | `AlertsAdapterSpec` | Agentic alerts adapter deployment and user-managed runtime config reference. Replicas forced to 1
Expand All @@ -142,7 +142,7 @@ Field path (relative to `spec.ols.deployment`) | JSON key | Go type | Notes
`agenticConsole` | `agenticConsole` | `Config` | Agentic console plugin container. Replicas forced to 1
`otelCollector` | `otelCollector` | `Config` | OTEL Collector container ([OLS-3510](https://redhat.atlassian.net/browse/OLS-3510)). Replicas forced to 1

20. Replicas are user-configurable for the API container (`spec.ols.deployment.api.replicas`). For console, database, alerts adapter, agentic console, and otel collector, the operator always overrides replicas to 1 regardless of spec value.
20. Replicas are user-configurable for the API container (`spec.ols.deployment.api.replicas`). For console, database, alerts adapter, agentic console, otel collector, and RHOKP [PLANNED: OLS-3697], the operator always overrides replicas to 1 regardless of spec value.

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Update the replica constraint to include RHOKP.

This line makes RHOKP replicas operator-forced to 1, but the later replica constraint still lists only console, database, alerts adapter, agentic console, and OTEL collector. Update that constraint so the CRD contract is unambiguous.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ai/spec/what/crd-api.md at line 145, Update the replica constraint in the
CRD contract to explicitly include RHOKP alongside console, database, alerts
adapter, agentic console, and otel collector. Keep the existing operator-forced
replica value of 1 and ensure the earlier RHOKP planned note remains consistent.


##### Config Fields

Expand Down Expand Up @@ -244,16 +244,16 @@ Field | JSON key | Go type | Required | Validation

#### Boolean/String Fields

41. `spec.ols.byokRAGOnly` -- `bool`, optional. When true, only BYOK RAG sources are used: the operator does not deploy the RHOKP sidecar, does not write `solr_hybrid` into `olsconfig.yaml`, and does not set `OCP_CLUSTER_VERSION` on the app-server pod.
41. `spec.ols.byokRAGOnly` -- `bool`, optional. When true, only BYOK RAG sources are used: the operator does not deploy the standalone RHOKP operand, does not write `solr_hybrid` into `olsconfig.yaml`, and does not set `OCP_CLUSTER_VERSION` on the app-server pod.

#### Operator-managed OKP (not on CR)

OKP / Solr hybrid RAG has no `spec.ols.solrHybrid` (or similar) field. It is enabled by default and turned off only via `byokRAGOnly`. When active, the operator:
- deploys the RHOKP sidecar and writes `ols_config.solr_hybrid` with operator defaults (`http://localhost:9080`, hybrid tuning);
- [PLANNED: OLS-3697] deploys the standalone RHOKP Deployment/Service (`lightspeed-rhokp`) and writes `ols_config.solr_hybrid` with operator defaults (`https://lightspeed-rhokp.<ns>.svc:8443`, hybrid tuning);
- sets `OCP_CLUSTER_VERSION` on the app-server container for Solr version filtering;
- serves OCP product documentation via Solr hybrid only; `reference_content.indexes` lists BYOK FAISS indexes from `spec.ols.rag` only.

RHOKP sidecar CPU, memory, and ephemeral storage are overridable via `spec.ols.deployment.rhokp.resources` (defaults: 2 CPU, 2 GiB memory, and 75 GiB ephemeral storage requests).
[PLANNED: OLS-3697] RHOKP standalone Deployment resources are overridable via `spec.ols.deployment.rhokp` (`Config`: replicas forced to 1, resources, tolerations, nodeSelector). Default resource requests: 2 CPU, 2 GiB memory. Storage: 75 GiB EmptyDir with sizeLimit.
42. `spec.ols.querySystemPrompt` -- `string`, optional. Custom system prompt for LLM queries. If unset, the default OpenShift Lightspeed prompt is used.
43. `spec.ols.maxIterations` -- `int`. Default: `5`. Minimum=1. Maximum number of iterations for agent execution.
44. `spec.ols.imagePullSecrets` -- `[]corev1.LocalObjectReference`, optional. Pull secrets for BYOK RAG images.
Expand Down Expand Up @@ -401,8 +401,11 @@ Path | Type | Default | Required | Validation | Description
`spec.ols.deployment.dataCollector.resources` | `*ResourceRequirements` | -- | No | -- | Data collector resources
`spec.ols.deployment.mcpServer` | `Config` | -- | No | -- | Standalone OpenShift MCP server Deployment
`spec.ols.deployment.mcpServer.resources` | `*ResourceRequirements` | -- | No | -- | MCP server resources
`spec.ols.deployment.rhokp` | `ContainerConfig` | -- | No | -- | RHOKP sidecar container
`spec.ols.deployment.rhokp.resources` | `*ResourceRequirements` | -- | No | -- | RHOKP sidecar resources (default requests: 2 CPU, 2 GiB memory, 75 GiB ephemeral storage)
`spec.ols.deployment.rhokp` | `Config` | -- | No | -- | [PLANNED: OLS-3697] Standalone RHOKP Deployment
`spec.ols.deployment.rhokp.replicas` | `*int32` | `1` | No | Min=0 | RHOKP replicas (operator forces 1)
`spec.ols.deployment.rhokp.resources` | `*ResourceRequirements` | -- | No | -- | RHOKP resources (default requests: 2 CPU, 2 GiB memory)
`spec.ols.deployment.rhokp.tolerations` | `[]Toleration` | -- | No | -- | RHOKP tolerations
`spec.ols.deployment.rhokp.nodeSelector` | `map[string]string` | -- | No | -- | RHOKP node selector
`spec.ols.deployment.console` | `Config` | -- | No | -- | Console container
`spec.ols.deployment.console.replicas` | `*int32` | `1` | No | Min=0 | Console replicas (operator forces 1)
`spec.ols.deployment.console.resources` | `*ResourceRequirements` | -- | No | -- | Console resources
Expand Down Expand Up @@ -521,3 +524,4 @@ Path | Type | Default | Required | Validation | Description
- [PLANNED: OLS-3442] Add `reasoningConfig` field (`map[string]interface{}`) to `ModelParametersSpec`. Freeform map passed through to the service as `reasoning_config` for provider-specific reasoning/thinking parameters. Includes release notes and user-facing documentation for valid keys per provider.
- Agentic/sandbox reuse of the standalone MCP Service and CA depends on inter-operator config handoff ([OLS-3572](https://redhat.atlassian.net/browse/OLS-3572)); optional agentic auto-injection is separately deferred ([OLS-3594](https://redhat.atlassian.net/browse/OLS-3594)).
- [PLANNED: OLS-3572] Add `spec.agenticOLS` section with `sandboxMode` field (`bare-pod` default, `sandbox-claim`). The classic operator builds a base `corev1.PodSpec` for sandbox pods and writes it to the `lightspeed-sandbox-config` ConfigMap for the agentic operator. See design spec `docs/superpowers/specs/2026-07-21-inter-operator-handoff.md`.
- [PLANNED: OLS-3697] Change `spec.ols.deployment.rhokp` from `ContainerConfig` to `Config`. RHOKP becomes a standalone Deployment with replicas (forced to 1), resources, tolerations, and nodeSelector. See design spec `docs/superpowers/specs/2026-07-27-rhokp-standalone.md`.
7 changes: 5 additions & 2 deletions .ai/spec/what/ocpmcp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,11 @@ Gated by `spec.ols.introspectionEnabled` (default `true` when absent). When fals
17. Toolsets are pinned to `core`, `config`, `helm`, `metrics`. Metrics uses in-cluster Thanos Querier and Alertmanager URLs. Metrics `guardrails = "!tsdb"` (PromQL query safety, not RBAC) follows upstream OpenShift guidance when Thanos lacks the TSDB status API; auth remains the caller's bearer token.
18. User-defined MCP servers (`spec.mcpServers`) are out of scope for this operand.

### Monitoring [PLANNED: OLS-3697]
19. ServiceMonitor `openshift-mcp-server-monitor` — scrapes MCP server metrics via HTTPS on port 8443, path `/metrics` (Go promhttp). Uses service-ca TLS for the scrape connection. Skipped if Prometheus Operator CRDs are not installed.

### Finalizer
19. On CR deletion, `ocpmcp.Remove()` deletes Deployment, Service, NetworkPolicy, both ConfigMaps, ServiceAccount, and TLS Secret (`openshift-mcp-server-tls`) before owned-resource sweep.
20. On CR deletion, `ocpmcp.Remove()` deletes Deployment, Service, NetworkPolicy, both ConfigMaps, ServiceAccount, TLS Secret (`openshift-mcp-server-tls`), and ServiceMonitor (`openshift-mcp-server-monitor`) before owned-resource sweep.

## Configuration Surface

Expand All @@ -72,4 +75,4 @@ Gated by `spec.ols.introspectionEnabled` (default `true` when absent). When fals

## Planned Changes

None for the standalone HTTPS cutover itself. Agentic handoff remains planned (OLS-3572, OLS-3594).
- [PLANNED: OLS-3697] Add ServiceMonitor for Prometheus scraping of MCP server `/metrics` endpoint via HTTPS. Requires the MCP server binary to expose promhttp metrics on the existing HTTPS port 8443.
Loading