backport 1.10 - fix(lightspeed): pre-create /data/vector_db/notebooks in init container by JslYoon · Pull Request #3042 · redhat-developer/rhdh-operator

JslYoon · 2026-06-22T21:14:41Z

On EKS/AKS, the RAG init container copies /rag/. to /data/ but never creates the notebooks subdirectory. At runtime, llama-stack tries to write /rag-content/vector_db/notebooks/faiss_store.db (same volume, mounted at /rag-content in the sidecar) and fails with PermissionError because it cannot create the directory. OCP avoids this via fsGroup defaults; EKS/AKS do not.

The fix pre-creates /data/vector_db/notebooks before the existing chmod so the directory exists and is writable when the sidecar starts.

Fixes: RHDHBUGS-3371

Description

Which issue(s) does this PR fix or relate to

Fixes #issue_number

PR acceptance criteria

Tests
Documentation

How to test changes / Special notes to the reviewer

rhdh-qodo-merge · 2026-06-22T21:15:28Z

PR Summary by Qodo

Fix Lightspeed RAG permissions by creating vector_db/notebooks in init container
🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

Description

• Create /data/vector_db/notebooks during RAG init copy to prevent runtime PermissionError.
• Ensure directory exists before chmod so sidecar can write FAISS store on EKS/AKS.
• Apply the same init-command fix across source manifests and generated install output.

Diagram

graph TD
  A["RAG init container"] --> B["Copy /rag -> /data"] --> C["mkdir -p notebooks"] --> D[("Shared PVC /data")] --> E["llama-stack sidecar"] --> F["Write faiss_store.db"]
  subgraph Legend
    direction LR
    _step["Container step"] ~~~ _vol[("Persistent volume")]
  end

High-Level Assessment

The following are alternative approaches to this PR:

1. Set fsGroup / runAsGroup on the pod securityContext

➕ Aligns with Kubernetes-native permission management; avoids 777 chmod.
➕ Applies broadly to all paths on the volume without per-directory handling.
➖ Depends on cluster/storage behavior and security policies; may not work uniformly across CSI drivers.
➖ May require broader manifest/security review and could have unintended permission effects.

2. Use chown/chmod with a non-world-writable mode (e.g., 775)

➕ More secure than chmod -R 777 while still solving write access.
➕ Keeps the fix localized to init container behavior.
➖ Requires knowing/standardizing the runtime UID/GID of the sidecar.
➖ More brittle if images/users change.

3. Make llama-stack create the directory on startup

➕ Application-level robustness regardless of init container behavior.
➖ Still fails if the parent path is not writable; doesn’t address underlying volume permission mismatch.
➖ Requires app change/release rather than a manifest-only fix.

Recommendation: The PR’s approach (mkdir -p before existing chmod) is the fastest, lowest-risk fix for EKS/AKS because it directly addresses the missing directory that triggers the PermissionError. If security hardening is a follow-up goal, consider replacing chmod -R 777 with a tighter mode coupled with an explicit fsGroup/runAsGroup strategy once runtime UID/GID assumptions are validated.

Files changed (3) +3 / -3

Bug fix (3) +3 / -3

rhdh-flavour-lightspeed-config_v1_configmap.yamlCreate vector_db/notebooks during RAG init copy in ConfigMap manifest +1/-1

Create vector_db/notebooks during RAG init copy in ConfigMap manifest
• Updates the init container shell command to mkdir -p /data/vector_db/notebooks after copying /rag into /data. Ensures the notebooks directory exists before chmod -R 777 is applied.
bundle/rhdh/manifests/rhdh-flavour-lightspeed-config_v1_configmap.yaml

deployment.yamlCreate vector_db/notebooks in Lightspeed deployment init container command +1/-1

Create vector_db/notebooks in Lightspeed deployment init container command
• Adds mkdir -p /data/vector_db/notebooks to the init container command sequence. Prevents runtime failures when the sidecar tries to create/write the FAISS store under notebooks.
config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml

install.yamlPropagate init container notebooks directory creation into generated install manifest +1/-1

Propagate init container notebooks directory creation into generated install manifest
• Mirrors the same init container command change in the distribution install YAML. Keeps the generated installation output consistent with the source manifests.
dist/rhdh/install.yaml

rhdh-qodo-merge · 2026-06-22T21:17:58Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: RHDHBUGS-3371

1. No fail-fast on missing data 🐞 Bug ☼ Reliability

Description

Because the init command now runs mkdir -p /data/vector_db/notebooks, /data/vector_db will exist
even if the copied image content did not include the required /rag/vector_db tree, so init may
succeed and defer the failure to runtime. This can make incorrectly-packaged/missing RAG content
harder to diagnose since llama-stack expects DB files under /rag-content/vector_db/....

Code

config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[12]

+            - "echo 'Copying RAG data...'; cp -r /rag/. /data/ && mkdir -p /data/vector_db/notebooks && chmod -R 777 /data/vector_db && echo 'Copy complete.'"

Evidence

The init container performs a broad cp -r /rag/. /data/ and then unconditionally ensures
/data/vector_db/notebooks exists, which guarantees /data/vector_db exists even if the copy did
not include it. The runtime configuration explicitly points SQLite DB paths at
/rag-content/vector_db/..., so missing packaged content will surface as runtime DB-open errors
instead of an init-time failure.

config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[6-46]
config/profile/rhdh/default-config/flavours/lightspeed/configmap-files.yaml[163-176]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The new `mkdir -p /data/vector_db/notebooks` changes behavior so the init container can succeed even if `cp -r /rag/. /data/` did not bring in the expected `vector_db` content. This shifts detection from init-time to runtime, where failures are harder to attribute.

### Issue Context
Runtime config references DB paths under `/rag-content/vector_db/...`, so missing packaged content should ideally fail the init container with a clear error.

### Fix Focus Areas
- config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[6-52]
- bundle/rhdh/manifests/rhdh-flavour-lightspeed-config_v1_configmap.yaml[267-313]
- dist/rhdh/install.yaml[3288-3335]

### Suggested fix
After the copy, add explicit checks for required directories/files (example):
- `test -d /data/vector_db/rhdh_product_docs || { echo 'ERROR: missing vector_db content'; exit 1; }`
- optionally `test -f /data/vector_db/rhdh_product_docs/1.10/faiss_store.db || ...`
Then run the `mkdir -p /data/vector_db/notebooks` and permission adjustments.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~World-writable vector DB~~ ✓ Resolved 🐞 Bug ⛨ Security

Description

The init container runs chmod -R 777 /data/vector_db, making the RAG vector database writable by
any UID inside the pod, which allows unintended tampering/corruption of the SQLite/FAISS stores used
by llama-stack. This is a least-privilege violation on a runtime-critical data path mounted into the
main container.

Code

config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[12]

+            - "echo 'Copying RAG data...'; cp -r /rag/. /data/ && mkdir -p /data/vector_db/notebooks && chmod -R 777 /data/vector_db && echo 'Copy complete.'"

Evidence
The deployment mounts the same rag-data-volume at /data in the init container and at
/rag-content in the main container, and the init command applies chmod -R 777 to that shared
path. The lightspeed/llama-stack config stores its SQLite DB at
/rag-content/vector_db/notebooks/faiss_store.db, so broad write permissions apply to
runtime-critical DB files.
config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[6-52]
config/profile/rhdh/default-config/flavours/lightspeed/configmap-files.yaml[163-176]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The init container recursively applies world-writable permissions (`chmod -R 777`) to `/data/vector_db`, which is mounted into the runtime container at `/rag-content`. This broad permission can allow any process UID in the pod to modify or corrupt the vector DB files.

### Issue Context
The runtime configuration uses SQLite DB files under `/rag-content/vector_db/...`, so integrity of this directory matters.

### Fix Focus Areas
- config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml[6-52]
- bundle/rhdh/manifests/rhdh-flavour-lightspeed-config_v1_configmap.yaml[267-313]
- dist/rhdh/install.yaml[3288-3335]

### Suggested fix
Replace `chmod -R 777 /data/vector_db` with a least-privilege approach, e.g.:
- set pod/container `securityContext` (`runAsUser` + `fsGroup`) to the UID/GID the app runs as, and
- use `chmod -R g+rwX` (or `ug+rwX`) on the directory tree, and/or `chown -R <runtime_uid>:<runtime_gid> /data/vector_db` in the init container.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

rm3l · 2026-06-24T11:30:25Z

The CI failures will be fixed by #2293

rm3l · 2026-06-25T07:00:39Z

/build-images 2be164d

github-actions · 2026-06-25T07:03:48Z

✅ PR images built successfully!

Images are available for testing:

Operator: quay.io/rhdh-community/operator:0.10.2-pr-3042-2be164d
Bundle: quay.io/rhdh-community/operator-bundle:0.10.2-pr-3042-2be164d
Catalog: quay.io/rhdh-community/operator-catalog:0.10.2-pr-3042-2be164d

Also available with PR number tag:

quay.io/rhdh-community/operator:0.10.2-pr-3042
quay.io/rhdh-community/operator-bundle:0.10.2-pr-3042
quay.io/rhdh-community/operator-catalog:0.10.2-pr-3042

_{Triggered by @rm3l}

rm3l · 2026-07-01T09:00:27Z

/cherry-pick main

openshift-cherrypick-robot · 2026-07-01T09:00:31Z

@rm3l: once the present PR merges, I will cherry-pick it on top of main in a new PR and assign it to you.

Details

In response to this:

/cherry-pick main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

On EKS/AKS, the RAG init container copies /rag/. to /data/ but never creates the notebooks subdirectory. At runtime, llama-stack tries to write /rag-content/vector_db/notebooks/faiss_store.db (same volume, mounted at /rag-content in the sidecar) and fails with PermissionError because it cannot create the directory. OCP avoids this via fsGroup defaults; EKS/AKS do not. The fix pre-creates /data/vector_db/notebooks before the existing chmod so the directory exists and is writable when the sidecar starts. Fixes: RHDHBUGS-3371 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…ebooks permissions Apply the same fix used in rhdh-chart: - Use cp --no-preserve=mode,ownership so copied RAG files get the container's default permissions rather than inheriting restrictive ones from the source image - Set podSecurityContext.fsGroup: 1001 so Kubernetes chowns volumes to GID 1001 on mount, matching how OCP handles this via SCCs on EKS/AKS where fsGroup is not set automatically Together with the existing mkdir -p /data/vector_db/notebooks this ensures the notebooks directory is writable on all Kubernetes flavours. Fixes: RHDHBUGS-3371 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Mirror the final fix from rhdh-chart (backstage 5.12.5): - Remove podSecurityContext.fsGroup: 1001 added in previous commit; the chart does not set this on the deployment spec - Replace 'chmod -R 777 /data/vector_db' with 'chmod -R a+rwX /data/embeddings_model', matching the chart which only widens permissions on the embeddings_model directory and relies on --no-preserve=mode,ownership + mkdir -p for the rest The operator volume mounts /data in the init container and /rag-content in the sidecar from the same volume, so /data/embeddings_model maps to /rag-content/embeddings_model -- identical to the chart fix. Fixes: RHDHBUGS-3371 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…ent.yaml Co-authored-by: Armel Soro <armel@rm3l.org>

…9/go-toolset:9.8-1782377916 [skip-build] [skip-e2e] (redhat-developer#3113) Signed-off-by: rhdh-bot service account <rhdh-bot@redhat.com>

Signed-off-by: Lucas <lyoon@redhat.com>

sonarqubecloud · 2026-07-01T20:14:08Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

rm3l · 2026-07-01T21:01:28Z

/build-images

github-actions · 2026-07-01T21:04:26Z

✅ PR images built successfully!

Images are available for testing:

Operator: quay.io/rhdh-community/operator:0.10.2-pr-3042-fa4cb27
Bundle: quay.io/rhdh-community/operator-bundle:0.10.2-pr-3042-fa4cb27
Catalog: quay.io/rhdh-community/operator-catalog:0.10.2-pr-3042-fa4cb27

Also available with PR number tag:

quay.io/rhdh-community/operator:0.10.2-pr-3042
quay.io/rhdh-community/operator-bundle:0.10.2-pr-3042
quay.io/rhdh-community/operator-catalog:0.10.2-pr-3042

_{Triggered by @rm3l}

openshift-cherrypick-robot · 2026-07-01T21:47:32Z

@rm3l: #3042 failed to apply on top of branch "main":

Applying: fix(lightspeed): pre-create /data/vector_db/notebooks in init container
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To record the empty patch as an empty commit, run "git am --allow-empty".
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Applying: fix(lightspeed): use --no-preserve=mode,ownership and fsGroup for notebooks permissions
Applying: fix(lightspeed): align init container permissions with rhdh-chart fix
Applying: Update config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml
Patch is empty.

Details

In response to this:

/cherry-pick main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

… in init container (#3042) (#3134) * fix(lightspeed): pre-create /data/vector_db/notebooks in init container On EKS/AKS, the RAG init container copies /rag/. to /data/ but never creates the notebooks subdirectory. At runtime, llama-stack tries to write /rag-content/vector_db/notebooks/faiss_store.db (same volume, mounted at /rag-content in the sidecar) and fails with PermissionError because it cannot create the directory. OCP avoids this via fsGroup defaults; EKS/AKS do not. The fix pre-creates /data/vector_db/notebooks before the existing chmod so the directory exists and is writable when the sidecar starts. Fixes: RHDHBUGS-3371 * fix(lightspeed): use --no-preserve=mode,ownership and fsGroup for notebooks permissions Apply the same fix used in rhdh-chart: - Use cp --no-preserve=mode,ownership so copied RAG files get the container's default permissions rather than inheriting restrictive ones from the source image - Set podSecurityContext.fsGroup: 1001 so Kubernetes chowns volumes to GID 1001 on mount, matching how OCP handles this via SCCs on EKS/AKS where fsGroup is not set automatically Together with the existing mkdir -p /data/vector_db/notebooks this ensures the notebooks directory is writable on all Kubernetes flavours. Fixes: RHDHBUGS-3371 * fix(lightspeed): align init container permissions with rhdh-chart fix Mirror the final fix from rhdh-chart (backstage 5.12.5): - Remove podSecurityContext.fsGroup: 1001 added in previous commit; the chart does not set this on the deployment spec - Replace 'chmod -R 777 /data/vector_db' with 'chmod -R a+rwX /data/embeddings_model', matching the chart which only widens permissions on the embeddings_model directory and relies on --no-preserve=mode,ownership + mkdir -p for the rest The operator volume mounts /data in the init container and /rag-content in the sidecar from the same volume, so /data/embeddings_model maps to /rag-content/embeddings_model -- identical to the chart fix. Fixes: RHDHBUGS-3371 * Update config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml * chore: update release-1.10 to ubi9/go-toolset:9.8-1782736563 from ubi9/go-toolset:9.8-1782377916 [skip-build] [skip-e2e] (#3113) * chore: regenerate bundle manifests --------- Signed-off-by: rhdh-bot service account <rhdh-bot@redhat.com> Signed-off-by: Lucas <lyoon@redhat.com> Co-authored-by: Lucas Yoon <94267691+JslYoon@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: rhdh-bot service account <rhdh-bot@redhat.com>

JslYoon requested a review from a team as a code owner June 22, 2026 21:14

openshift-ci Bot requested review from gazarenkov and rm3l June 22, 2026 21:14

JslYoon requested a deployment to external June 22, 2026 21:14 — with GitHub Actions Waiting

JslYoon temporarily deployed to external June 22, 2026 21:14 — with GitHub Actions Inactive

JslYoon had a problem deploying to external June 22, 2026 21:14 — with GitHub Actions Error

rhdh-qodo-merge Bot added the bug_fix label Jun 22, 2026

JslYoon changed the title ~~fix(lightspeed): pre-create /data/vector_db/notebooks in init container~~ backport 1.10 - fix(lightspeed): pre-create /data/vector_db/notebooks in init container Jun 23, 2026

JslYoon deployed to external June 23, 2026 18:03 — with GitHub Actions Active

JslYoon temporarily deployed to external June 23, 2026 18:03 — with GitHub Actions Inactive

rm3l reviewed Jul 1, 2026

View reviewed changes

Comment thread config/profile/rhdh/default-config/flavours/lightspeed/deployment.yaml Outdated

openshift-ci Bot added the needs-rebase label Jul 1, 2026

JslYoon and others added 5 commits July 1, 2026 16:10

Update config/profile/rhdh/default-config/flavours/lightspeed/deploym…

8a999c9

…ent.yaml Co-authored-by: Armel Soro <armel@rm3l.org>

chore: update release-1.10 to ubi9/go-toolset:9.8-1782736563 from ubi…

c7f973c

…9/go-toolset:9.8-1782377916 [skip-build] [skip-e2e] (redhat-developer#3113) Signed-off-by: rhdh-bot service account <rhdh-bot@redhat.com>

JslYoon force-pushed the fix/RHDHBUGS-3371-lightspeed-notebooks-permissions branch from ab694f4 to 14b8bc0 Compare July 1, 2026 20:10

openshift-ci Bot removed the needs-rebase label Jul 1, 2026

chore: regenerate bundle manifests

fa4cb27

Signed-off-by: Lucas <lyoon@redhat.com>

JslYoon force-pushed the fix/RHDHBUGS-3371-lightspeed-notebooks-permissions branch from 14b8bc0 to fa4cb27 Compare July 1, 2026 20:13

JslYoon requested a review from rm3l July 1, 2026 20:14

rm3l approved these changes Jul 1, 2026

View reviewed changes

openshift-ci Bot assigned rm3l Jul 1, 2026

openshift-ci Bot added the lgtm label Jul 1, 2026

openshift-merge-bot Bot merged commit 2b81b22 into redhat-developer:release-1.10 Jul 1, 2026
8 checks passed

This was referenced Jul 2, 2026

fix(lightspeed): fix rag-content folder ownership in init container #3133

Closed

[main] fix(lightspeed): fix rag-content folder ownership in init container #3134

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

backport 1.10 - fix(lightspeed): pre-create /data/vector_db/notebooks in init container#3042

backport 1.10 - fix(lightspeed): pre-create /data/vector_db/notebooks in init container#3042
openshift-merge-bot[bot] merged 6 commits into
redhat-developer:release-1.10from
JslYoon:fix/RHDHBUGS-3371-lightspeed-notebooks-permissions

JslYoon commented Jun 22, 2026 •

edited by openshift-ci Bot

Loading

Uh oh!

rhdh-qodo-merge Bot commented Jun 22, 2026

Uh oh!

rhdh-qodo-merge Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

rm3l commented Jun 24, 2026

Uh oh!

rm3l commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Uh oh!

rm3l commented Jul 1, 2026

Uh oh!

openshift-cherrypick-robot commented Jul 1, 2026

Uh oh!

sonarqubecloud Bot commented Jul 1, 2026

Uh oh!

rm3l commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

Uh oh!

openshift-cherrypick-robot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

JslYoon commented Jun 22, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Which issue(s) does this PR fix or relate to

PR acceptance criteria

How to test changes / Special notes to the reviewer

Uh oh!

rhdh-qodo-merge Bot commented Jun 22, 2026

PR Summary by Qodo

Uh oh!

rhdh-qodo-merge Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

rm3l commented Jun 24, 2026

Uh oh!

rm3l commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Uh oh!

rm3l commented Jul 1, 2026

Uh oh!

openshift-cherrypick-robot commented Jul 1, 2026

Uh oh!

sonarqubecloud Bot commented Jul 1, 2026

Quality Gate passed

Uh oh!

rm3l commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

Uh oh!

openshift-cherrypick-robot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JslYoon commented Jun 22, 2026 •

edited by openshift-ci Bot

Loading

rhdh-qodo-merge Bot commented Jun 22, 2026 •

edited

Loading