Skip to content

[llmd] Rework the test harness and its surrounding #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kpouget merged 34 commits into from
Jun 3, 2026
Merged

[llmd] Rework the test harness and its surrounding #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a9b2db5
[llm-d] s/LOGGER/logger
kpouget Jun 1, 2026
d8b2625
[llm_] Move the K8s tools to core/dsl/utils/k8s
kpouget Jun 1, 2026
b2c0901
[llm_d] per-step vault initialization
kpouget Jun 1, 2026
38c2e75
[llm-d] remove the test_runtime for the time being
kpouget Jun 1, 2026
669c7f8
[llm_d] toolbox: finish moving things around
kpouget Jun 1, 2026
ac914f2
[llm_d] toolbox: finish moving things around
kpouget Jun 2, 2026
80fb8d5
[cluster] toolbox: cluster_deploy_operator: extend
kpouget Jun 2, 2026
2851424
[llm-d] Improve the toolbox
kpouget Jun 2, 2026
6d06d2d
[llm_d] orchestration: config: fix the leader-worker-set channel
kpouget Jun 2, 2026
a8d849f
[core] dsl: runtime: use exec
kpouget Jun 2, 2026
45b9010
[core] dsl: improve the signal handling
kpouget Jun 2, 2026
e6d08d7
[core] dsl: improve the retry logging
kpouget Jun 2, 2026
1601fa8
fournos: gitops: base/workflows/pipeline-prepare-only: new pipeline f…
kpouget Jun 2, 2026
5d3e9c0
[kserve] toolbox: deploy_llmisvc: update
kpouget Jun 2, 2026
a78373d
[llm_d] orchestration: ci: correctly initialize the vault
kpouget Jun 2, 2026
27da635
[core] dsl: runtime: reorder and stringify
kpouget Jun 3, 2026
eb3dd0b
[rhoai] toolbox: apply_datasciencecluster: always capture the DSC
kpouget Jun 3, 2026
64f7802
[rhoai] toolbox: wait_datasciencecluster: always capture the DSC
kpouget Jun 3, 2026
42714a0
[core] library: config: don't merge the __multiple preset
kpouget Jun 3, 2026
d3457e3
[core] library: config: generate presets_applied.txt in CI_METADATA_D…
kpouget Jun 3, 2026
64a8390
[core] ci_entrypoint: prepare_ci: add flag to always send notifications
kpouget Jun 3, 2026
faf10b6
fournos: gitops: base/workflows/task-forge: add always-send-notificat…
kpouget Jun 3, 2026
b64da3f
remove wait_timeout_seconds
kpouget Jun 3, 2026
da5a219
New wait_for_crds toolbox
kpouget Jun 3, 2026
5da30e7
improve
kpouget Jun 3, 2026
a8c1377
[llm_d] toolbox: cleanup_test_resources: new toolbox script
kpouget Jun 3, 2026
cd42a3f
improve
kpouget Jun 3, 2026
22c9b9d
Improve
kpouget Jun 3, 2026
bb72dc7
Improve
kpouget Jun 3, 2026
96600d8
Improve
kpouget Jun 3, 2026
1336d71
Keep working
kpouget Jun 3, 2026
f200419
Keep working
kpouget Jun 3, 2026
8bb3b01
pyproject: update ruff to use 3.12
kpouget Jun 3, 2026
c54af5a
fournos: gitops: base/workflows/pipeline-prepare-test: new pipeline
kpouget Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/toolbox/dsl.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Place `@always` tasks **after** the main pipeline so they behave as teardown (se
- Writes metadata (`_meta/metadata.yaml`, `_meta/restart.sh`) and `task.log`.
- Runs tasks from the **calling file** only (`ScriptManager` path must match `Path(__file__).relative_to(FORGE_HOME)` vs `os.path.relpath` at task definition — run commands from the repository root as the toolbox does).

Interrupts (`KeyboardInterrupt`, `SignalError`) stop execution and still emit completion banners where implemented (not covered by `test_dsl_toolbox.py`; see `runtime.py`).
Interrupts (`KeyboardInterrupt`, `SignalInterrupt`) stop execution and still emit completion banners where implemented (not covered by `test_dsl_toolbox.py`; see `runtime.py`).

### Trace and artifacts (post-mortem)

Expand Down Expand Up @@ -133,4 +133,4 @@ Declare orchestration and operator inputs on the **public entrypoint** (typed pa
| Decorator stack | `@retry` / `@when` **without** `@task` raise **`TypeError` at definition time** with the “Put `@task` BELOW …” message. |
| Success return | `execute_tasks` returns **`shared_context`** with task attributes and **`artifact_dir`** set. |

Not in that file: interrupt handling (`KeyboardInterrupt` / `SignalError`), and CLI wiring (`create_toolbox_main` / `run_toolbox_command`)—those are documented above but not exercised by these unit tests.
Not in that file: interrupt handling (`KeyboardInterrupt` / `SignalInterrupt`), and CLI wiring (`create_toolbox_main` / `run_toolbox_command`)—those are documented above but not exercised by these unit tests.
59 changes: 59 additions & 0 deletions fournos/gitops/base/workflows/pipeline-prepare-only.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Single-step pipeline: run only (no prepare, no cleanup).
# Use on pre-configured clusters where setup/teardown is handled externally.
Comment on lines +1 to +2
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Header comment contradicts the pipeline. This is forge-prepare-only and runs a prepare step, but the comment reads "run only (no prepare, no cleanup)" — looks copy/pasted from the run-only pipeline.

📝 Proposed fix 
-# Single-step pipeline: run only (no prepare, no cleanup).
-# Use on pre-configured clusters where setup/teardown is handled externally.
+# Single-step pipeline: prepare only (no run, no cleanup).
+# Use to provision/prepare a cluster without executing the workload or teardown.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Single-step pipeline: run only (no prepare, no cleanup).
# Use on pre-configured clusters where setup/teardown is handled externally.
# Single-step pipeline: prepare only (no run, no cleanup).
# Use to provision/prepare a cluster without executing the workload or teardown.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@fournos/gitops/base/workflows/pipeline-prepare-only.yaml` around lines 1 - 2,
Update the header comment to accurately describe the pipeline named
"forge-prepare-only": replace the misleading "Single-step pipeline: run only (no
prepare, no cleanup)" text with a brief note that this pipeline runs only the
prepare step (no run/cleanup) on pre-configured clusters, so the comment matches
the pipeline behavior for "forge-prepare-only".

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: forge-prepare-only
annotations:
fournos.dev/resolve-image: image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/forge-core:latest
spec:
workspaces:
- name: artifacts
params:
- name: fjob-name
type: string
- name: fournos-workload-namespace
type: string
- name: kubeconfig-secret
type: string
tasks:
- name: prepare
taskRef:
name: forge-step
kind: Task
workspaces:
- name: artifacts
workspace: artifacts
params:
- name: fjob-name
value: "$(params.fjob-name)"
- name: fournos-workload-namespace
value: "$(params.fournos-workload-namespace)"
- name: kubeconfig-secret
value: "$(params.kubeconfig-secret)"
- name: job-step
value: prepare
- name: job-step-name
value: 000__prepare
- name: always-send-notification
value: "true"

finally:
- name: export-artifacts
taskRef:
name: forge-step
kind: Task
workspaces:
- name: artifacts
workspace: artifacts
params:
- name: fjob-name
value: "$(params.fjob-name)"
- name: fournos-workload-namespace
value: "$(params.fournos-workload-namespace)"
- name: kubeconfig-secret
value: "$(params.kubeconfig-secret)"
- name: job-step
value: export-artifacts
- name: job-step-name
value: 001__export-artifacts
77 changes: 77 additions & 0 deletions fournos/gitops/base/workflows/pipeline-prepare-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Full pipeline: pre_cleanup -> prepare → test → export-artifacts + post-cleanup.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: forge-prepare-test
annotations:
fournos.dev/resolve-image: image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/forge-core:latest
spec:
workspaces:
- name: artifacts
params:
- name: fjob-name
type: string
- name: fournos-workload-namespace
type: string
- name: kubeconfig-secret
type: string

tasks:
- name: 01-prepare
taskRef:
name: forge-step
kind: Task
workspaces:
- name: artifacts
workspace: artifacts
params:
- name: fjob-name
value: "$(params.fjob-name)"
- name: fournos-workload-namespace
value: "$(params.fournos-workload-namespace)"
- name: kubeconfig-secret
value: "$(params.kubeconfig-secret)"
- name: job-step
value: prepare
- name: job-step-name
value: 001__prepare

- name: 02-test
runAfter: [01-prepare]
taskRef:
name: forge-step
kind: Task
workspaces:
- name: artifacts
workspace: artifacts
params:
- name: fjob-name
value: "$(params.fjob-name)"
- name: fournos-workload-namespace
value: "$(params.fournos-workload-namespace)"
- name: kubeconfig-secret
value: "$(params.kubeconfig-secret)"
- name: job-step
value: test
- name: job-step-name
value: 002__test

finally:
- name: 03-export-artifacts
taskRef:
name: forge-step
kind: Task
workspaces:
- name: artifacts
workspace: artifacts
params:
- name: fjob-name
value: "$(params.fjob-name)"
- name: fournos-workload-namespace
value: "$(params.fournos-workload-namespace)"
- name: kubeconfig-secret
value: "$(params.kubeconfig-secret)"
- name: job-step
value: export-artifacts
- name: job-step-name
value: 003__export-artifacts
5 changes: 5 additions & 0 deletions fournos/gitops/base/workflows/task-forge-step.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ spec:
type: string
- name: job-step-name
type: string
- name: always-send-notification
type: string
default: "false"
steps:
- name: forge
image: image-registry.openshift-image-registry.svc:5000/$(context.taskRun.namespace)/forge-core:latest
Expand All @@ -36,6 +39,8 @@ spec:
value: /var/run/secrets/fournos
- name: FOURNOS_CI
value: "true"
- name: PSAP_FORGE_ALWAYS_SEND_NOTIFICATION
value: "$(params.always-send-notification)"
volumeMounts:
- name: kubeconfig
mountPath: /var/run/secrets/fournos-kubeconfig
Expand Down
Loading
Loading