no-jira: Add skillsaw lint and LLM security scan Prow jobs for agentic-skills by stbenjam · Pull Request #79006 · openshift/release

stbenjam · 2026-05-07T19:50:05Z

Summary

Add reusable step-registry refs openshift-harness-lint and openshift-harness-security-scan for agent skill validation
openshift-harness-lint runs skillsaw to validate skills against the agentskills.io spec
openshift-harness-security-scan runs cisco-ai-skill-scanner with LLM-as-judge (Vertex AI / Claude Sonnet 4.6) to detect prompt injection, data exfiltration, and command injection
Wire both refs to the openshift/agentic-skills main branch CI config, replacing the previous inline eval test
Both refs share a single skill-tools pipeline image

Summary

This PR updates OpenShift CI configuration to add two reusable step-registry refs and wire them into the openshift/agentic-skills main-branch pipeline, replacing the previous inline eval test.

What changed (practical terms)

Affected area: openshift/agentic-skills CI (ci-operator config) and new step-registry entries under ci-operator/step-registry/openshift/harness.
The agentic-skills CI image build now installs both skillsaw and cisco-ai-skill-scanner[vertex] and publishes that image as to: skill-tools. The main-branch pipeline replaces the old inline eval test with two reusable refs:
- lint — ref: openshift-harness-lint (runs skillsaw)
- security-scan — ref: openshift-harness-security-scan (runs cisco-ai-skill-scanner, LLM-enabled by default)
Both jobs use a skip-if-only-changed pattern: ^(README.md|OWNERS|LICENSE|.gitignore)$.

New reusable step-registry refs

openshift-harness-lint
- Runs skillsaw via openshift-harness-lint-commands.sh from the skill-tools image.
- Validates agentskills.io frontmatter, naming conventions, directory structure, and eval formats.
- Requests: 100m CPU / 100Mi RAM; timeout 5m; 30s grace period.
openshift-harness-security-scan
- Runs skill-scanner (cisco-ai-skill-scanner) via openshift-harness-security-scan-commands.sh.
- Default enables LLM-based semantic analysis (USE_LLM="true") with SKILL_SCANNER_LLM_MODEL=vertex_ai/claude-sonnet-4-6 and VERTEXAI_LOCATION=global.
- Emits JSON and HTML reports to ARTIFACT_DIR, fails on medium severity findings by default, and accepts extra CLI args via SCAN_ADDITIONAL_ARGS.
- Mounts credential sa-claude-openshift-ci from namespace test-credentials and sets GOOGLE_APPLICATION_CREDENTIALS to the mounted token path.
- Requests: 100m CPU / 100Mi RAM; timeout 10m; 5m grace period.

Metadata and OWNERS

Added/updated OWNERS and step-ref metadata for the new refs and parent harness directory, adding approvers and reviewers: bryan-cox, cblecker, Cali0707, enxebre, stbenjam.

Test plan / operational notes

Prow rehearsals must pass for the new lint and security-scan jobs.
The sa-claude-openshift-ci credential must be provisioned in the test-credentials namespace for LLM-based scans to authenticate to Vertex AI.

openshift-ci-robot · 2026-05-07T19:50:10Z

@stbenjam: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

Add reusable step-registry refs openshift-harness-lint and openshift-harness-security-scan for agent skill validation

openshift-harness-lint runs skillsaw to validate skills against the agentskills.io spec

openshift-harness-security-scan runs cisco-ai-skill-scanner with LLM-as-judge (Vertex AI / Claude Sonnet 4.6) to detect prompt injection, data exfiltration, and command injection

Wire both refs to the openshift/agentic-skills main branch CI config, replacing the previous inline eval test

Both refs share a single skill-tools pipeline image

Test plan

Prow rehearsal passes for both lint and security-scan jobs

Verify sa-claude-openshift-ci credential is provisioned in test-credentials namespace

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-05-07T19:50:36Z

Walkthrough

Installs cisco-ai-skill-scanner[vertex] into the CI image, replaces the previous eval test with two harness steps (lint using openshift-harness-lint/skillsaw and security-scan using openshift-harness-security-scan/skill-scanner with optional Vertex LLM), and adds step refs, command scripts, metadata, and OWNERS entries.

Changes

Lint & Security Scan Pipeline Steps

Layer / File(s)	Summary
Image build / Package changes `ci-operator/config/openshift/agentic-skills/.../openshift-agentic-skills-main.yaml`	CI image build updated to install `cisco-ai-skill-scanner[vertex]` and `skillsaw`, and repository copied into the image.
Pipeline wiring `ci-operator/config/openshift/agentic-skills/.../openshift-agentic-skills-main.yaml`	Removed previous `eval` usage of `skill-scanner` and added two test stages: `lint` (uses `openshift-harness-lint`) and `security-scan` (uses `openshift-harness-security-scan`), each gated by `skip_if_only_changed` for `README.md
Step registry refs (wiring & config) `ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.yaml`, `ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.yaml`	Added `openshift-harness-lint` ref (from `skill-tools`, small resources, 5m timeout) and `openshift-harness-security-scan` ref (from `skill-tools`, mounts CI credential, sets `GOOGLE_APPLICATION_CREDENTIALS`, exposes env vars for LLM/model/location/toggles, requests resources, 10m timeout).
Command scripts `ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-commands.sh`, `ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-commands.sh`	Added lint script (`set -euo pipefail`, echoes message, runs `skillsaw -v`) and security-scan script (strict shell, optional `--use-llm` when `USE_LLM=true`, runs `skill-scanner scan-all` producing JSON/HTML to `${ARTIFACT_DIR}`, `--fail-on-severity medium`, accepts `${SCAN_ADDITIONAL_ARGS}`).
Step metadata / OWNERS `ci-operator/step-registry/openshift/harness//openshift-harness--ref.metadata.json`, `ci-operator/step-registry/openshift/harness/*/OWNERS`, `ci-operator/step-registry/openshift/harness/OWNERS`	Added metadata JSON files pointing to new ref YAMLs and added/defined `approvers` and `reviewers` lists (five usernames) for harness, lint, and security-scan directories.

sequenceDiagram
    participant CI as CI Pipeline
    participant Image as skill-tools Image
    participant Lint as Skillsaw Linter
    participant Scanner as Skill-Scanner Step
    participant Vertex as Vertex AI
    participant Artifacts as Artifact Store

    CI->>Image: Build image (includes cisco-ai-skill-scanner[vertex], skillsaw)
    CI->>Lint: Run openshift-harness-lint step
    Lint->>Lint: execute `skillsaw -v`
    CI->>Scanner: Run openshift-harness-security-scan step
    Scanner->>Scanner: prepare env & credential (GOOGLE_APPLICATION_CREDENTIALS)
    alt USE_LLM=true
        Scanner->>Vertex: request semantic analysis (model via SKILL_SCANNER_LLM_MODEL)
        Vertex-->>Scanner: semantic analysis results
    end
    Scanner->>Artifacts: write JSON/HTML reports to ${ARTIFACT_DIR}

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 12

✅ Passed checks (12 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: adding skillsaw lint and LLM security scan Prow jobs for agentic-skills, which directly corresponds to the file additions and CI configuration updates in the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Check is not applicable to this PR. The custom check targets Ginkgo test names in Go code, but this PR contains only CI/pipeline YAML configuration, shell scripts, and metadata files—no Go test code.
Test Structure And Quality	✅ Passed	Custom check for Ginkgo test code is not applicable. PR contains only CI configuration (YAML), bash scripts, and metadata files—no Go test code.
Microshift Test Compatibility	✅ Passed	No Ginkgo e2e tests are added in this PR. It only adds CI/CD infrastructure (step-registry configs and shell scripts). The MicroShift Test Compatibility check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	This PR does not add any Ginkgo e2e tests. It adds CI pipeline steps (shell scripts and YAML configuration) for linting and security scanning. The custom check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	This PR modifies CI/CD pipeline configuration and step-registry definitions, not deployment manifests, operator code, or controllers. The custom check is not applicable to CI infrastructure files.
Ote Binary Stdout Contract	✅ Passed	Check not applicable. PR contains only CI pipeline configuration and shell scripts. No Go test code or OTE binaries present that could violate stdout contract.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR adds only CI/Prow configuration and bash scripts for tools, not Ginkgo e2e tests. IPv6/disconnected network check is not applicable.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2026-05-07T19:51:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: stbenjam
Once this PR has been reviewed and has the lgtm label, please assign harche for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

ci-operator/config/openshift/agentic-skills/OWNERS
~~ci-operator/jobs/openshift/agentic-skills/OWNERS~~ [stbenjam]
~~ci-operator/step-registry/openshift/OWNERS~~ [stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift/agentic-skills/openshift-agentic-skills-main.yaml`:
- Line 16: Update the Dockerfile RUN line that performs pip install: replace the
unpinned "cisco-ai-skill-scanner[vertex]" and the nonexistent "skillsaw" with
exact pinned package versions (e.g., "cisco-ai-skill-scanner[vertex]==2.0.11")
and either correct the intended package name and pin "skillsaw" to its verified
package name and version or remove it if it was included in error; modify the
RUN pip install "cisco-ai-skill-scanner[vertex]" skillsaw entry accordingly so
CI installs only valid, version-pinned packages.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 90a10552-6a40-4884-ae9d-de5f7e97094a

📥 Commits

Reviewing files that changed from the base of the PR and between 4b53613 and db2950c.

⛔ Files ignored due to path filters (1)

ci-operator/jobs/openshift/agentic-skills/openshift-agentic-skills-main-presubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (9)

ci-operator/config/openshift/agentic-skills/openshift-agentic-skills-main.yaml
ci-operator/step-registry/openshift/harness/lint/OWNERS
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-commands.sh
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.metadata.json
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.yaml
ci-operator/step-registry/openshift/harness/security-scan/OWNERS
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-commands.sh
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.metadata.json
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.yaml

stbenjam · 2026-05-07T20:01:44Z

/pj-rehearse

openshift-merge-bot · 2026-05-07T20:01:47Z

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

stbenjam · 2026-05-07T20:15:14Z

/pj-rehearse

openshift-merge-bot · 2026-05-07T20:15:17Z

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Add reusable step-registry refs for agent skill linting and security scanning, and wire them up to the agentic-skills CI config: - openshift-harness-lint: runs skillsaw to validate skills against the agentskills.io specification - openshift-harness-security-scan: runs cisco-ai-skill-scanner with LLM-as-judge (Vertex AI / Claude Sonnet 4.6) for prompt injection, data exfiltration, and command injection detection Both refs share a single skill-tools pipeline image built with cisco-ai-skill-scanner[vertex] and skillsaw. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-merge-bot · 2026-05-07T20:48:18Z

[REHEARSALNOTIFIER]
@stbenjam: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-agentic-skills-main-lint	openshift/agentic-skills	presubmit	Presubmit changed
pull-ci-openshift-agentic-skills-main-security-scan	openshift/agentic-skills	presubmit	Presubmit changed
pull-ci-openshift-agentic-skills-main-images	openshift/agentic-skills	presubmit	Ci-operator config changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

stbenjam · 2026-05-07T20:50:49Z

/pj-rehearse

openshift-merge-bot · 2026-05-07T20:50:52Z

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-commands.sh`:
- Around line 15-21: The command appends ${SCAN_ADDITIONAL_ARGS} after the
enforced flag so users can override --fail-on-severity; move
${SCAN_ADDITIONAL_ARGS} to appear before --fail-on-severity in the skill-scanner
invocation (i.e., place ${SCAN_ADDITIONAL_ARGS} earlier in the argument list for
the skill-scanner scan-all command) so that the explicit --fail-on-severity
medium option cannot be superseded by later duplicate options.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 1da9b7e6-4328-4acb-b73d-0b546f435247

📥 Commits

Reviewing files that changed from the base of the PR and between 41072a1 and 4850ca0.

⛔ Files ignored due to path filters (1)

ci-operator/jobs/openshift/agentic-skills/openshift-agentic-skills-main-presubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (10)

ci-operator/config/openshift/agentic-skills/openshift-agentic-skills-main.yaml
ci-operator/step-registry/openshift/harness/OWNERS
ci-operator/step-registry/openshift/harness/lint/OWNERS
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-commands.sh
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.metadata.json
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.yaml
ci-operator/step-registry/openshift/harness/security-scan/OWNERS
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-commands.sh
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.metadata.json
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.yaml

✅ Files skipped from review due to trivial changes (7)

ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-commands.sh
ci-operator/step-registry/openshift/harness/security-scan/OWNERS
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.yaml
ci-operator/step-registry/openshift/harness/OWNERS
ci-operator/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-ref.metadata.json
ci-operator/step-registry/openshift/harness/lint/OWNERS
ci-operator/step-registry/openshift/harness/lint/openshift-harness-lint-ref.metadata.json

🚧 Files skipped from review as they are similar to previous changes (1)

ci-operator/config/openshift/agentic-skills/openshift-agentic-skills-main.yaml

openshift-ci · 2026-05-07T21:14:20Z

@stbenjam: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/agentic-skills/main/lint	`4850ca0`	link	unknown	`/pj-rehearse pull-ci-openshift-agentic-skills-main-lint`
ci/rehearse/openshift/agentic-skills/main/images	`4850ca0`	link	unknown	`/pj-rehearse pull-ci-openshift-agentic-skills-main-images`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

stbenjam · 2026-05-07T21:26:01Z

images is aws outage
lint will be fixed by openshift/agentic-skills#19

/pj-rehearse ack

openshift-merge-bot · 2026-05-07T21:26:03Z

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci · 2026-05-18T09:14:44Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 7, 2026

openshift-ci Bot requested review from harche and mrunalp May 7, 2026 19:50

coderabbitai Bot reviewed May 7, 2026

View reviewed changes

Comment thread ci-operator/config/openshift/agentic-skills/openshift-agentic-skills-main.yaml

openshift-ci Bot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label May 7, 2026

openshift-ci Bot removed the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label May 7, 2026

stbenjam force-pushed the agentic-skills-prow-jobs branch from 41072a1 to 4850ca0 Compare May 7, 2026 20:44

coderabbitai Bot reviewed May 7, 2026

View reviewed changes

Comment thread ...or/step-registry/openshift/harness/security-scan/openshift-harness-security-scan-commands.sh

openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label May 7, 2026

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 18, 2026

Conversation

stbenjam commented May 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Summary

What changed (practical terms)

New reusable step-registry refs

Metadata and OWNERS

Test plan / operational notes

Uh oh!

openshift-ci-robot commented May 7, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

openshift-ci Bot commented May 7, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stbenjam commented May 7, 2026

Uh oh!

openshift-merge-bot Bot commented May 7, 2026

Uh oh!

stbenjam commented May 7, 2026

Uh oh!

openshift-merge-bot Bot commented May 7, 2026

Uh oh!

openshift-merge-bot Bot commented May 7, 2026

Uh oh!

stbenjam commented May 7, 2026

Uh oh!

openshift-merge-bot Bot commented May 7, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

openshift-ci Bot commented May 7, 2026

Uh oh!

stbenjam commented May 7, 2026

Uh oh!

openshift-merge-bot Bot commented May 7, 2026

Uh oh!

openshift-ci Bot commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stbenjam commented May 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 7, 2026 •

edited

Loading