RHIDP-14083: extend customization profile for safety shield prompts - #2374
RHIDP-14083: extend customization profile for safety shield prompts#2374Jdubrick wants to merge 2 commits into
Conversation
When model_prompt or invalid_question_response are omitted, fill them at configuration load from the profile (then LCORE defaults) so GET /v1/shields and agent QV see effective text without duplicating YAML. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Warning Review limit reached
Next review available in: 49 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
WalkthroughQuestion-validity configuration fields are now nullable and resolve from custom profiles or built-in defaults. ChangesQuestion-validity resolution
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Configuration
participant CustomProfile
participant QuestionValidity
participant Moderation
CustomProfile->>Configuration: Load validation and invalid-response values
Configuration->>QuestionValidity: Pass resolved prompt configuration
QuestionValidity->>Moderation: Submit rendered agent input or raw responses input
Moderation-->>QuestionValidity: Return validation result
QuestionValidity-->>Configuration: Return refusal text and moderation state
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 7✅ Passed checks (7 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/models/successful_responses.md`:
- Around line 576-577: Update the PROFILE_CONFIG references in the validation
and invalid_resp table entries to use Markdown-safe subscript notation, such as
double-quoted keys or escaped brackets, so markdownlint MD052 passes while
preserving the documented configuration paths.
In `@tests/unit/app/endpoints/test_shields.py`:
- Line 106: Update the test covering the changed model_prompt in the endpoint
response to assert that the returned prompt includes the configured “Is this
question valid? ${message}” value, alongside the existing model_id assertion.
Anchor the assertion to the response validation in the test so it verifies the
endpoint preserves the updated prompt.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6d84467a-998d-43ab-bb2c-0c201d99ff99
📒 Files selected for processing (19)
docs/devel_doc/openapi.jsondocs/models/successful_responses.jsondocs/models/successful_responses.mddocs/user_doc/config.mddocs/user_doc/shields_guide.mdexamples/lightspeed-stack-shields.yamlsrc/models/api/responses/successful/catalog.pysrc/models/config.pysrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pytests/profiles/empty_qv_strings/profile.pytests/profiles/no_qv_keys/profile.pytests/profiles/test/profile.pytests/unit/app/endpoints/test_shields.pytests/unit/models/responses/test_successful_responses.pytests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.pytests/unit/qv_config.pytests/unit/test_configuration.pytests/unit/utils/test_prompts.pytests/unit/utils/test_pydantic_ai.py
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-0-7-on-pull-request
⚠️ CI failures not shown inline (1)
GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-0-8-on-pull-request: Failed
Conclusion: failure
Konflux kflux-prd-rh02/lightspeed-stack-0-8-on-pull-request has <b>failed</b>.
<ul>
<li><b>Namespace</b>: <a href="https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant">lightspeed-core-tenant</a></li>
<li><b>PipelineRun:</b> <a href="https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/pipelinerun/lightspeed-stack-0-8-on-pull-request-24kll">lightspeed-stack-0-8-on-pull-request-24kll</a></li>
</ul>
<hr>
<h4>Task Statuses:</h4>
<table>
<tr><th>Status</th><th>Duration</th><th>Name</th></tr>
<tr>
<td>🟢 Succeeded</td>
<td>6 seconds</td><td>
[init](https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/pipelinerun/lightspeed-stack-0-8-on-pull-request-24kll/logs/init)
</td></tr>
<tr>
<td>🟢 Succeeded</td>
<td>16 seconds</td><td>
[clone-repository](https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/pipelinerun/lightspeed-stack-0-8-on-pull-request-24kll/logs/clone-repository)
</td></tr>
<tr>
<td>🟢 Succeeded</td>
<td>10 seconds</td><td>
[prefetch-dependencies](https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/pipelinerun/lightspeed-stack-0-8-on-pull-request-24kll/logs/prefetch-dependencies)
</td></tr>
<tr>
<td>🔴 Failed</td>
<td>12 seconds</td><td>
[build-images](https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/pipelinerun/lightspeed-stack-0-8-on-pull-request-24kll/logs/build-images)
</td></tr>
</table>
🧰 Additional context used
📓 Path-based instructions (4)
**/*
📄 CodeRabbit inference engine (Custom checks)
**/*: Flag meaningful O(n^2)+ algorithms on non-trivial inputs, including handlers and Kubernetes list operations.
Flag N+1 patterns that list items and then query once per item, including Kubernetes API and database access.
Flag expensive work inside loops, including API calls, JSON parsing, and regex compilation.
Flag unbounded growth in caches, watchers, or buffers when eviction or limits are missing.
Flag missing pagination or limits on list operations and API endpoints.
Flag secrets or tokens logged in plaintext or hardcoded in source.
Flag API endpoints missing authentication or authorization.
Flag injection vulnerabilities, including SQL injection, command injection, and path traversal.
Flag sensitive data leaked in API responses, WebSocket messages, or logs.
Flag Kubernetes Secrets and Red Hat secrets missing OwnerReferences.Never commit secrets or keys; use environment variables for sensitive data.
Files:
tests/profiles/no_qv_keys/profile.pytests/profiles/empty_qv_strings/profile.pytests/unit/qv_config.pydocs/user_doc/config.mdsrc/models/api/responses/successful/catalog.pyexamples/lightspeed-stack-shields.yamltests/profiles/test/profile.pydocs/models/successful_responses.jsontests/unit/utils/test_pydantic_ai.pydocs/user_doc/shields_guide.mdsrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pytests/unit/app/endpoints/test_shields.pydocs/models/successful_responses.mddocs/devel_doc/openapi.jsontests/unit/test_configuration.pysrc/models/config.pytests/unit/utils/test_prompts.pytests/unit/models/responses/test_successful_responses.pytests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.py
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use absolute imports for internal Python modules.
Every module must begin with a descriptive docstring explaining its purpose.
Uselogger = get_logger(__name__)fromlog.pyfor module logging.
Define shared constants in the centralconstants.pymodule and annotate constants withFinal[type].
All functions must have complete parameter and return type annotations and descriptive docstrings.
Use snake_case, descriptive, action-oriented names for functions, such asget_,validate_, andcheck_.
Avoid modifying mutable parameters in place; return a new data structure instead.
Useasync deffor I/O operations and external API calls.
HandleAPIConnectionErrorfrom Llama Stack integrations.
Use standard logger levels appropriately:debugfor diagnostics,infofor general execution,warningfor unexpected or potentially problematic conditions, anderrorfor serious failures.
All classes must have descriptive docstrings, use PascalCase names, and provide complete, specific type annotations for class attributes.
Use Google Python docstring conventions, includingParameters,Returns,Raises, andAttributessections when applicable.
Files:
tests/profiles/no_qv_keys/profile.pytests/profiles/empty_qv_strings/profile.pytests/unit/qv_config.pysrc/models/api/responses/successful/catalog.pytests/profiles/test/profile.pytests/unit/utils/test_pydantic_ai.pysrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pytests/unit/app/endpoints/test_shields.pytests/unit/test_configuration.pysrc/models/config.pytests/unit/utils/test_prompts.pytests/unit/models/responses/test_successful_responses.pytests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.py
tests/unit/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
tests/unit/**/*.py: Use pytest for unit tests; do not use unittest.
Useconftest.pyfor shared fixtures,pytest-mockfor mocks, andpytest.mark.asynciofor asynchronous tests.
Files:
tests/unit/qv_config.pytests/unit/utils/test_pydantic_ai.pytests/unit/app/endpoints/test_shields.pytests/unit/test_configuration.pytests/unit/utils/test_prompts.pytests/unit/models/responses/test_successful_responses.pytests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.py
src/models/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/models/**/*.py: Pydantic configuration models must extendConfigurationBase; data models must extendBaseModel.
Use@field_validatorand@model_validatorfor Pydantic model validation; model validators should usetyping_extensions.Selfwhere applicable.
Files:
src/models/api/responses/successful/catalog.pysrc/models/config.py
🧠 Learnings (6)
📚 Learning: 2026-06-24T13:45:37.249Z
Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 1971
File: src/utils/markdown_repair.py:31-36
Timestamp: 2026-06-24T13:45:37.249Z
Learning: In the lightspeed-stack repository, docstrings must use the section header name "Parameters:" (not "Args:") for function arguments, even if the project references Google Python docstring conventions. Ensure docstrings follow the project’s established "Parameters:" header format for any documented function parameters.
Applied to files:
tests/profiles/no_qv_keys/profile.pytests/profiles/empty_qv_strings/profile.pytests/unit/qv_config.pysrc/models/api/responses/successful/catalog.pytests/profiles/test/profile.pytests/unit/utils/test_pydantic_ai.pysrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pytests/unit/app/endpoints/test_shields.pytests/unit/test_configuration.pysrc/models/config.pytests/unit/utils/test_prompts.pytests/unit/models/responses/test_successful_responses.pytests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.py
📚 Learning: 2026-01-12T10:58:40.230Z
Learnt from: blublinsky
Repo: lightspeed-core/lightspeed-stack PR: 972
File: src/models/config.py:459-513
Timestamp: 2026-01-12T10:58:40.230Z
Learning: In lightspeed-core/lightspeed-stack, for Python files under src/models, when a user claims a fix is done but the issue persists, verify the current code state before accepting the fix. Steps: review the diff, fetch the latest changes, run relevant tests, reproduce the issue, search the codebase for lingering references to the original problem, confirm the fix is applied and not undone by subsequent commits, and validate with local checks to ensure the issue is resolved.
Applied to files:
src/models/api/responses/successful/catalog.pysrc/models/config.py
📚 Learning: 2026-02-25T07:46:33.545Z
Learnt from: asimurka
Repo: lightspeed-core/lightspeed-stack PR: 1211
File: src/models/responses.py:8-16
Timestamp: 2026-02-25T07:46:33.545Z
Learning: In the Python codebase, requests.py should use OpenAIResponseInputTool as Tool while responses.py uses OpenAIResponseTool as Tool. This difference is intentional due to differing schemas for input vs output tools in llama-stack-api. Apply this distinction consistently to other models under src/models (e.g., ensure request-related tools use the InputTool variant and response-related tools use the ResponseTool variant). If adding new tools, choose the corresponding InputTool or Tool class based on whether the tool represents input or output, and document the rationale in code comments.
Applied to files:
src/models/api/responses/successful/catalog.pysrc/models/config.py
📚 Learning: 2026-07-06T15:26:18.398Z
Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2071
File: src/models/config.py:2416-2422
Timestamp: 2026-07-06T15:26:18.398Z
Learning: In this repo’s Python code under src/**, don’t treat differences in string concatenation style as a style inconsistency when Black has effectively forced (or made clearer) use of explicit `+` string concatenation in multi-line logger/string expressions. If adjacent-literal implicit concatenation is avoided/changed specifically to accommodate Black’s formatting in these call sites, accept the `+` usage and don’t recommend converting it solely for consistency with nearby blocks that use implicit concatenation.
Applied to files:
src/models/api/responses/successful/catalog.pysrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pysrc/models/config.py
📚 Learning: 2026-07-17T19:25:05.325Z
Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 2166
File: src/utils/saved_prompts.py:129-157
Timestamp: 2026-07-17T19:25:05.325Z
Learning: For any endpoint that handles saved prompts and calls `src/utils/saved_prompts.py::create_saved_prompt`, treat the endpoint as the validation boundary. Before calling `create_saved_prompt`, validate the incoming saved-prompt name and content, specifically using `validate_saved_prompt_name` and then persist (store) the normalized value it returns. Do not call `create_saved_prompt` with unvalidated/raw name/content.
Applied to files:
src/models/api/responses/successful/catalog.pysrc/pydantic_ai_lightspeed/capabilities/question_validity/_capability.pysrc/models/config.py
📚 Learning: 2026-05-20T08:09:30.641Z
Learnt from: max-svistunov
Repo: lightspeed-core/lightspeed-stack PR: 1580
File: docs/design/llama-stack-config-merge/poc-results/library-mode/synthesized-run.yaml:107-110
Timestamp: 2026-05-20T08:09:30.641Z
Learning: In Llama-stack config YAMLs, when defining a Llama Guard safety shield entry, set `provider_shield_id` to the *guard model identifier* (e.g., `meta-llama/Llama-Guard-3-8B`). Do not use a chat/generative model id (e.g., `openai/gpt-4o-mini`): a chat-model id (or `native_override`) indicates only an override landed and does **not** mean the safety shield is actually gating queries. Ensure any E2E coverage for the related implementation (JIRA/E2E tests) exercises a real Llama Guard model to verify that the shield is effective.
Applied to files:
examples/lightspeed-stack-shields.yaml
🪛 markdownlint-cli2 (0.23.2)
docs/models/successful_responses.md
[warning] 576-576: Reference links and images should use a label that is defined
Missing link or image reference definition: "'validation'"
(MD052, reference-links-images)
[warning] 577-577: Reference links and images should use a label that is defined
Missing link or image reference definition: "'invalidresp'"
(MD052, reference-links-images)
🔇 Additional comments (20)
src/models/config.py (1)
1639-1686: LGTM!Also applies to: 1697-1713, 2724-2743, 3257-3293
tests/profiles/empty_qv_strings/profile.py (1)
1-9: LGTM!tests/profiles/no_qv_keys/profile.py (1)
1-7: LGTM!tests/unit/qv_config.py (1)
1-23: LGTM!tests/unit/test_configuration.py (1)
253-580: LGTM!tests/unit/utils/test_prompts.py (1)
338-354: LGTM!src/pydantic_ai_lightspeed/capabilities/question_validity/_capability.py (1)
119-155: LGTM!Also applies to: 173-173, 216-236, 238-254
tests/unit/pydantic_ai_lightspeed/capabilities/question_validity/test_capability.py (1)
13-25: LGTM!Also applies to: 111-121, 163-192, 203-245, 305-665, 693-768
tests/unit/utils/test_pydantic_ai.py (1)
25-25: LGTM!Also applies to: 77-77, 134-134, 300-300, 369-369
docs/devel_doc/openapi.json (3)
1443-1443: LGTM!Also applies to: 21275-21275
13876-13926: LGTM!
18826-18865: 🗄️ Data Integrity & IntegrationNo change needed for
GET /v1/shieldsnull fields.
GET /v1/shieldsserializes resolvedconfiguration.shields, andConfiguration.resolve_question_validity_shield_prompts()fills omittedQuestionValidityConfig.model_promptandinvalid_question_responsebefore exposure.docs/models/successful_responses.json (1)
1527-1545: LGTM!Also applies to: 4493-4504, 5860-5860
docs/models/successful_responses.md (1)
1956-1957: LGTM!docs/user_doc/config.md (1)
284-289: LGTM!Also applies to: 772-773
docs/user_doc/shields_guide.md (1)
84-137: LGTM!Also applies to: 150-152, 164-173
examples/lightspeed-stack-shields.yaml (1)
22-29: LGTM!src/models/api/responses/successful/catalog.py (1)
98-98: LGTM!tests/profiles/test/profile.py (1)
35-38: LGTM!tests/unit/models/responses/test_successful_responses.py (1)
196-196: LGTM!
Signed-off-by: Jordan Dubrick <jdubrick@redhat.com>
1 participant
Description
When a question_validity shield omits model_prompt or invalid_question_response, we now fill those at configuration load from the customization profile (then LCORE defaults). That way GET /v1/shields and the agent QV path see the effective text without copying the same prompts into YAML. This lets users include long model_prompts in the profile, rather than stuffing them inline in the yaml.
Resolution order:
Profile keys come from the profile Python module (PROFILE_CONFIG), not from YAML under customization:.
Example (omit the prompt fields and let the profile supply them):
With a profile that looks like:
After load, both fields are set to the profile values. GET /v1/shields returns those resolved strings.
Type of change
pyproject.toml+uv.lock]requirements.*.txtfor Konflux]Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit
New Features
${message}substitution for agent execution; responses moderation evaluates raw input.Documentation