LCORE-1801: Wrap lightspeed-stack startup with opentelemetry-instrument#2221
Conversation
WalkthroughStartup commands for the Lightspeed stack now run through ChangesRuntime instrumentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 7✅ Passed checks (7 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Makefile`:
- Line 23: Keep Python-version selection consistent across startup paths: update
Makefile lines 23-23 to use the selected interpreter or explicitly require
Python 3.12, and update the separate-process example at
docs/user_doc/deployment_guide.md lines 471-471 plus the library-mode example at
lines 564-564 to match the supported Python 3.12/3.13 behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c6892066-d7e9-4812-92fc-436e956f60a2
📒 Files selected for processing (5)
MakefileREADME.mddeploy/lightspeed-stack/Containerfiledocs/user_doc/deployment_guide.mdtests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
📜 Review details
⏰ Context from checks skipped due to timeout. (16)
- GitHub Check: build-pr
- GitHub Check: spectral
- GitHub Check: mypy
- GitHub Check: Pylinter
- GitHub Check: unit_tests (3.12)
- GitHub Check: unit_tests (3.13)
- GitHub Check: integration_tests (3.13)
- GitHub Check: integration_tests (3.12)
- GitHub Check: E2E: server mode / ci / group 2
- GitHub Check: E2E: server mode / ci / group 1
- GitHub Check: E2E: library mode / ci / group 2
- GitHub Check: E2E: library mode / ci / group 3
- GitHub Check: E2E: library mode / ci / group 1
- GitHub Check: E2E: server mode / ci / group 3
- GitHub Check: E2E Tests for Lightspeed Evaluation job
- GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-0-7-on-pull-request
🧰 Additional context used
📓 Path-based instructions (1)
**/*
📄 CodeRabbit inference engine (Custom checks)
**/*: Flag meaningful O(n^2)+ algorithms on non-trivial inputs, including handlers and Kubernetes list operations.
Flag N+1 patterns that list items and then query once per item, including Kubernetes API and database access.
Flag expensive work inside loops, including API calls, JSON parsing, and regex compilation.
Flag unbounded growth in caches, watchers, or buffers when eviction or limits are missing.
Flag missing pagination or limits on list operations and API endpoints.
Flag secrets or tokens logged in plaintext or hardcoded in source.
Flag API endpoints missing authentication or authorization.
Flag injection vulnerabilities, including SQL injection, command injection, and path traversal.
Flag sensitive data leaked in API responses, WebSocket messages, or logs.
Flag Kubernetes Secrets and Red Hat secrets missing OwnerReferences.Never commit secrets or keys; use environment variables for sensitive data.
Files:
deploy/lightspeed-stack/Containerfiletests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yamlMakefileREADME.mddocs/user_doc/deployment_guide.md
🧠 Learnings (2)
📚 Learning: 2026-02-19T10:06:50.647Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 1181
File: tests/e2e-prow/rhoai/manifests/lightspeed/mock-jwks.yaml:32-34
Timestamp: 2026-02-19T10:06:50.647Z
Learning: In the rhoai tests under tests/e2e-prow/rhoai/manifests, avoid static ConfigMap definitions for mock-jwks-script and mcp-mock-server-script since these ConfigMaps are created dynamically by the pipeline.sh deployment script using 'oc create configmap'. Ensure there are no static ConfigMap resources for these names in the manifests. If such ConfigMaps are added in the future, coordinate with the pipeline to reflect dynamic creation or adjust tests to rely on the dynamic provisioning.
Applied to files:
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
📚 Learning: 2026-05-20T08:09:30.641Z
Learnt from: max-svistunov
Repo: lightspeed-core/lightspeed-stack PR: 1580
File: docs/design/llama-stack-config-merge/poc-results/library-mode/synthesized-run.yaml:107-110
Timestamp: 2026-05-20T08:09:30.641Z
Learning: In Llama-stack config YAMLs, when defining a Llama Guard safety shield entry, set `provider_shield_id` to the *guard model identifier* (e.g., `meta-llama/Llama-Guard-3-8B`). Do not use a chat/generative model id (e.g., `openai/gpt-4o-mini`): a chat-model id (or `native_override`) indicates only an override landed and does **not** mean the safety shield is actually gating queries. Ensure any E2E coverage for the related implementation (JIRA/E2E tests) exercises a real Llama Guard model to verify that the shield is effective.
Applied to files:
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
🔇 Additional comments (3)
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml (1)
40-40: LGTM!README.md (1)
1133-1133: LGTM!deploy/lightspeed-stack/Containerfile (1)
138-138: 🩺 Stability & AvailabilityNo change needed for
opentelemetry-instrument.
opentelemetry-distrois a production dependency and provides theopentelemetry-instrumentexecutable used by the entrypoint.
60f6cf3 to
541639a
Compare
Update all LCORE startup paths to use `opentelemetry-instrument` as the process wrapper to enable automatic OpenTelemetry instrumentation of the FastAPI application. Signed-off-by: Anik Bhattacharjee <anbhatta@redhat.com>
541639a to
c58774f
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/e2e/utils/prow_utils.py`:
- Around line 65-70: Complete the docstring for wait_for_pod_health by adding a
Parameters: section documenting pod_name and max_attempts, plus a Returns:
section stating that the function returns None. Follow the repository’s existing
docstring convention and preserve the current behavior.
In `@tests/e2e/utils/utils.py`:
- Around line 195-196: Correct the timeout documentation near the retry
configuration in the test utilities to reflect the actual worst-case duration
from 20 attempts, 5-second docker inspect timeouts, and 2-second sleeps,
approximately 138 seconds; alternatively, reduce the per-attempt timeout so the
documented 40-second budget is accurate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 22b646a4-df57-4b2a-ab1d-968b0fd104f0
📒 Files selected for processing (7)
MakefileREADME.mddeploy/lightspeed-stack/Containerfiledocs/user_doc/deployment_guide.mdtests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yamltests/e2e/utils/prow_utils.pytests/e2e/utils/utils.py
📜 Review details
⏰ Context from checks skipped due to timeout. (16)
- GitHub Check: integration_tests (3.12)
- GitHub Check: integration_tests (3.13)
- GitHub Check: spectral
- GitHub Check: Pylinter
- GitHub Check: mypy
- GitHub Check: build-pr
- GitHub Check: unit_tests (3.12)
- GitHub Check: unit_tests (3.13)
- GitHub Check: E2E Tests for Lightspeed Evaluation job
- GitHub Check: E2E: server mode / ci / group 2
- GitHub Check: E2E: server mode / ci / group 3
- GitHub Check: E2E: library mode / ci / group 1
- GitHub Check: E2E: library mode / ci / group 3
- GitHub Check: E2E: library mode / ci / group 2
- GitHub Check: E2E: server mode / ci / group 1
- GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-0-7-on-pull-request
🧰 Additional context used
📓 Path-based instructions (3)
**/*
📄 CodeRabbit inference engine (Custom checks)
**/*: Flag meaningful O(n^2)+ algorithms on non-trivial inputs, including handlers and Kubernetes list operations.
Flag N+1 patterns that list items and then query once per item, including Kubernetes API and database access.
Flag expensive work inside loops, including API calls, JSON parsing, and regex compilation.
Flag unbounded growth in caches, watchers, or buffers when eviction or limits are missing.
Flag missing pagination or limits on list operations and API endpoints.
Flag secrets or tokens logged in plaintext or hardcoded in source.
Flag API endpoints missing authentication or authorization.
Flag injection vulnerabilities, including SQL injection, command injection, and path traversal.
Flag sensitive data leaked in API responses, WebSocket messages, or logs.
Flag Kubernetes Secrets and Red Hat secrets missing OwnerReferences.
Files:
deploy/lightspeed-stack/Containerfiletests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yamlREADME.mddocs/user_doc/deployment_guide.mdMakefiletests/e2e/utils/prow_utils.pytests/e2e/utils/utils.py
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use absolute imports for internal Python modules.
All modules must begin with descriptive docstrings explaining their purpose.
Uselogger = get_logger(__name__)fromlog.pyfor module logging.
Use complete type annotations for function parameters and return types.
Use modern union syntax such asstr | int; useOptional[Type]for optional values.
Usetyping_extensions.Selffor model validators.
Functions must use descriptive, action-oriented snake_case names such asget_,validate_, andcheck_.
Avoid modifying mutable parameters in place; return a new data structure instead.
Useasync deffor I/O operations and external API calls.
HandleAPIConnectionErrorfrom Llama Stack.
All classes must have descriptive docstrings and complete type annotations for class attributes; use specific types instead ofAny.
Use PascalCase for classes and descriptive standard suffixes such asConfiguration,Error/Exception,Resolver, andInterface.
Use ABC and@abstractmethodfor abstract interfaces.
Follow Google Python docstring conventions; document all modules, classes, and functions, includingParameters,Returns,Raises, andAttributessections as applicable.
Never commit secrets or keys; use environment variables for sensitive data.
Files:
tests/e2e/utils/prow_utils.pytests/e2e/utils/utils.py
tests/e2e/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Use behave with Gherkin feature files for end-to-end tests.
Files:
tests/e2e/utils/prow_utils.pytests/e2e/utils/utils.py
🧠 Learnings (3)
📚 Learning: 2026-02-19T10:06:50.647Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 1181
File: tests/e2e-prow/rhoai/manifests/lightspeed/mock-jwks.yaml:32-34
Timestamp: 2026-02-19T10:06:50.647Z
Learning: In the rhoai tests under tests/e2e-prow/rhoai/manifests, avoid static ConfigMap definitions for mock-jwks-script and mcp-mock-server-script since these ConfigMaps are created dynamically by the pipeline.sh deployment script using 'oc create configmap'. Ensure there are no static ConfigMap resources for these names in the manifests. If such ConfigMaps are added in the future, coordinate with the pipeline to reflect dynamic creation or adjust tests to rely on the dynamic provisioning.
Applied to files:
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
📚 Learning: 2026-05-20T08:09:30.641Z
Learnt from: max-svistunov
Repo: lightspeed-core/lightspeed-stack PR: 1580
File: docs/design/llama-stack-config-merge/poc-results/library-mode/synthesized-run.yaml:107-110
Timestamp: 2026-05-20T08:09:30.641Z
Learning: In Llama-stack config YAMLs, when defining a Llama Guard safety shield entry, set `provider_shield_id` to the *guard model identifier* (e.g., `meta-llama/Llama-Guard-3-8B`). Do not use a chat/generative model id (e.g., `openai/gpt-4o-mini`): a chat-model id (or `native_override`) indicates only an override landed and does **not** mean the safety shield is actually gating queries. Ensure any E2E coverage for the related implementation (JIRA/E2E tests) exercises a real Llama Guard model to verify that the shield is effective.
Applied to files:
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
📚 Learning: 2026-06-24T13:45:37.249Z
Learnt from: Jdubrick
Repo: lightspeed-core/lightspeed-stack PR: 1971
File: src/utils/markdown_repair.py:31-36
Timestamp: 2026-06-24T13:45:37.249Z
Learning: In the lightspeed-stack repository, docstrings must use the section header name "Parameters:" (not "Args:") for function arguments, even if the project references Google Python docstring conventions. Ensure docstrings follow the project’s established "Parameters:" header format for any documented function parameters.
Applied to files:
tests/e2e/utils/prow_utils.pytests/e2e/utils/utils.py
🔇 Additional comments (3)
tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml (1)
40-40: LGTM!README.md (1)
1133-1133: LGTM!deploy/lightspeed-stack/Containerfile (1)
138-138: 🩺 Stability & AvailabilityRuntime dependency already provides the OpenTelemetry console script.
opentelemetry-distrois declared and locked, and it provides theopentelemetry-instrumententrypoint used here.
| def wait_for_pod_health(pod_name: str, max_attempts: int = 20) -> None: | ||
| """Wait for pod to be ready in OpenShift/Prow environment. | ||
|
|
||
| Generous number of attempts to account for OpenTelemetry instrumentation | ||
| initialization overhead during service startup. | ||
| """ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Complete the updated function docstring.
Document pod_name, max_attempts, and the None return value using the repository’s Parameters: convention.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/e2e/utils/prow_utils.py` around lines 65 - 70, Complete the docstring
for wait_for_pod_health by adding a Parameters: section documenting pod_name and
max_attempts, plus a Returns: section stating that the function returns None.
Follow the repository’s existing docstring convention and preserve the current
behavior.
Sources: Coding guidelines, Learnings
| OpenTelemetry instrumentation adds initialization overhead, so the default | ||
| has been set to 20 attempts (40 seconds) to prevent timeouts. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Correct the documented timeout budget.
With 20 attempts, 5-second docker inspect timeouts, and 2-second sleeps, the worst-case wait is about 138 seconds—not 40 seconds. Document the actual upper bound or reduce the per-attempt timeout.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/e2e/utils/utils.py` around lines 195 - 196, Correct the timeout
documentation near the retry configuration in the test utilities to reflect the
actual worst-case duration from 20 attempts, 5-second docker inspect timeouts,
and 2-second sleeps, approximately 138 seconds; alternatively, reduce the
per-attempt timeout so the documented 40-second budget is accurate.
3 participants
Description
Update all LCORE startup paths to use
opentelemetry-instrumentas the process wrapper to enable automatic OpenTelemetry instrumentation of the FastAPI application.Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit