feat: add Prometheus & Grafana Monitoring for LLMs Using OpenTelemetry #427
Conversation
akotyla
force-pushed
the
414-feat-add-prometheus-grafana-monitoring-for-llms-using-opentelemetry
branch
from
bb1433f to
db9da5f
Compare
|
Code Coverage SummaryDiff against mainResults for commit: 6749b76 Minimum allowed coverage is ♻️ This comment has been updated with latest results |
|
Trivy scanning results. Report Summary ┌─────────┬──────┬─────────────────┬─────────┐
For OSS Maintainers: VEX NoticeIf you're an OSS maintainer and Trivy has detected vulnerabilities in your project that you believe are not actually exploitable, consider issuing a VEX (Vulnerability Exploitability eXchange) statement. To disable this notice, set the TRIVY_DISABLE_VEX_NOTICE environment variable. uv.lock (uv)Total: 21 (MEDIUM: 11, HIGH: 9, CRITICAL: 1) ┌──────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐ |
akotyla
force-pushed
the
414-feat-add-prometheus-grafana-monitoring-for-llms-using-opentelemetry
branch
from
6e7a967 to
12eb8ad
Compare
| ), | ||
| } | ||
|
|
||
| def record(self, metric_name: str, value: float, attributes: dict | None = None) -> None: |
There was a problem hiding this comment.
I think we should have some enum for metrics, passing string directly is error-prone.
|
|
||
| Handler = str | TraceHandler | ||
|
|
||
| P = ParamSpec("P") | ||
| R = TypeVar("R") | ||
|
|
||
|
|
||
| def set_metric_handler(meter: Meter) -> None: |
There was a problem hiding this comment.
It should be configurable to add some sort of prefix to all metrics. For example input_tokens -> ragbits_input_tokens
|
|
||
| 1. Open your browser and navigate to http://localhost:3000. | ||
| 2. Use the default credentials to log in (username: `admin`, password: `admin`). | ||
| 3. Once logged in, you can either navigate to the Explore section to query and visualize metrics directly or create a new dashboard to build custom visualizations tailored to your needs. |
There was a problem hiding this comment.
I think we should provide some prebuilt dashboard :D
There was a problem hiding this comment.
Pull Request Overview
This PR adds Prometheus and Grafana monitoring for LLM interactions using OpenTelemetry.
- Updates import paths in audit trace and CLI modules
- Adds metric recording in LLM call functions and introduces a new metric handler using OpenTelemetry
- Provides accompanying documentation and CHANGELOG updates
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/ragbits-core/tests/unit/audit/test_trace.py | Updated import path for trace handlers |
| packages/ragbits-core/tests/unit/audit/test_cli.py | Updated import path for CLI trace handlers |
| packages/ragbits-core/src/ragbits/core/llms/local.py | Instrumented LLM calls with metric recording |
| packages/ragbits-core/src/ragbits/core/llms/litellm.py | Enhanced metric recording in LLM calls and streaming; duplicate error check observed |
| packages/ragbits-core/src/ragbits/core/audit/trace_handlers/otel.py | Updated import path for trace handlers |
| packages/ragbits-core/src/ragbits/core/audit/trace_handlers/cli.py | Updated import path for CLI trace handlers |
| packages/ragbits-core/src/ragbits/core/audit/otel_metric_handler.py | New module for handling OTEL metrics |
| packages/ragbits-core/src/ragbits/core/audit/init.py | Added global metric handler setup and record_metric interface |
| packages/ragbits-core/CHANGELOG.md | Documented monitoring feature addition |
| mkdocs.yml | Updated navigation to include metrics documentation |
| docs/how-to/project/use_metrics.md | Added guide for setting up metrics with Ragbits |
| if not response.choices: # type: ignore | ||
| raise LLMEmptyResponseError() | ||
|
|
There was a problem hiding this comment.
The check for response.choices is repeated after already being checked earlier. Consider removing the duplicate check to avoid redundant error handling.
| if not response.choices: # type: ignore | |
| raise LLMEmptyResponseError() |
Copilot is powered by AI, so mistakes are possible. Review output carefully before use.
| Each metric is tagged with the following **labels** (attributes) for better filtering and analysis: the class name of the prompt being processed (`promp`) and the name of the LLM being used (`model`). | ||
|
|
There was a problem hiding this comment.
There is a typo in the label name 'promp'. It should likely be corrected to 'prompt' for clarity.
| Each metric is tagged with the following **labels** (attributes) for better filtering and analysis: the class name of the prompt being processed (`promp`) and the name of the LLM being used (`model`). | |
| Each metric is tagged with the following **labels** (attributes) for better filtering and analysis: the class name of the prompt being processed (`prompt`) and the name of the LLM being used (`model`). |
Copilot is powered by AI, so mistakes are possible. Review output carefully before use.
No description provided.