docs(monitor): update the monitor doc #106
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughNavigation reorganized: Monitoring group expanded with two new pages and Datasets removed; a new "Prompt Management" group was added. Two new Monitoring docs were introduced and the Monitoring introduction was reworked. Trace image assets were moved into Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant UI as Monitors UI
participant EvalLib as Evaluator Library
participant Config as Monitor Config
participant Dashboard as Monitor Dashboard
Note over UI,EvalLib #dfefff: Documentation aligns with this control flow
User->>UI: Open Monitors > New
UI->>EvalLib: Open Evaluator Library
EvalLib-->>UI: Select evaluator (LLM or built-in)
UI->>Config: Define Span Filters & Map Input
Config-->>UI: Live preview (JSON/regex extraction)
UI->>Config: Save Monitor
Config-->>Dashboard: Register monitor (real-time)
User->>Dashboard: View results, charts, matching spans
Dashboard-->>User: Links to trace explorer
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
nina-kollman
changed the title
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Changes requested ❌
Reviewed everything up to 5bbd4ed in 2 minutes and 24 seconds. Click for details.
- Reviewed
282lines of code in6files - Skipped
20files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. monitoring/defining-monitors.mdx:93
- Draft comment:
Add a newline at the end of the file. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =20%<= threshold50%The comment is asking for a newline at the end of the file, which is a common style guideline in many coding standards. However, it is purely informative and does not provide a specific code suggestion or address a potential issue with the code's functionality. Therefore, it violates the rule against making purely informative comments.
2. monitoring/defining-monitors.mdx:35
- Draft comment:
Typo: 'Eviornment' should be corrected to 'Environment'. - Reason this comment was not posted:
Marked as duplicate.
3. monitoring/introduction.mdx:23
- Draft comment:
It looks like there are a couple of typos in the QA Relevancy section. Consider revising "Asses the relevant" to "Assesses the relevance" to improve clarity. - Reason this comment was not posted:
Comment was on unchanged code.
4. monitoring/using-monitors.mdx:28
- Draft comment:
Typo: 'Evalutor Output Rates' should be 'Evaluator Output Rates'. - Reason this comment was not posted:
Marked as duplicate.
Workflow ID: wflow_Wfr2m8gv74k5tgOu
You can customize 
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (9)
openllmetry/introduction.mdx (1)
6-7: Add descriptive alt text to trace imagesThe image files exist at img/trace/trace-light.png and img/trace/trace-dark.png; include descriptive alt attributes for accessibility and SEO:
- <img className="block dark:hidden" src="/img/trace/trace-light.png" /> - <img className="hidden dark:block" src="/img/trace/trace-dark.png" /> + <img className="block dark:hidden" src="/img/trace/trace-light.png" alt="Trace view (light)" /> + <img className="hidden dark:block" src="/img/trace/trace-dark.png" alt="Trace view (dark)" />openllmetry/integrations/traceloop.mdx (1)
7-8: Image path updates are consistent with the new /img/trace/ structure; add alt text.Small accessibility improvement.
- <img className="block dark:hidden" src="/img/trace/trace-light.png" /> - <img className="hidden dark:block" src="/img/trace/trace-dark.png" /> + <img className="block dark:hidden" src="/img/trace/trace-light.png" alt="Trace view (light)" /> + <img className="hidden dark:block" src="/img/trace/trace-dark.png" alt="Trace view (dark)" />monitoring/using-monitors.mdx (2)
56-56: Style nit: example values should be lowercase for consistency.-- **Bucket Size**: 6h, Hourly, Daily, etc. +- **Bucket Size**: 6h, hourly, daily, etc.
70-73: Image references OK; suggest adding alt text.- <img className="block dark:hidden" src="/img/trace/trace-light.png" /> - <img className="hidden dark:block" src="/img/trace/trace-dark.png" /> + <img className="block dark:hidden" src="/img/trace/trace-light.png" alt="Trace details (light)" /> + <img className="hidden dark:block" src="/img/trace/trace-dark.png" alt="Trace details (dark)" />monitoring/introduction.mdx (2)
12-13: Images moved to /img/monitor/... — good; consider alt text.- <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" /> - <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" alt="Monitor list (light)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" alt="Monitor list (dark)" />
30-30: Hyphenate “Real-time” in heading (adjectival use).-## Useful Real Time Monitors +## Useful Real-time Monitorsmonitoring/defining-monitors.mdx (3)
13-14: Ensure consistent punctuation in numbered list.Items 2–3 lack terminal periods; item 1 has one.
-2. **Span Filter**: Criteria that determine which spans the monitor will evaluate -3. **Configuration**: Settings for how the monitor operates +2. **Span Filter**: Criteria that determine which spans the monitor will evaluate. +3. **Configuration**: Settings for how the monitor operates.
19-19: Tighten wording.Avoid future tense; keep imperative voice.
-Next, you will be able to configure which spans will be monitored. +Then configure which spans to monitor.
68-68: Clarify the example’s target element.Be explicit that we’re extracting from the first array element.
-For example, if your content is an array and you want to extract the "text" field from the object: +For example, if your content is an array and you want to extract the text field from the first object:
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (20)
img/monitor-dark.pngis excluded by!**/*.pngimg/monitor-light.pngis excluded by!**/*.pngimg/monitor/monitor-filter-dark.pngis excluded by!**/*.pngimg/monitor/monitor-filter-light.pngis excluded by!**/*.pngimg/monitor/monitor-filter-options-dark.pngis excluded by!**/*.pngimg/monitor/monitor-filter-options-light.pngis excluded by!**/*.pngimg/monitor/monitor-json-dark.pngis excluded by!**/*.pngimg/monitor/monitor-json-light.pngis excluded by!**/*.pngimg/monitor/monitor-list-dark.pngis excluded by!**/*.pngimg/monitor/monitor-list-light.pngis excluded by!**/*.pngimg/monitor/monitor-page-buckets-dark.pngis excluded by!**/*.pngimg/monitor/monitor-page-buckets-light.pngis excluded by!**/*.pngimg/monitor/monitor-page-line-dark.pngis excluded by!**/*.pngimg/monitor/monitor-page-line-light.pngis excluded by!**/*.pngimg/monitor/monitor-regex-dark.pngis excluded by!**/*.pngimg/monitor/monitor-regex-light.pngis excluded by!**/*.pngimg/monitor/monitor-settings-dark.pngis excluded by!**/*.pngimg/monitor/monitor-settings-light.pngis excluded by!**/*.pngimg/trace/trace-dark.pngis excluded by!**/*.pngimg/trace/trace-light.pngis excluded by!**/*.png
📒 Files selected for processing (6)
mint.json(1 hunks)monitoring/defining-monitors.mdx(1 hunks)monitoring/introduction.mdx(2 hunks)monitoring/using-monitors.mdx(1 hunks)openllmetry/integrations/traceloop.mdx(1 hunks)openllmetry/introduction.mdx(1 hunks)
🧰 Additional context used
🪛 LanguageTool
monitoring/using-monitors.mdx
[grammar] ~25-~25: There might be a mistake here.
Context: ... value that was evaluated and its result - Trend Analysis: Performance over time ...
(QB_NEW_EN)
[grammar] ~26-~26: There might be a mistake here.
Context: ...Trend Analysis*: Performance over time - Volume Metrics: Number of evaluations ...
(QB_NEW_EN)
[grammar] ~27-~27: Ensure spelling is correct
Context: ...trics**: Number of evaluations performed - Evalutor Output Rates: Such as success rates f...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~54-~54: There might be a mistake here.
Context: ... Range**: 24h, 7d, 14d, or custom ranges - Metric: Select which evaluator output ...
(QB_NEW_EN)
[grammar] ~56-~56: There might be a mistake here.
Context: ...Bucket Size: 6h, Hourly, Daily, etc. - Aggregation: Choose average, median, s...
(QB_NEW_EN)
[grammar] ~61-~61: There might be a mistake here.
Context: ... matched your monitor's filter criteria: - Timestamp: When the evaluation occurre...
(QB_NEW_EN)
[grammar] ~62-~62: There might be a mistake here.
Context: ...imestamp**: When the evaluation occurred - Input: The actual content that was was...
(QB_NEW_EN)
[grammar] ~63-~63: There might be a mistake here.
Context: ... Input: The actual content that was was mapped to be evaluated - Output: Th...
(QB_NEW_EN)
[grammar] ~63-~63: There might be a mistake here.
Context: ...tent that was was mapped to be evaluated - Output: The evaluation result/score - ...
(QB_NEW_EN)
[grammar] ~64-~64: There might be a mistake here.
Context: ... Output: The evaluation result/score - Completed Runs: Total successful/error...
(QB_NEW_EN)
[grammar] ~65-~65: There might be a mistake here.
Context: ...ns**: Total successful/error evaluations - Error Runs: Failed evaluation attempts...
(QB_NEW_EN)
monitoring/defining-monitors.mdx
[grammar] ~13-~13: There might be a mistake here.
Context: ...ne which spans the monitor will evaluate 3. Configuration: Settings for how the mo...
(QB_NEW_EN)
[grammar] ~18-~18: There might be a mistake here.
Context: ...aluator you want to run in your monitor. Next, you will be able to configure whic...
(QB_NEW_EN)
[grammar] ~42-~42: There might be a mistake here.
Context: ...es <img className="block dark:hidden" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~44-~44: There might be a mistake here.
Context: ...ight.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~45-~45: There might be a mistake here.
Context: ...t.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~46-~46: There might be a mistake here.
Context: ...} /> <img className="hidden dark:block" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~48-~48: There might be a mistake here.
Context: ...ark.png" style={{maxWidth: '500px'}} /> ## Monitor Settings ### Map Input ...
(QB_NEW_EN)
[grammar] ~92-~92: There might be a mistake here.
Context: ...egex-dark.png" /> ### Advanced You can set a Rate sample to control...
(QB_NEW_EN)
monitoring/introduction.mdx
[grammar] ~18-~18: There might be a mistake here.
Context: ...e evaluator and log the monitor result. This allows you to continuously assess t...
(QB_NEW_EN)
[style] ~34-~34: ‘with respect to’ might be wordy. Consider a shorter alternative.
Context: ...xt is within a given range (constant or with respect to an input). - Word Count: Checks if ...
(EN_WORDINESS_PREMIUM_WITH_RESPECT_TO)
[grammar] ~34-~34: There might be a mistake here.
Context: ... (constant or with respect to an input). - Word Count: Checks if the number of wo...
(QB_NEW_EN)
🔇 Additional comments (3)
mint.json (1)
148-149: Navigation: Monitoring pages verified
monitoring/introduction.mdx, monitoring/defining-monitors.mdx, and monitoring/using-monitors.mdx all exist.monitoring/defining-monitors.mdx (2)
12-12: Link and image assets verified
/evaluators/introexists (evaluators/intro.mdx) and all referencedmonitor-*.pngimages are present underimg/monitor/.
93-93: Update terminology and grammar
Replace the heading and sentence to read “Sample rate” and tighten the phrasing to match common usage. Confirm whether the in-product UI uses “Rate sample” or “Sample rate” and align the docs accordingly.-You can set a **Rate sample** to control the percentage of spans, within the selected filter group apply to, that the monitor will run on. +Set the **Sample rate** to control the percentage of spans within the selected filter group that the monitor evaluates.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 9ddf7d1 in 43 seconds. Click for details.
- Reviewed
34lines of code in2files - Skipped
0files when reviewing. - Skipped posting
3draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. monitoring/defining-monitors.mdx:92
- Draft comment:
Improved sentence clarity by removing unnecessary comma. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =0%<= threshold50%This comment is purely informative and does not provide any actionable feedback or suggestions for improvement. It simply states what was done without offering any guidance or questions for the PR author.
2. monitoring/using-monitors.mdx:28
- Draft comment:
Fixed typo: changed 'Evalutor' to 'Evaluator'. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =0%<= threshold50%This comment is purely informative as it only states that a typo was fixed. It does not provide any actionable feedback or suggestions for improvement.
3. monitoring/using-monitors.mdx:62
- Draft comment:
Removed duplicate 'was' for grammatical correctness. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =0%<= threshold50%This comment is purely informative as it only points out a grammatical correction without suggesting any code changes or improvements. It doesn't ask for any specific action or confirmation from the PR author.
Workflow ID: wflow_SWOgy9X52UtKNfA7
You can customize
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 319e312 in 47 seconds. Click for details.
- Reviewed
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. monitoring/defining-monitors.mdx:35
- Draft comment:
Fix typo: change 'Eviornment' to 'Environment'. - Reason this comment was not posted:
Comment looked like it was already resolved.
Workflow ID: wflow_TcM8xuipQXS0WB1j
You can customize
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (9)
monitoring/using-monitors.mdx (2)
25-28: Standardize list casing and tighten phrasing; fix “Evaluator Output Rates” style.Align to sentence case and remove “Such as” capitalization.
-- **Run Details**: The span value that was evaluated and its result -- **Trend Analysis**: Performance over time -- **Volume Metrics**: Number of evaluations performed -- **Evaluator Output Rates**: Such as success rates for threshold-based evaluators +- **Run details**: the evaluated span value and its result +- **Trend analysis**: performance over time +- **Volume metrics**: number of evaluations performed +- **Evaluator output rates**: such as success rates for threshold-based evaluators
61-67: Clarify run-count labels and simplify wording.Avoid ambiguity between “completed” vs “error” runs and tighten phrasing.
-- **Input**: The actual content that was mapped to be evaluated -- **Output**: The evaluation result/score -- **Completed Runs**: Total successful/error evaluations -- **Error Runs**: Failed evaluation attempts +- **Input**: the content that was mapped to be evaluated +- **Output**: the evaluation result or score +- **Completed runs**: number of successful evaluations +- **Error runs**: number of failed evaluation attemptsmonitoring/defining-monitors.mdx (7)
83-83: Use a safer, simpler regex and fix “real time” phrasing.-You can use Regex like `text":"(.+?)"` to extract just the text content. The regex will be applied to the Preview table, allowing you to see the extracted result in real-time. +Use a regex like `text":"([^"]+)"` to extract just the text content. The regex is applied to the Preview table, allowing you to see the extracted result in real time.
93-93: Use standard term “sampling rate”.-You can set a **Rate sample** to control the percentage of spans within the selected filter group that the monitor will run on. +You can set a **sampling rate** to control the percentage of spans within the selected filter group that the monitor will run on.
26-30: Add alt text to images in Span Filtering section.A11y/SEO; mirror descriptions across light/dark.
- <img className="block dark:hidden" - src="/img/monitor/monitor-filter-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-filter-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-filter-light.png" alt="Span filtering modal with filters list (light theme)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-filter-dark.png" alt="Span filtering modal with filters list (dark theme)" />
42-49: Add alt text to filter options images.-<img className="block dark:hidden" - src="/img/monitor/monitor-filter-options-light.png" - style={{maxWidth: '500px'}} -/> -<img className="hidden dark:block" - src="/img/monitor/monitor-filter-options-dark.png" - style={{maxWidth: '500px'}} -/> +<img className="block dark:hidden" src="/img/monitor/monitor-filter-options-light.png" style={{maxWidth: '500px'}} alt="Filter options: Environment, Workflow Name, Service Name, AI Data, Attributes (light theme)" /> +<img className="hidden dark:block" src="/img/monitor/monitor-filter-options-dark.png" style={{maxWidth: '500px'}} alt="Filter options: Environment, Workflow Name, Service Name, AI Data, Attributes (dark theme)" />
60-64: Add alt text to monitor settings images.- <img className="block dark:hidden" - src="/img/monitor/monitor-settings-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-settings-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-settings-light.png" alt="Monitor settings with input mapping preview (light theme)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-settings-dark.png" alt="Monitor settings with input mapping preview (dark theme)" />
77-81: Add alt text to JSON mapping images.- <img className="block dark:hidden" - src="/img/monitor/monitor-json-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-json-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-json-light.png" alt="JSON key mapping preview extracting [0].text (light theme)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-json-dark.png" alt="JSON key mapping preview extracting [0].text (dark theme)" />
86-90: Add alt text to regex images.- <img className="block dark:hidden" - src="/img/monitor/monitor-regex-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-regex-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-regex-light.png" alt="Regex extraction preview on text value (light theme)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-regex-dark.png" alt="Regex extraction preview on text value (dark theme)" />
🧹 Nitpick comments (4)
monitoring/using-monitors.mdx (2)
54-57: Make control list items consistent (sentence case; lower-case examples).-- **Time Range**: 24h, 7d, 14d, or custom ranges -- **Metric**: Select which evaluator output property to measure -- **Bucket Size**: 6h, Hourly, Daily, etc. -- **Aggregation**: Choose average, median, sum, min, max, or count +- **Time range**: 24h, 7d, 14d, or custom ranges +- **Metric**: select the evaluator output property to measure +- **Bucket size**: 6h, hourly, daily, etc. +- **Aggregation**: average, median, sum, min, max, or count
75-75: Comma splice fix after “tracing”.-For further information on tracing refer to [OpenLLMetry](/openllmetry/introduction). +For further information on tracing, refer to [OpenLLMetry](/openllmetry/introduction).monitoring/defining-monitors.mdx (2)
74-76: Use “real time” (noun) consistently.-You can use JSON key mapping like `[0].text` to extract just the text content. The JSON key mapping will be applied to the Preview table, allowing you to see the extracted result in real-time. +You can use JSON key mapping like `[0].text` to extract just the text content. The JSON key mapping is applied to the Preview table, allowing you to see the extracted result in real time.
12-15: End punctuation in numbered list for consistency.1. **Evaluator**: The evaluation logic that will run on matching spans. You can define your own custom evaluators or use the pre-built ones by Traceloop. See [Evaluators](/evaluators/intro) for more details. -2. **Span Filter**: Criteria that determine which spans the monitor will evaluate -3. **Configuration**: Settings for how the monitor operates +2. **Span Filter**: Criteria that determine which spans the monitor will evaluate. +3. **Configuration**: Settings for how the monitor operates.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
monitoring/defining-monitors.mdx(1 hunks)monitoring/using-monitors.mdx(1 hunks)
🧰 Additional context used
🪛 LanguageTool
monitoring/defining-monitors.mdx
[grammar] ~13-~13: There might be a mistake here.
Context: ...ne which spans the monitor will evaluate 3. Configuration: Settings for how the mo...
(QB_NEW_EN)
[grammar] ~18-~18: There might be a mistake here.
Context: ...aluator you want to run in your monitor. Next, you will be able to configure whic...
(QB_NEW_EN)
[grammar] ~42-~42: There might be a mistake here.
Context: ...es <img className="block dark:hidden" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~43-~43: There might be a mistake here.
Context: ...nitor/monitor-filter-options-light.png" style={{maxWidth: '500px'}} /> <img clas...
(QB_NEW_EN)
[grammar] ~44-~44: There might be a mistake here.
Context: ...ight.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~45-~45: There might be a mistake here.
Context: ...t.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~46-~46: There might be a mistake here.
Context: ...} /> <img className="hidden dark:block" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~47-~47: There might be a mistake here.
Context: ...monitor/monitor-filter-options-dark.png" style={{maxWidth: '500px'}} /> ## Moni...
(QB_NEW_EN)
[grammar] ~48-~48: There might be a mistake here.
Context: ...ark.png" style={{maxWidth: '500px'}} /> ## Monitor Settings ### Map Input ...
(QB_NEW_EN)
[grammar] ~92-~92: There might be a mistake here.
Context: ...egex-dark.png" /> ### Advanced You can set a Rate sample to control...
(QB_NEW_EN)
monitoring/using-monitors.mdx
[grammar] ~25-~25: There might be a mistake here.
Context: ... value that was evaluated and its result - Trend Analysis: Performance over time ...
(QB_NEW_EN)
[grammar] ~26-~26: There might be a mistake here.
Context: ...Trend Analysis*: Performance over time - Volume Metrics: Number of evaluations ...
(QB_NEW_EN)
[grammar] ~27-~27: There might be a mistake here.
Context: ...trics**: Number of evaluations performed - Evaluator Output Rates: Such as succes...
(QB_NEW_EN)
[grammar] ~54-~54: There might be a mistake here.
Context: ... Range**: 24h, 7d, 14d, or custom ranges - Metric: Select which evaluator output ...
(QB_NEW_EN)
[grammar] ~56-~56: There might be a mistake here.
Context: ...Bucket Size: 6h, Hourly, Daily, etc. - Aggregation: Choose average, median, s...
(QB_NEW_EN)
[grammar] ~61-~61: There might be a mistake here.
Context: ... matched your monitor's filter criteria: - Timestamp: When the evaluation occurre...
(QB_NEW_EN)
[grammar] ~62-~62: There might be a mistake here.
Context: ...imestamp**: When the evaluation occurred - Input: The actual content that was map...
(QB_NEW_EN)
[grammar] ~63-~63: There might be a mistake here.
Context: ... content that was mapped to be evaluated - Output: The evaluation result/score - ...
(QB_NEW_EN)
[grammar] ~64-~64: There might be a mistake here.
Context: ... Output: The evaluation result/score - Completed Runs: Total successful/error...
(QB_NEW_EN)
[grammar] ~65-~65: There might be a mistake here.
Context: ...ns**: Total successful/error evaluations - Error Runs: Failed evaluation attempts...
(QB_NEW_EN)
🔇 Additional comments (1)
monitoring/using-monitors.mdx (1)
80-80: EOF clean – no stray “80” line detected
| <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" /> | ||
| </Frame> | ||
|
|
||
| ## What is a Monitor? |
There was a problem hiding this comment.
This should talk about LLM-as-a-judge vs "made by traceloop" and connect it to our evals. I wouldn't put the list of monitors here
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 9233710 in 1 minute and 47 seconds. Click for details.
- Reviewed
102lines of code in3files - Skipped
0files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. mint.json:144
- Draft comment:
Navigation reordering: The 'Monitoring' & 'Prompt Management' groups were removed from their initial positions and reinserted later. Confirm that this intended reordering preserves the desired navigation order. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =0%<= threshold50%The comment is asking the PR author to confirm their intention regarding the reordering of navigation items. This violates the rule against asking the author to confirm their intention or ensure behavior is intended.
2. monitoring/defining-monitors.mdx:85
- Draft comment:
Changed JSON key mapping from '[0].text' to '0.text'. Verify that this new syntax is correct and consistent with parsing logic. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =0%<= threshold50%The comment is asking the PR author to verify the correctness and consistency of a change in JSON key mapping syntax. This falls under asking the author to double-check their work, which is against the rules. The comment does not provide a specific suggestion or point out a clear issue, so it should be removed.
3. monitoring/introduction.mdx:16
- Draft comment:
Updated 'What is a Monitor?' section: The paragraph was consolidated and evaluator types simplified. Confirm that omitting the previous detailed metric breakdown is intentional. - Reason this comment was not posted:
Comment looked like it was already resolved.
4. monitoring/introduction.mdx:23
- Draft comment:
Typo: Consider hyphenating "built in" to "built-in" in "Traceloop built in evaluators" to improve readability. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% While the suggestion is technically correct, I need to consider if this is important enough to warrant a comment. This is documentation, so clarity and correctness are somewhat important. However, the meaning is perfectly clear either way, and this is a very minor stylistic issue. The comment is technically correct, but it might be too minor of an issue to warrant a PR comment. Many style guides might not even require this hyphenation. However, since this is documentation that will be publicly visible, maintaining proper grammar could be considered important for professionalism. While technically correct, this comment is too minor and obvious to warrant inclusion in the PR review.
Workflow ID: wflow_99S7kEkpEmBgJUtD
You can customize
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (3)
monitoring/defining-monitors.mdx (3)
103-104: Terminology: “Rate sample” → “Sample rate”.-### Advanced -You can set a **Rate sample** to control the percentage of spans within the selected filter group that the monitor will run on. +### Advanced +Set a **Sample rate** to control the percentage of spans within the selected filter group that the monitor will run on.
37-40: Add alt text to all images (accessibility + SEO).All
tags lack alt. Provide concise, equivalent alt text for light/dark variants.
- <img className="block dark:hidden" - src="/img/monitor/monitor-filter-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-filter-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-filter-light.png" alt="Monitor span filter UI (light)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-filter-dark.png" alt="Monitor span filter UI (dark)" /> @@ -<img className="block dark:hidden" - src="/img/monitor/monitor-filter-options-light.png" - style={{maxWidth: '500px'}} -/> -<img className="hidden dark:block" - src="/img/monitor/monitor-filter-options-dark.png" - style={{maxWidth: '500px'}} -/> +<img className="block dark:hidden" src="/img/monitor/monitor-filter-options-light.png" style={{maxWidth: '500px'}} alt="Filter options: Environment, Workflow, Service, AI Data, Attributes (light)" /> +<img className="hidden dark:block" src="/img/monitor/monitor-filter-options-dark.png" style={{maxWidth: '500px'}} alt="Filter options: Environment, Workflow, Service, AI Data, Attributes (dark)" /> @@ - <img className="block dark:hidden" - src="/img/monitor/monitor-settings-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-settings-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-settings-light.png" alt="Monitor settings with input mapping (light)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-settings-dark.png" alt="Monitor settings with input mapping (dark)" /> @@ - <img className="block dark:hidden" - src="/img/monitor/monitor-json-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-json-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-json-light.png" alt="JSON key mapping preview extracting 0.text (light)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-json-dark.png" alt="JSON key mapping preview extracting 0.text (dark)" /> @@ - <img className="block dark:hidden" - src="/img/monitor/monitor-regex-light.png" /> - <img className="hidden dark:block" - src="/img/monitor/monitor-regex-dark.png" /> + <img className="block dark:hidden" src="/img/monitor/monitor-regex-light.png" alt="Regex extraction preview on text value (light)" /> + <img className="hidden dark:block" src="/img/monitor/monitor-regex-dark.png" alt="Regex extraction preview on text value (dark)" />Also applies to: 53-60, 71-74, 88-91, 97-100
94-94: Use a safer regex that handles escaped quotes.
(.+?)can over-capture; prefer a class-limited group that allows escapes.-You can use Regex like `text":"(.+?)"` to extract just the text content. The regex will be applied to the Preview table, allowing you to see the extracted result in real-time. +Use a regex like `text":"([^"\\]*(?:\\.[^"\\]*)*)"` to extract the text content without spanning escaped quotes. The regex is applied to the Preview table in real time.
🧹 Nitpick comments (6)
mint.json (3)
147-149: Inconsistency: Datasets group still present (summary says removed).AI summary says Datasets was removed, but these lines keep it. Either remove this group or update the summary/PR description for consistency.
- { - "group": "Datasets", - "pages": ["datasets/quick-start", "datasets/sdk-usage"] - }, + // Datasets group intentionally removed (migrated/archived).If removal is intended, consider adding redirects from /datasets/* to replacement pages.
80-90: Duplicate “Quick Start” groups can confuse navigation.You have two groups named “Quick Start.” Consider renaming the Hub one (e.g., “Hub Quick Start”) or merging for clarity.
- { - "group": "Quick Start", + { + "group": "Hub Quick Start", "pages": ["hub/getting-started", "hub/configuration"] },Also applies to: 143-145
106-137: Split “Integrations” groups; consolidate PostHog under the main Integrations.Two separate “Integrations” groups appear; fold PostHog into the first to keep outcomes in one place.
- { - "group": "Integrations", - "pages": ["integrations/posthog"] - }, + // Move "integrations/posthog" into the primary Integrations list above.Also applies to: 181-183
monitoring/defining-monitors.mdx (2)
30-31: Tighten phrasing for clarity.-Next, you will be able to configure which spans will be monitored. +Then configure which spans to monitor.
26-26: Cross-link to “Using Monitors.”Add a forward link to help users move from setup to interpretation.
</Steps> +See how to interpret results in [Using Monitors](/monitoring/using-monitors).monitoring/introduction.mdx (1)
23-23: Hyphenation: “built in” → “built-in”.-- **Traceloop built in evaluators**: deterministic evaluations for structural validation, safety checks, and syntactic analysis. +- **Traceloop built-in evaluators**: deterministic evaluations for structural validation, safety checks, and syntactic analysis.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
mint.json(2 hunks)monitoring/defining-monitors.mdx(1 hunks)monitoring/introduction.mdx(1 hunks)
🧰 Additional context used
🪛 LanguageTool
monitoring/defining-monitors.mdx
[grammar] ~29-~29: There might be a mistake here.
Context: ...aluator you want to run in your monitor. Next, you will be able to configure whic...
(QB_NEW_EN)
[grammar] ~53-~53: There might be a mistake here.
Context: ...es <img className="block dark:hidden" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~54-~54: There might be a mistake here.
Context: ...nitor/monitor-filter-options-light.png" style={{maxWidth: '500px'}} /> <img clas...
(QB_NEW_EN)
[grammar] ~55-~55: There might be a mistake here.
Context: ...ight.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~56-~56: There might be a mistake here.
Context: ...t.png" style={{maxWidth: '500px'}} /> <img className="hidden dark:block" ...
(QB_NEW_EN)
[grammar] ~57-~57: There might be a mistake here.
Context: ...} /> <img className="hidden dark:block" src="/img/monitor/monitor-filter-options...
(QB_NEW_EN)
[grammar] ~58-~58: There might be a mistake here.
Context: ...monitor/monitor-filter-options-dark.png" style={{maxWidth: '500px'}} /> ## Moni...
(QB_NEW_EN)
[grammar] ~59-~59: There might be a mistake here.
Context: ...ark.png" style={{maxWidth: '500px'}} /> ## Monitor Settings ### Map Input ...
(QB_NEW_EN)
[grammar] ~103-~103: There might be a mistake here.
Context: ...egex-dark.png" /> ### Advanced You can set a Rate sample to control...
(QB_NEW_EN)
🔇 Additional comments (4)
mint.json (1)
172-179: Monitoring and Prompt Management groups: LGTM.Groups and page slugs align with the new docs. No issues spotted.
monitoring/defining-monitors.mdx (2)
12-25: Nice use of Mintlify Steps.Clear, actionable flow. Good cross-links to OpenLLMetry and Evaluators.
81-86: Confirm JSON array index syntax: Verify whether the UI expects bracketed indices ([0].text) instead of0.text, then update the example or add a note clarifying the correct array-index notation.monitoring/introduction.mdx (1)
6-6: Intro restructure: LGTM.Clear definition of Monitors and evaluator types; aligns with the new Monitoring docs.
Also applies to: 16-26
| One of the key features of Traceloop is the ability to monitor the quality of your LLM outputs. It helps you to detect hallucinations and regressions in the quality of your models and prompts. | ||
| One of the key features of Traceloop is the ability to monitor the quality of your LLM outputs in **real time**. It helps you to detect hallucinations and regressions in the quality of your models and prompts. | ||
|
|
||
| To start monitoring your LLM outputs, make sure you installed OpenLLMetry and configured it to send data to Traceloop. If you haven't done that yet, you can follow the instructions in the [Getting Started](/openllmetry/getting-started) guide. |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Broken link risk: “Getting Started” path may not exist.
Link points to /openllmetry/getting-started, but navigation lists per-language pages (…-python, …-ts, …-nextjs, …-go, …-ruby). Verify target exists or update to a valid page.
If missing, consider linking to the language selector page or to /openllmetry/introduction.
🏁 Script executed:
#!/bin/bash
# Verify whether a generic getting-started page exists.
fd -a --strip-cwd-prefix 'getting-started.mdx' | sed 's/^/FILE: /'
fd -a --strip-cwd-prefix 'openllmetry' | rg -nI -e 'getting-started' -e 'Getting Started' -n -H -SLength of output: 212
🏁 Script executed:
#!/bin/bash
# List top-level directories to inspect documentation structure
ls -1
# Check for a docs folder and list its subdirectories if present
if [ -d docs ]; then
echo "DIR: docs"
ls -1 docs
fi
# Find any directory named 'openllmetry'
fd -t d --strip-cwd-prefix openllmetry
# Find all .mdx files and filter for 'getting-started' (case-insensitive)
fd -t f -e mdx --strip-cwd-prefix . | rg -i 'getting-started'
# Preview the generic getting-started.mdx if it exists at repository root
if [ -f getting-started.mdx ]; then
echo "---- beginning of getting-started.mdx ----"
head -n20 getting-started.mdx
echo "---- end of snippet ----"
fiLength of output: 596
Broken “Getting Started” link—update to a valid per-language path
In monitoring/introduction.mdx:8, the link /openllmetry/getting-started 404s. Only language-specific guides exist under /openllmetry/getting-started-python, …-ts, …-nextjs, …-go, and …-ruby. Point this link to the correct per-language page (or add a generic landing page if intended).
🤖 Prompt for AI Agents
In monitoring/introduction.mdx around line 8, the link
`/openllmetry/getting-started` 404s because only language-specific guides exist;
update the anchor to point to the appropriate per-language landing (e.g.
`/openllmetry/getting-started-python`, `/openllmetry/getting-started-ts`,
`/openllmetry/getting-started-nextjs`, `/openllmetry/getting-started-go`, or
`/openllmetry/getting-started-ruby`) based on this doc’s audience, or create a
generic `/openllmetry/getting-started` landing page and change the link to that;
ensure the new target path is valid and the link text remains the same.
| <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" /> | ||
| <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" /> |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add alt text to images (accessibility + SEO).
- <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" />
- <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" />
+ <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" alt="Monitors list in the dashboard (light)" />
+ <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" alt="Monitors list in the dashboard (dark)" />📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" /> | |
| <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" /> | |
| <img className="block dark:hidden" src="/img/monitor/monitor-list-light.png" alt="Monitors list in the dashboard (light)" /> | |
| <img className="hidden dark:block" src="/img/monitor/monitor-list-dark.png" alt="Monitors list in the dashboard (dark)" /> |
🤖 Prompt for AI Agents
In monitoring/introduction.mdx around lines 12 to 13, the two <img> tags lack
alt attributes which harms accessibility and SEO; update each <img> element to
include meaningful alt text (e.g., "Monitor list — light theme" and "Monitor
list — dark theme" or other context-appropriate descriptions) so screen readers
and search engines receive descriptive content, keeping the alt strings concise
and matching the image purpose.
Important
Updated monitoring documentation with new pages, expanded content, and reorganized navigation for improved clarity and usability.
monitoring/defining-monitors.mdxandmonitoring/using-monitors.mdx.monitoring/introduction.mdxwith "What is a Monitor?" section.mint.jsonto include new Monitoring pages and reorganized navigation structure.This description was created by
for 9233710. You can customize this summary. It will automatically update as commits are pushed.
Summary by CodeRabbit
Navigation
Documentation
Assets