feat(telemetry): add human-driven eval foundation#88
Merged
khaliqgant merged 4 commits intomainfrom May 8, 2026
Merged
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR extends HumanEvalTrial with optional input and expected fields, adds a deterministic maxQuestionMarks check, and makes renderHumanEvalReview conditionally emit input bullets, Must/Must Not sections, a filtered Deterministic Checks subsection, and an Output block only when data exists; tests updated accordingly. ChangesHuman Review Rendering Enhancements
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/telemetry/src/evals/human-suite.ts`:
- Around line 397-412: The Must/Must Not sections use
(test.expected?.must?.length ?? 0) and iterate with for...of without verifying
the values are arrays; change both guards to use
Array.isArray(test.expected?.must) and Array.isArray(test.expected?.mustNot)
before pushing headers or iterating, and only run the for...of loops when those
Array.isArray checks pass (mirroring the existing safe pattern used for other
expected fields in this file); update the conditional around the
`lines.push('### Must')`/`lines.push('### Must Not')` blocks to use these
Array.isArray checks for `test.expected.must` and `test.expected.mustNot`
respectively.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2fc95317-1342-48c8-bc51-c4309a5d090c
📒 Files selected for processing (2)
packages/telemetry/src/evals/human-suite.tspackages/telemetry/test/human-suite.test.ts
There was a problem hiding this comment.
♻️ Duplicate comments (1)
packages/telemetry/src/evals/human-suite.ts (1)
407-422:⚠️ Potential issue | 🟠 Major | ⚡ Quick winGuard
must/mustNotwithArray.isArray()before iterating.Line 407 and Line 415 rely on
.lengthand then iterate withfor...of; malformed JSON like{ must: { length: 1 } }passes the length check but throws at runtime during iteration.Proposed fix
- if ((test.expected?.must?.length ?? 0) > 0) { + const mustItems = Array.isArray(test.expected?.must) ? test.expected.must : []; + if (mustItems.length > 0) { lines.push('### Must'); lines.push(''); - for (const item of test.expected?.must ?? []) { + for (const item of mustItems) { lines.push(`- ${item}`); } lines.push(''); } - if ((test.expected?.mustNot?.length ?? 0) > 0) { + const mustNotItems = Array.isArray(test.expected?.mustNot) ? test.expected.mustNot : []; + if (mustNotItems.length > 0) { lines.push('### Must Not'); lines.push(''); - for (const item of test.expected?.mustNot ?? []) { + for (const item of mustNotItems) { lines.push(`- ${item}`); } lines.push(''); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/telemetry/src/evals/human-suite.ts` around lines 407 - 422, The code in human-suite.ts checks test.expected?.must?.length and mustNot?.length then uses for...of, which can crash if must/mustNot are non-array objects with a length property; update both checks and iteration to first verify Array.isArray(test.expected?.must) and Array.isArray(test.expected?.mustNot) (e.g., if (Array.isArray(test.expected?.must) && test.expected.must.length > 0) { ... for (const item of test.expected.must) { ... } }) so only real arrays are iterated. Ensure you change both the length guards and the for...of loops that reference test.expected?.must and test.expected?.mustNot.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@packages/telemetry/src/evals/human-suite.ts`:
- Around line 407-422: The code in human-suite.ts checks
test.expected?.must?.length and mustNot?.length then uses for...of, which can
crash if must/mustNot are non-array objects with a length property; update both
checks and iteration to first verify Array.isArray(test.expected?.must) and
Array.isArray(test.expected?.mustNot) (e.g., if
(Array.isArray(test.expected?.must) && test.expected.must.length > 0) { ... for
(const item of test.expected.must) { ... } }) so only real arrays are iterated.
Ensure you change both the length guards and the for...of loops that reference
test.expected?.must and test.expected?.mustNot.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 98efd583-46b8-4a6c-a56f-43e19c1f9b77
📒 Files selected for processing (2)
packages/telemetry/src/evals/human-suite.tspackages/telemetry/test/human-suite.test.ts
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/telemetry/src/evals/human-suite.ts`:
- Around line 292-300: The check for maxQuestionMarks currently allows JS
coercion of malformed rubric values; update the block that references
expected.maxQuestionMarks, questionMarkCount, addCheck and checks to first
validate that expected.maxQuestionMarks is a finite number (e.g., using typeof
=== 'number' && Number.isFinite(expected.maxQuestionMarks')) before performing
the comparison against questionMarkCount computed from actual.content, and if
the value is not a finite number, skip adding the maxQuestionMarks check so
invalid config is ignored.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 028a0dd3-9679-49ae-a04f-b783b2bef153
📒 Files selected for processing (2)
packages/telemetry/src/evals/human-suite.tspackages/telemetry/test/human-suite.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Follow-up