fix: skip validation for defaultTest to allow partial test case properties #4732
Conversation
…rties Fixes #4478 Previously, defaultTest was validated as a complete test case, causing errors when it only contained partial properties like options.provider.embedding. This change adds an isDefaultTest parameter to readTest() that skips validation when loading defaultTest configurations. - Added isDefaultTest parameter to readTest function - Updated config loader to pass true when loading defaultTest - Added comprehensive test coverage for various provider configurations
…ons\n\n- Add tests to verify defaultTest with embedding provider gets loaded correctly\n- Add tests to verify defaultTest options are merged with test case options\n- Fix TypeScript errors in tests (use 'similar' instead of 'similarity')
…and loadApiProviders\n- Mock readPrompts and loadApiProviders to return expected formats for test scenarios\n- Mock readTests to return specific test data for defaultTest test case\n- These changes ensure tests match the actual behavior of resolveConfigs
|
⏩ No test execution environment matched (1866a1b) View output ↗ View check history
|
TestGru AssignmentSummary
Files
Tip You can |
📝 WalkthroughWalkthroughThe changes update the Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes were found. All modifications and additions are directly related to the objectives in the linked issue, focusing on the handling and validation of Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! ✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
test/util/testCaseReader.test.ts (1)
631-644: Potential test duplication.This test appears to duplicate the functionality already tested in "readTest with string input (path to test config)" starting at line 434. Consider whether this additional test case is necessary or if it should be consolidated.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
src/util/config/load.ts(1 hunks)src/util/testCaseReader.ts(2 hunks)test/evaluator.test.ts(1 hunks)test/util/config/load.test.ts(5 hunks)test/util/testCaseReader.test.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
`**/*.{ts,tsx}`: Use TypeScript with strict type checking.
**/*.{ts,tsx}: Use TypeScript with strict type checking.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/util/testCaseReader.tstest/util/testCaseReader.test.tssrc/util/config/load.tstest/evaluator.test.tstest/util/config/load.test.ts
`**/*.{js,jsx,ts,tsx}`: Follow established import order with @trivago/prettier-p...
**/*.{js,jsx,ts,tsx}: Follow established import order with @trivago/prettier-plugin-sort-imports.
Use consistent curly braces for all control statements.
Prefer const over let; avoid var.
Use object shorthand syntax whenever possible.
Use async/await for asynchronous code.
Use consistent error handling with proper type checks.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/util/testCaseReader.tstest/util/testCaseReader.test.tssrc/util/config/load.tstest/evaluator.test.tstest/util/config/load.test.ts
`**/*.{ts,tsx}`: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
**/*.{ts,tsx}: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
📄 Source: CodeRabbit Inference Engine (.cursor/rules/gh-cli-workflow.mdc)
List of files the instruction was applied to:
src/util/testCaseReader.tstest/util/testCaseReader.test.tssrc/util/config/load.tstest/evaluator.test.tstest/util/config/load.test.ts
`test/**/*.{test,spec}.{js,jsx,ts,tsx}`: Follow Jest best practices with describe/it blocks.
test/**/*.{test,spec}.{js,jsx,ts,tsx}: Follow Jest best practices with describe/it blocks.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
test/util/testCaseReader.test.tstest/evaluator.test.tstest/util/config/load.test.ts
`**/*.{test,spec}.{js,ts,tsx}`: Avoid disabling or skipping tests unless absolutely necessary and documented
**/*.{test,spec}.{js,ts,tsx}: Avoid disabling or skipping tests unless absolutely necessary and documented
📄 Source: CodeRabbit Inference Engine (.cursor/rules/gh-cli-workflow.mdc)
List of files the instruction was applied to:
test/util/testCaseReader.test.tstest/evaluator.test.tstest/util/config/load.test.ts
`test/**/*.{test,spec}.ts`: Mock as few functions as possible to keep tests real...
test/**/*.{test,spec}.ts: Mock as few functions as possible to keep tests realistic
Never increase the function timeout - fix the test instead
Organize tests in descriptivedescribeanditblocks
When writing expectations, prefer assertions on entire objects rather than individual keys
Clean up after tests to prevent side effects (e.g., use afterEach(() => { jest.resetAllMocks(); }))
Run tests with--randomizeflag to ensure your mocks setup and teardown don't affect other tests
Use Jest's mocking utilities rather than complex custom mocks
Prefer shallow mocking over deep mocking
Mock external dependencies but not the code being tested
Reset mocks between tests to prevent test pollution
For database tests, use in-memory instances or proper test fixtures
Test both success and error cases for each provider
Mock API responses to avoid external dependencies in tests
Validate that provider options are properly passed to the underlying service
Test error handling and edge cases (rate limits, timeouts, etc.)
Ensure provider caching behaves as expected
Always include both--coverageand--randomizeflags when running tests
Run tests in a single pass (no watch mode for CI)
Ensure all tests are independent and can run in any order
Clean up any test data or mocks after each test
📄 Source: CodeRabbit Inference Engine (.cursor/rules/jest.mdc)
List of files the instruction was applied to:
test/util/testCaseReader.test.tstest/evaluator.test.tstest/util/config/load.test.ts
🧠 Learnings (6)
📓 Common learnings
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test both success and error cases for each provider
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Use Jest's mocking utilities rather than complex custom mocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Reset mocks between tests to prevent test pollution
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Prefer shallow mocking over deep mocking
src/util/testCaseReader.ts (13)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests in a single pass (no watch mode for CI)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Always include both `--coverage` and `--randomize` flags when running tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Never increase the function timeout - fix the test instead
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test both success and error cases for each provider
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : For database tests, use in-memory instances or proper test fixtures
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Organize tests in descriptive `describe` and `it` blocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : When writing expectations, prefer assertions on entire objects rather than individual keys
test/util/testCaseReader.test.ts (14)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test both success and error cases for each provider
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure all tests are independent and can run in any order
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Always include both `--coverage` and `--randomize` flags when running tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Organize tests in descriptive `describe` and `it` blocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Reset mocks between tests to prevent test pollution
Learnt from: CR
PR: promptfoo/promptfoo#0
File: CLAUDE.md:0-0
Timestamp: 2025-06-30T13:43:03.637Z
Learning: Applies to test/**/*.{test,spec}.{js,jsx,ts,tsx} : Follow Jest best practices with describe/it blocks.
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Use Jest's mocking utilities rather than complex custom mocks
src/util/config/load.ts (10)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests in a single pass (no watch mode for CI)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Never increase the function timeout - fix the test instead
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Always include both `--coverage` and `--randomize` flags when running tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock external dependencies but not the code being tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure all tests are independent and can run in any order
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
test/evaluator.test.ts (12)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test both success and error cases for each provider
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure all tests are independent and can run in any order
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Reset mocks between tests to prevent test pollution
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Organize tests in descriptive `describe` and `it` blocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : When writing expectations, prefer assertions on entire objects rather than individual keys
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Clean up after tests to prevent side effects (e.g., use afterEach(() => { jest.resetAllMocks(); }))
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : For database tests, use in-memory instances or proper test fixtures
test/util/config/load.test.ts (17)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock external dependencies but not the code being tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Reset mocks between tests to prevent test pollution
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Use Jest's mocking utilities rather than complex custom mocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Prefer shallow mocking over deep mocking
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Clean up after tests to prevent side effects (e.g., use afterEach(() => { jest.resetAllMocks(); }))
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.304Z
Learning: When developing or testing examples locally, use 'npm run local' commands instead of 'npx promptfoo@latest' to ensure local changes are tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: CLAUDE.md:0-0
Timestamp: 2025-06-30T13:43:03.637Z
Learning: Use CommonJS modules (type: "commonjs" in package.json).
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test both success and error cases for each provider
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: After resolving conflicts, run the full test suite from root: `npm test`
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Clean up any test data or mocks after each test
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: CLAUDE.md:0-0
Timestamp: 2025-06-30T13:43:03.637Z
Learning: Applies to test/**/*.{test,spec}.{js,jsx,ts,tsx} : Follow Jest best practices with describe/it blocks.
🧬 Code Graph Analysis (2)
test/util/testCaseReader.test.ts (1)
src/util/testCaseReader.ts (1)
readTest(235-285)
src/util/config/load.ts (1)
src/util/testCaseReader.ts (1)
readTest(235-285)
⏰ Context from checks skipped due to timeout of 90000ms (18)
- GitHub Check: webui tests
- GitHub Check: Redteam
- GitHub Check: Redteam Custom Enterprise Server
- GitHub Check: Test on Node 18.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Test on Node 20.x and windows-latest
- GitHub Check: Test on Node 18.x and windows-latest
- GitHub Check: Share Test
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Test on Node 18.x and macOS-latest
- GitHub Check: Cursor BugBot
- GitHub Check: Build Docs
- GitHub Check: Build on Node 18.x
- GitHub Check: Build on Node 22.x
- GitHub Check: Build on Node 24.x
- GitHub Check: Build on Node 20.x
- GitHub Check: Style Check
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (12)
src/util/config/load.ts (1)
526-526: LGTM! Correctly implements the default test validation skip.This change properly passes the
isDefaultTestflag astruewhen loading default test configurations, allowing partial test case properties without triggering validation errors. The implementation aligns perfectly with the PR objective to fix issue #4478.src/util/testCaseReader.ts (2)
238-238: LGTM! Clean backward-compatible parameter addition.The optional
isDefaultTestparameter with a default value offalsemaintains backward compatibility while enabling the new functionality to skip validation for default test configurations.
264-264: LGTM! Proper conditional validation implementation.The validation logic correctly skips the required properties check when
isDefaultTestis true, allowing partial default test configurations to be loaded without errors. The inline comment clearly explains the purpose of this conditional behavior.test/evaluator.test.ts (1)
2919-3030: Well-structured tests for defaultTest merging functionality.These tests appropriately verify the runtime merging behavior of
defaultTestoptions during evaluation, which complements the enhanced loading and validation mechanisms introduced in the PR. The test scenarios cover both basic merging and override behavior effectively.Key strengths:
- Proper Jest structure with describe/it blocks
- Realistic mock provider setup with appropriate token usage
- Comprehensive assertions using
toEqual()for object comparisons- Good coverage of both merging and override scenarios
- Follows the guideline of asserting on entire objects rather than individual properties
The tests align well with the PR objective of allowing partial
defaultTestconfigurations while ensuring proper merging behavior during evaluation.test/util/testCaseReader.test.ts (5)
534-552: LGTM: Well-structured test for embedding provider validation skip.This test effectively verifies that
defaultTestconfigurations with embedding provider options bypass validation, which aligns with the PR objectives.
554-565: LGTM: Good coverage for model-graded eval provider scenario.This test case properly validates that string-based provider configurations for model-graded evaluation work correctly with the
isDefaultTestflag.
567-593: LGTM: Comprehensive test for text provider configuration.This test case thoroughly validates complex nested provider configurations, ensuring the entire structure is preserved when validation is skipped for default tests.
595-618: LGTM: Good test coverage for object-based provider configuration.This test case validates that object-based provider configurations (with id and config properties) work correctly with the validation skip functionality.
620-629: LGTM: Important negative test case for validation behavior.This test ensures that the existing validation logic remains intact when
isDefaultTestis false, which is crucial for maintaining backward compatibility.test/util/config/load.test.ts (3)
62-138: LGTM: Comprehensive mocking setup for integration testing.The mocking setup is well-structured and properly handles the new
isDefaultTestparameter in thereadTestmock. The realistic mock implementations will help ensure accurate integration testing.
1319-1397: LGTM: Comprehensive integration test for embedding provider configuration.This test effectively validates the end-to-end loading of
defaultTestconfigurations with embedding providers. The test setup is realistic and the assertions thoroughly verify both the defaultTest and regular test loading.
1399-1439: LGTM: Good integration test for model-graded eval provider.This test provides good coverage for the string-based provider configuration scenario in
defaultTest. The test structure is consistent and the assertions properly verify the provider configuration is preserved.
Relates to #4478
Previously, defaultTest was validated as a complete test case, causing errors when it only contained partial properties like options.provider.embedding.
This change adds an isDefaultTest parameter to readTest() that skips validation when loading defaultTest configurations, allowing it to contain only the properties needed for merging with actual test cases.