feat(config): add support for loading defaultTest from external files #4720
Conversation
- Allow defaultTest to reference external YAML/JSON files using file:// syntax - Update Zod schema to accept string values for defaultTest with file:// prefix - Add documentation explaining the new feature in configuration guide - Update reference documentation to reflect the new string type option - Migrate assertions-generate example to demonstrate external defaultTest usage - Regenerate JSON schema to include the new validation rules This enables better configuration reusability across projects and cleaner main config files by allowing shared test configurations to be maintained in separate files.
TestGru AssignmentSummary
Files
Tip You can |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
📝 WalkthroughWalkthroughThis change introduces support for specifying the Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! ✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (1)
examples/assertions-generate/README.md (1)
5-7: Incorrect example name in the init command
assertion-scoring-overridedoesn’t match the folder nameassertions-generate; copy-pasting this will fail.-npx promptfoo@latest init --example assertion-scoring-override +npx promptfoo@latest init --example assertions-generate
🧹 Nitpick comments (5)
src/generated-constants.ts (1)
1-3: POSTHOG_KEY hard-codes an empty string – consider falling back to the env varSetting the constant to
''removes any ability to enable analytics at runtime.
If that is intentional, fine—otherwise restore the env fallback so users can opt-in viaPROMPTFOO_POSTHOG_KEY.-export const POSTHOG_KEY = ''; +export const POSTHOG_KEY = process.env.PROMPTFOO_POSTHOG_KEY ?? '';site/docs/configuration/reference.md (1)
32-32: Minor grammar tweak for clarityAdd the pronoun to make the sentence complete.
-| defaultTest | string \| Partial [Test Case](#test-case) | No | Sets the default properties for each test case. Can be an inline object or a `file://` path to an external YAML/JSON file. | +| defaultTest | string \| Partial [Test Case](#test-case) | No | Sets the default properties for each test case. It can be an inline object or a `file://` path to an external YAML/JSON file. |examples/assertions-generate/shared/defaultTest.yaml (1)
14-15: Trailing whitespace and missing newline at EOFYAML-lint flags both issues; fix to keep the repo lint-clean.
- embedding: openai:embedding:text-embedding-3-small + embedding: openai:embedding:text-embedding-3-small +src/util/config/load.ts (2)
518-528: Simplify redundant basePath handling.The basePath manipulation is unnecessary since
cliState.basePathwas already set tobasePathon line 512. The temporary save/restore pattern adds complexity without benefit.Apply this diff to simplify the code:
- // Load defaultTest from file:// reference if needed - let processedDefaultTest: Partial<TestCase> | undefined; - if (typeof defaultTestRaw === 'string' && defaultTestRaw.startsWith('file://')) { - // Set basePath in cliState temporarily for file resolution - const originalBasePath = cliState.basePath; - cliState.basePath = basePath; - const loaded = maybeLoadFromExternalFile(defaultTestRaw); - cliState.basePath = originalBasePath; - processedDefaultTest = loaded as Partial<TestCase>; - } else if (defaultTestRaw) { - processedDefaultTest = defaultTestRaw as Partial<TestCase>; - } + // Load defaultTest from file:// reference if needed + let processedDefaultTest: Partial<TestCase> | undefined; + if (typeof defaultTestRaw === 'string' && defaultTestRaw.startsWith('file://')) { + const loaded = maybeLoadFromExternalFile(defaultTestRaw); + processedDefaultTest = loaded as Partial<TestCase>; + } else if (defaultTestRaw) { + processedDefaultTest = defaultTestRaw as Partial<TestCase>; + }
523-525: Consider adding type validation for loaded defaultTest content.The code casts the loaded content to
Partial<TestCase>without validation. Consider validating the structure to catch configuration errors early.- const loaded = maybeLoadFromExternalFile(defaultTestRaw); - processedDefaultTest = loaded as Partial<TestCase>; + const loaded = maybeLoadFromExternalFile(defaultTestRaw); + if (loaded && typeof loaded === 'object') { + processedDefaultTest = loaded as Partial<TestCase>; + } else { + throw new Error(`Invalid defaultTest content loaded from ${defaultTestRaw}. Expected an object but got ${typeof loaded}`); + }
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
examples/assertions-generate/README.md(1 hunks)examples/assertions-generate/promptfooconfig.yaml(1 hunks)examples/assertions-generate/shared/defaultTest.yaml(1 hunks)site/docs/configuration/guide.md(1 hunks)site/docs/configuration/reference.md(1 hunks)site/static/config-schema.json(1 hunks)src/generated-constants.ts(1 hunks)src/types/index.ts(2 hunks)src/util/config/load.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (7)
`**/*.{ts,tsx}`: Use TypeScript with strict type checking.
**/*.{ts,tsx}: Use TypeScript with strict type checking.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/generated-constants.tssrc/types/index.tssrc/util/config/load.ts
`**/*.{js,jsx,ts,tsx}`: Follow established import order with @trivago/prettier-p...
**/*.{js,jsx,ts,tsx}: Follow established import order with @trivago/prettier-plugin-sort-imports.
Use consistent curly braces for all control statements.
Prefer const over let; avoid var.
Use object shorthand syntax whenever possible.
Use async/await for asynchronous code.
Use consistent error handling with proper type checks.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/generated-constants.tssrc/types/index.tssrc/util/config/load.ts
`**/*.{ts,tsx}`: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
**/*.{ts,tsx}: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
📄 Source: CodeRabbit Inference Engine (.cursor/rules/gh-cli-workflow.mdc)
List of files the instruction was applied to:
src/generated-constants.tssrc/types/index.tssrc/util/config/load.ts
`examples/**/*`: Structure examples consistently with common patterns across the...
examples/**/*: Structure examples consistently with common patterns across the project
Ensure all examples are functional and up-to-date
Keep examples as simple as possible while still demonstrating the concept
Follow the same code style guidelines as the main project in all example code and configuration
Include comments to explain non-obvious parts in example code and configuration
Use descriptive variable and function names in example code and configuration
Keep code DRY within reason; examples may duplicate code for clarity
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/assertions-generate/shared/defaultTest.yamlexamples/assertions-generate/README.mdexamples/assertions-generate/promptfooconfig.yaml
`examples/*/README.md`: The README.md must begin with the folder name as an H1 h...
examples/*/README.md: The README.md must begin with the folder name as an H1 heading
Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Include a comprehensive README.md that explains the purpose, prerequisites, instructions, and expected outputs for each example
Document any model-specific capabilities or limitations in example README files
Clearly list all required environment variables at the beginning of the README
For each environment variable, explain its purpose, how to obtain it, and any default values or constraints in the README
Include a sample '.env' file or instructions when multiple environment variables are needed
Document any required API keys or credentials in the README
Provide instructions for cleaning up resources after running the example in the README
When creating examples for specific providers, explain any provider-specific configuration in the README
Document required environment variables for provider-specific examples in the README
Include information about pricing or usage limits for provider-specific examples in the README
Highlight unique features or capabilities of providers in the README
Compare to similar providers where appropriate in the README
Make it clear which features each example demonstrates in the README
Include links to more comprehensive documentation where appropriate in the README
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/assertions-generate/README.md
`site/{docs,blog,src/pages}/**/*.{md,mdx}`: Prioritize minimal edits when updati...
site/{docs,blog,src/pages}/**/*.{md,mdx}: Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally
Structure content to reveal information progressively: begin with essential actions and information, then provide deeper context as necessary; organize information from most important to least important
Use action-oriented language: clearly outline actionable steps, use concise and direct language, prefer active voice over passive voice, and use imperative mood for instructions
Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Use 'Promptfoo' (capitalized) at the beginning of sentences or in headings, and 'promptfoo' (lowercase) in code examples, terminal commands, or when referring to the package name; be consistent with the chosen capitalization within each document or section
Each markdown documentation page must include required front matter fields: 'title' (page title) and 'description' (concise summary, ideally 150-160 characters)
Optional front matter fields for markdown documentation pages: 'image', 'keywords', and 'sidebar_position' can be included to enhance SEO and navigation
Keep titles under 60 characters for optimal display in search results; write description meta tags between 150-160 characters; include relevant keywords naturally in both title and description; use unique titles and descriptions for each page; for image thumbnails, use 1200×630 pixels for optimal social media display
Only add a title attribute to code blocks that represent complete, runnable files; do not add titles to code fragments, partial examples, or snippets that aren't meant to be used as standalone files; applies to all code blocks regardless of language
Use special comment directives to highlight specific lines in code blocks: 'highlight-next-line', 'highlight-start'/'highlight-end', or line numbers in curly braces; never remove existing highlight directives when editing a document; preserve the original author's intent by maintaining their highlighting
Always specify the language for syntax highlighting in code blocks; use meaningful titles for code blocks when appropriate; keep code examples concise and focused; ensure code examples are correct and up-to-date; include comments in code examples where helpful
Use Docusaurus admonitions (note, tip, info, warning, danger) by wrapping content with three colons ':::' followed by the type; always include empty lines before and after the content inside admonitions, especially when using Prettier; you can include Markdown formatting within admonitions
When using Prettier to format Markdown files, always include empty lines around admonition content to prevent Prettier from reformatting admonitions to invalid syntax
Search the codebase to provide accurate and relevant backlinks within the documentation; include references to specific files, functions, or concepts; ensure backlinks are relevant to the current section
End each documentation section with a concise 'See Also' or 'Related Concepts' heading; link to related documentation or concepts; format the 'See Also' section consistently as a markdown list of links
Use clear, concise language; maintain a consistent tone throughout the documentation; write for an international audience (avoid idioms, colloquialisms, and culturally specific references); spell out acronyms on first use, followed by the acronym in parentheses
Include practical examples where appropriate; make examples realistic and applicable to common use cases; when showing configuration examples, include all necessary fields; demonstrate both basic and advanced usage when relevant
Use diagrams, screenshots, or illustrations to explain complex concepts; ensure images have appropriate alt text for accessibility; maintain a consistent visual style across all documentation; optimize images for web display
📄 Source: CodeRabbit Inference Engine (.cursor/rules/docusaurus.mdc)
List of files the instruction was applied to:
site/docs/configuration/reference.mdsite/docs/configuration/guide.md
`examples/*/promptfooconfig.yaml`: Include a working 'promptfooconfig.yaml' (or ...
examples/*/promptfooconfig.yaml: Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Ensure all configuration files pass YAML lint validation
When referencing external files in configuration, always use the 'file://' prefix
For trivial test cases, make them quirky and fun to increase engagement
Always use the latest model versions available in 2025 for model selection in examples
Include a mix of providers when comparing model performance in examples
When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Include placeholder values for secrets/credentials in configuration files
Format configuration files consistently
Always use the latest available model versions for each provider in configuration files
Update model versions when new ones become available in configuration files
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/assertions-generate/promptfooconfig.yaml
🧠 Learnings (9)
📓 Common learnings
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When referencing external files in configuration, always use the 'file://' prefix
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock external dependencies but not the code being tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Use Jest's mocking utilities rather than complex custom mocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : For database tests, use in-memory instances or proper test fixtures
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : When writing expectations, prefer assertions on entire objects rather than individual keys
Learnt from: CR
PR: promptfoo/promptfoo#0
File: CLAUDE.md:0-0
Timestamp: 2025-06-30T13:43:03.637Z
Learning: Applies to test/**/*.{test,spec}.{js,jsx,ts,tsx} : Follow Jest best practices with describe/it blocks.
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
examples/assertions-generate/shared/defaultTest.yaml (2)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
examples/assertions-generate/README.md (19)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Make it clear which features each example demonstrates in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Include links to more comprehensive documentation where appropriate in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Document any model-specific capabilities or limitations in example README files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Include a comprehensive README.md that explains the purpose, prerequisites, instructions, and expected outputs for each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : For each environment variable, explain its purpose, how to obtain it, and any default values or constraints in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: If the change is a feature, update the relevant documentation under `site/`
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When referencing external files in configuration, always use the 'file://' prefix
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Ensure all configuration files pass YAML lint validation
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include placeholder values for secrets/credentials in configuration files
site/docs/configuration/reference.md (9)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Use 'Promptfoo' (capitalized) at the beginning of sentences or in headings, and 'promptfoo' (lowercase) in code examples, terminal commands, or when referring to the package name; be consistent with the chosen capitalization within each document or section
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{ts,tsx} : Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
site/docs/configuration/guide.md (14)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Include links to more comprehensive documentation where appropriate in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: If the change is a feature, update the relevant documentation under `site/`
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Include practical examples where appropriate; make examples realistic and applicable to common use cases; when showing configuration examples, include all necessary fields; demonstrate both basic and advanced usage when relevant
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When referencing external files in configuration, always use the 'file://' prefix
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include placeholder values for secrets/credentials in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Ensure all configuration files pass YAML lint validation
src/types/index.ts (13)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure all tests are independent and can run in any order
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : When writing expectations, prefer assertions on entire objects rather than individual keys
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Prefer shallow mocking over deep mocking
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Organize tests in descriptive `describe` and `it` blocks
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{ts,tsx} : Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : For database tests, use in-memory instances or proper test fixtures
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests in a single pass (no watch mode for CI)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Always include both `--coverage` and `--randomize` flags when running tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: CLAUDE.md:0-0
Timestamp: 2025-06-30T13:43:03.637Z
Learning: Applies to test/**/*.{test,spec}.{js,jsx,ts,tsx} : Follow Jest best practices with describe/it blocks.
examples/assertions-generate/promptfooconfig.yaml (10)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When referencing external files in configuration, always use the 'file://' prefix
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
src/util/config/load.ts (13)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock external dependencies but not the code being tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests in a single pass (no watch mode for CI)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock API responses to avoid external dependencies in tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Run tests with `--randomize` flag to ensure your mocks setup and teardown don't affect other tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-06-30T13:44:03.652Z
Learning: Applies to **/*.{test,spec}.{js,ts,tsx} : Avoid disabling or skipping tests unless absolutely necessary and documented
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:19.000Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure all tests are independent and can run in any order
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Ensure provider caching behaves as expected
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Reset mocks between tests to prevent test pollution
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Always include both `--coverage` and `--randomize` flags when running tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Mock as few functions as possible to keep tests realistic
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : For database tests, use in-memory instances or proper test fixtures
site/static/config-schema.json (9)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-06-30T13:44:18.999Z
Learning: Applies to test/**/*.{test,spec}.ts : Validate that provider options are properly passed to the underlying service
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
🧬 Code Graph Analysis (1)
src/util/config/load.ts (3)
src/types/index.ts (1)
TestCase(680-682)src/util/file.ts (1)
maybeLoadFromExternalFile(38-77)src/util/testCaseReader.ts (1)
readTest(235-283)
🪛 YAMLlint (1.37.1)
examples/assertions-generate/shared/defaultTest.yaml
[error] 15-15: no new line character at the end of file
(new-line-at-end-of-file)
[error] 15-15: trailing spaces
(trailing-spaces)
🪛 LanguageTool
site/docs/configuration/reference.md
[style] ~32-~32: To form a complete sentence, be sure to include a subject or ‘there’.
Context: ... default properties for each test case. Can be an inline object or a file:// path...
(MISSING_IT_THERE)
⏰ Context from checks skipped due to timeout of 90000ms (15)
- GitHub Check: Cursor BugBot
- GitHub Check: Build Docs
- GitHub Check: Test on Node 20.x and macOS-latest
- GitHub Check: Test on Node 24.x and macOS-latest
- GitHub Check: Test on Node 22.x and macOS-latest
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Redteam
- GitHub Check: Test on Node 20.x and windows-latest
- GitHub Check: Test on Node 18.x and macOS-latest
- GitHub Check: Test on Node 18.x and ubuntu-latest
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Test on Node 18.x and windows-latest
- GitHub Check: webui tests
- GitHub Check: Style Check
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (6)
site/docs/configuration/guide.md (1)
336-380: Great section – clearly explains externaldefaultTestusageThe new example and explanation are concise and follow style guidelines. 👍
src/types/index.ts (2)
810-818: Well-implemented union type for defaultTest configuration.The implementation correctly supports both external file references (with proper
file://validation) and inline object configuration. The use ofTestCaseSchema.partial().omit({ description: true })appropriately prevents defaultTest from overriding individual test descriptions.
948-955: Consistent implementation in TestSuiteConfigSchema.The defaultTest property implementation matches the TestSuiteSchema, maintaining consistency across both schemas while supporting the new external file reference capability.
src/util/config/load.ts (2)
514-516: LGTM: Clear variable naming and extraction of raw defaultTest value.The code correctly extracts the raw defaultTest value from either fileConfig or defaultConfig, maintaining backwards compatibility.
541-541: LGTM: Correct usage of processed defaultTest.The processed defaultTest is correctly passed to
readTestwhich will handle the TestCase processing and validation.examples/assertions-generate/promptfooconfig.yaml (1)
9-10: Excellent demonstration of the new file:// syntax for defaultTest.The change effectively demonstrates the new capability to load defaultTest from external files. The comment is helpful for users understanding the feature, and the file:// prefix is correctly used as specified in the coding guidelines.
| ## Running the Example | ||
|
|
||
| ```bash | ||
| npx promptfoo@latest eval | ||
| ``` |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add the init command before the eval instruction
Most users clone examples with init; show it here for consistency with project guidelines.
## Running the Example
```bash
+npx promptfoo@latest init --example assertions-generate
+npx promptfoo@latest cd assertions-generate # if needed
npx promptfoo@latest eval
<details>
<summary>🤖 Prompt for AI Agents</summary>
In examples/assertions-generate/README.md around lines 63 to 67, the
instructions for running the example are missing the initialization step. Add
the command "npx promptfoo@latest init --example assertions-generate" before the
eval command, and optionally include "npx promptfoo@latest cd
assertions-generate" if changing directories is needed, to align with project
guidelines and ensure users properly set up the example before evaluation.
</details>
<!-- This is an auto-generated comment by CodeRabbit -->
<!-- fingerprinting:phantom:triton:cougar -->
| "anyOf": [ | ||
| { | ||
| "type": "string" | ||
| }, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add validation for file:// prefix in JSON schema.
The JSON schema allows any string for defaultTest, but the corresponding TypeScript schema validates that strings must start with file://. This inconsistency could allow invalid configurations to pass JSON schema validation but fail at runtime.
Consider updating the JSON schema to include pattern validation:
"anyOf": [
{
- "type": "string"
+ "type": "string",
+ "pattern": "^file://"
},📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "anyOf": [ | |
| { | |
| "type": "string" | |
| }, | |
| "anyOf": [ | |
| { | |
| "type": "string", | |
| "pattern": "^file://" | |
| }, |
🤖 Prompt for AI Agents
In site/static/config-schema.json around lines 898 to 901, the JSON schema for
defaultTest currently allows any string, but the TypeScript schema requires
strings to start with "file://". To fix this inconsistency, add a pattern
property to the JSON schema that enforces strings to begin with "file://". This
will ensure the JSON schema validation matches the runtime TypeScript
validation.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
⏩ No test scenarios generated (2d458f0) View output ↗ View check history
|
mldangelo
force-pushed
the
feat/external-defaulttest-config
branch
from
abfd9de to
9f8c372
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Description
This PR adds support for loading
defaultTestconfiguration from external files using thefile://syntax, similar to how prompts and other configuration files can be loaded.Motivation
Currently,
defaultTestmust be defined inline in the configuration file. This can lead to:Changes
defaultTestwithfile://prefix validationresolveConfigsto handle external file referencesassertions-generateexample to demonstrate the featureUsage Example
Instead of:
You can now use:
Testing
npm run localBreaking Changes
None - this is a backward-compatible addition.