feat: Image understanding for Google providers #4767
Conversation
📝 WalkthroughWalkthroughThe changes introduce new YAML configuration files for image captioning examples in both the Google AI Studio Gemini and Google Vertex directories, each specifying prompt templates, providers, and test cases using a sample image of a red panda. Corresponding README files for these examples are updated to include instructions for running the new image understanding evaluations and reference relevant documentation. In the Google AI Studio Tools example, the provider ID in a configuration file is updated to reflect a different Gemini model version. In the codebase, the Gemini provider utility is enhanced to process embedded base64-encoded images in prompt contents, adding helper functions for image detection and MIME type inference. No exported or public entity signatures are changed. Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
| @@ -455,9 +455,106 @@ export function geminiFormatAndSystemInstructions( | |||
| throw new Error(`Template error: system instruction defined in prompt and config.`); | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Because of images being passed as inline data I process them here and consider the context vars. If you have any other suggestions for the implementation let me know.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
examples/google-aistudio-gemini/promptfooconfig.image.yaml (2)
1-1: Add the schema header for IDE validation & consistencyAll example config files should start with the PromptFoo JSON-schema header so editors can lint/auto-complete the YAML.
Add it as the very first line:+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: 'Image captioning'
7-9: Optional: include a non-Google baseline provider for easier side-by-side comparisonThe guidelines encourage mixing providers so users can quickly gauge relative performance.
Consider adding, for instance,openai:gpt-4o-mini(text-only) or another vision-capable model if available.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (2)
examples/google-aistudio-gemini/assets/red-panda.jpgis excluded by!**/*.jpgexamples/google-vertex/assets/red-panda.jpgis excluded by!**/*.jpg
📒 Files selected for processing (6)
examples/google-aistudio-gemini/README.md(1 hunks)examples/google-aistudio-gemini/promptfooconfig.image.yaml(1 hunks)examples/google-aistudio-tools/promptfooconfig.search.yaml(1 hunks)examples/google-vertex/README.md(2 hunks)examples/google-vertex/promptfooconfig.image.yaml(1 hunks)src/providers/google/util.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
`examples/*/README.md`: The README.md must begin with the folder name as an H1 h...
examples/*/README.md: The README.md must begin with the folder name as an H1 heading
Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Include a comprehensive README.md that explains the purpose, prerequisites, instructions, and expected outputs for each example
Document any model-specific capabilities or limitations in example README files
Clearly list all required environment variables at the beginning of the README
For each environment variable, explain its purpose, how to obtain it, and any default values or constraints in the README
Include a sample '.env' file or instructions when multiple environment variables are needed
Document any required API keys or credentials in the README
Provide instructions for cleaning up resources after running the example in the README
When creating examples for specific providers, explain any provider-specific configuration in the README
Document required environment variables for provider-specific examples in the README
Include information about pricing or usage limits for provider-specific examples in the README
Highlight unique features or capabilities of providers in the README
Compare to similar providers where appropriate in the README
Make it clear which features each example demonstrates in the README
Include links to more comprehensive documentation where appropriate in the README
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/google-aistudio-gemini/README.mdexamples/google-vertex/README.md
`examples/**/*`: Structure examples consistently with common patterns across the...
examples/**/*: Structure examples consistently with common patterns across the project
Ensure all examples are functional and up-to-date
Keep examples as simple as possible while still demonstrating the concept
Follow the same code style guidelines as the main project in all example code and configuration
Include comments to explain non-obvious parts in example code and configuration
Use descriptive variable and function names in example code and configuration
Keep code DRY within reason; examples may duplicate code for clarity
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/google-aistudio-gemini/README.mdexamples/google-vertex/README.mdexamples/google-vertex/promptfooconfig.image.yamlexamples/google-aistudio-tools/promptfooconfig.search.yamlexamples/google-aistudio-gemini/promptfooconfig.image.yaml
`**/*.{ts,tsx}`: Use TypeScript with strict type checking.
**/*.{ts,tsx}: Use TypeScript with strict type checking.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/providers/google/util.ts
`**/*.{js,jsx,ts,tsx}`: Follow established import order with @trivago/prettier-p...
**/*.{js,jsx,ts,tsx}: Follow established import order with @trivago/prettier-plugin-sort-imports.
Use consistent curly braces for all control statements.
Prefer const over let; avoid var.
Use object shorthand syntax whenever possible.
Use async/await for asynchronous code.
Use consistent error handling with proper type checks.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/providers/google/util.ts
`**/*.{ts,tsx}`: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
**/*.{ts,tsx}: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
📄 Source: CodeRabbit Inference Engine (.cursor/rules/gh-cli-workflow.mdc)
List of files the instruction was applied to:
src/providers/google/util.ts
🧠 Learnings (6)
📓 Common learnings
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Update examples when APIs or dependencies change
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
examples/google-aistudio-gemini/README.md (14)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.694Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Make it clear which features each example demonstrates in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Document any model-specific capabilities or limitations in example README files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Include a comprehensive README.md that explains the purpose, prerequisites, instructions, and expected outputs for each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/python.mdc:0-0
Timestamp: 2025-06-23T12:49:38.363Z
Learning: For all Python code in the promptfoo project, use Python 3.9 or later and follow the Google Python Style Guide to ensure consistency and modern language features.
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: When developing or testing examples locally, use 'npm run local' commands instead of 'npx promptfoo@latest' to ensure local changes are tested
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include placeholder values for secrets/credentials in configuration files
examples/google-vertex/README.md (15)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Make it clear which features each example demonstrates in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Update examples when APIs or dependencies change
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Document any model-specific capabilities or limitations in example README files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/README.md : Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.694Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Ensure all configuration files pass YAML lint validation
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
examples/google-vertex/promptfooconfig.image.yaml (10)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
examples/google-aistudio-tools/promptfooconfig.search.yaml (5)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
examples/google-aistudio-gemini/promptfooconfig.image.yaml (10)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.317Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
🪛 GitHub Check: Style Check
src/providers/google/util.ts
[failure] 493-493:
'getMimeTypeFromBase64' was used before it was defined
[failure] 492-492:
'isValidBase64Image' was used before it was defined
[failure] 475-475:
'isValidBase64Image' was used before it was defined
[failure] 459-459:
'processImagesInContents' was used before it was defined
🪛 GitHub Actions: CI
src/providers/google/util.ts
[error] 459-459: ESLint: 'processImagesInContents' was used before it was defined. (@typescript-eslint/no-use-before-define)
🔇 Additional comments (7)
src/providers/google/util.ts (1)
464-521: Excellent implementation for base64 image processing.The image processing logic is well-designed with proper base64 validation, MIME type detection, and integration with the existing Gemini format structure. The approach of mapping base64 data to variable names and processing line-by-line is sound.
examples/google-aistudio-gemini/README.md (1)
57-62: Good documentation of new image understanding capability.The addition clearly separates the basic example from the image understanding example, making it easy for users to understand the different capabilities available.
examples/google-aistudio-tools/promptfooconfig.search.yaml (1)
18-18: Good update to stable model version.The change from the experimental version to the stable
google:gemini-2.5-proaligns with best practices for using the latest stable model versions in examples.examples/google-vertex/README.md (2)
71-73: Consistent documentation enhancement.The addition of the image understanding example follows the same pattern as the AI Studio example, maintaining consistency across the documentation.
88-88: Valuable addition of Google documentation link.The link to Google's Image Understanding documentation provides users with additional context and technical details.
examples/google-vertex/promptfooconfig.image.yaml (1)
8-15: Well-structured image understanding configuration.The configuration demonstrates good practices:
- Uses the latest Gemini 2.5 Flash model that supports vision capabilities
- Simple, clear prompt structure for image understanding
- Effective test case that validates the core functionality
examples/google-aistudio-gemini/promptfooconfig.image.yaml (1)
10-15: Asset verified: no changes requiredThe file
examples/google-aistudio-gemini/assets/red-panda.jpgexists, so thefile://assets/red-panda.jpgreference inpromptfooconfig.image.yamlis valid.
| newParts.push({ | ||
| inlineData: { | ||
| mimeType, | ||
| data: trimmedLine, | ||
| }, | ||
| }); |
There was a problem hiding this comment.
Fix property naming to match schema.
The code uses camelCase property names (inlineData, mimeType) but the Part schema expects snake_case (inline_data, mime_type).
Update the property names to match the schema:
- newParts.push({
- inlineData: {
- mimeType,
- data: trimmedLine,
- },
- });
+ newParts.push({
+ inline_data: {
+ mime_type: mimeType,
+ data: trimmedLine,
+ },
+ });📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| newParts.push({ | |
| inlineData: { | |
| mimeType, | |
| data: trimmedLine, | |
| }, | |
| }); | |
| newParts.push({ | |
| inline_data: { | |
| mime_type: mimeType, | |
| data: trimmedLine, | |
| }, | |
| }); |
🤖 Prompt for AI Agents
In src/providers/google/util.ts around lines 495 to 500, the object properties
use camelCase names inlineData and mimeType, but the Part schema requires
snake_case names inline_data and mime_type. Rename the properties inlineData to
inline_data and mimeType to mime_type to match the schema exactly.
| @@ -15,7 +15,7 @@ providers: | |||
| - googleSearch: {} | |||
|
|
|||
| # Alternate version with thinking capabilities | |||
| - id: google:gemini-2.5-pro-exp-03-25 | |||
| - id: google:gemini-2.5-pro | |||
There was a problem hiding this comment.
This example simply errored with this model is not supported for API anymore so I added a more generic model to keep it easily up to date.
There was a problem hiding this comment.
This is great. Thank you @adelmuursepp. Only nits are that it can use some tests and you could update google.md and vertex.md on the site
|
Relates to #4849 |
Core Changes
src/providers/google/util.ts.Before (Broken Format). What promptfoo sent to Google API:
{ "contents": [ { "parts": [ { "text": "/9j/4AAAfter adding the recognition of image files and formatting:
{ "contents": [ { "parts": [ { "inlineData": { "mimeType": "image/jpeg", "data": "/9j/4ANow Google providers are adding the image data inline (as explained in the Google API docs). This means image passing as inline data will work for AI studio and Vertex Google provider. Google Live will inherit same image processing improvements if this becomes available for Live API.
Image path is shown in metadata, however the prompt looks quite ugly with the base64 data.
Example UI:
New Examples Added
panda image
Further Improvements
Test and showcase object recognition, image segmentation and with files available from URLs.