chore(cli): AWS SageMaker AI provider cleanup #4667
Conversation
|
⏩ No test execution environment matched (8c908af) View output ↗ Tip New to Tusk? Learn more here. View check history
|
TestGru AssignmentSummary
Tip You can |
will-holley
changed the title
📝 WalkthroughWalkthroughThis change set updates documentation and code related to Amazon SageMaker integration. All documentation references to "Amazon SageMaker" are revised to "Amazon SageMaker AI" for consistency. The example configuration file for Anthropic Claude models on SageMaker is deleted, and all code supporting the 'anthropic' model type is removed from the SageMaker provider implementation, including prompt formatting and response parsing logic specific to Anthropic models. Additionally, minor simplifications are made to cache retrieval and storage logic in the provider code. No changes are made to other model types, technical instructions, or remaining configuration examples. Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! ✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
examples/amazon-sagemaker/README.md (1)
23-25: Fix grammatical error: Missing possessive apostropheThere's a missing possessive apostrophe in the prerequisites section.
-2. Deployed SageMaker AI endpoints with your models +2. Deployed SageMaker AI endpoints with your models'Actually, let me reconsider this. The sentence should read:
-2. Deployed SageMaker AI endpoints with your models +2. SageMaker AI endpoints deployed with your modelsOr more clearly:
-2. Deployed SageMaker AI endpoints with your models +2. SageMaker AI endpoints with your deployed models
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
examples/amazon-sagemaker/README.md(3 hunks)examples/amazon-sagemaker/promptfooconfig.anthropic.yaml(0 hunks)site/docs/providers/sagemaker.md(1 hunks)src/providers/sagemaker.ts(4 hunks)
💤 Files with no reviewable changes (1)
- examples/amazon-sagemaker/promptfooconfig.anthropic.yaml
🧰 Additional context used
📓 Path-based instructions (6)
`examples/*/README.md`: The README.md must begin with the folder name as an H1 h...
examples/*/README.md: The README.md must begin with the folder name as an H1 heading
Every example README must include instructions on how to run it with 'npx promptfoo@latest init --example example-name'
Include a comprehensive README.md that explains the purpose, prerequisites, instructions, and expected outputs for each example
Document any model-specific capabilities or limitations in example README files
Clearly list all required environment variables at the beginning of the README
For each environment variable, explain its purpose, how to obtain it, and any default values or constraints in the README
Include a sample '.env' file or instructions when multiple environment variables are needed
Document any required API keys or credentials in the README
Provide instructions for cleaning up resources after running the example in the README
When creating examples for specific providers, explain any provider-specific configuration in the README
Document required environment variables for provider-specific examples in the README
Include information about pricing or usage limits for provider-specific examples in the README
Highlight unique features or capabilities of providers in the README
Compare to similar providers where appropriate in the README
Make it clear which features each example demonstrates in the README
Include links to more comprehensive documentation where appropriate in the README
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/amazon-sagemaker/README.md
`examples/**/*`: Structure examples consistently with common patterns across the...
examples/**/*: Structure examples consistently with common patterns across the project
Ensure all examples are functional and up-to-date
Keep examples as simple as possible while still demonstrating the concept
Follow the same code style guidelines as the main project in all example code and configuration
Include comments to explain non-obvious parts in example code and configuration
Use descriptive variable and function names in example code and configuration
Keep code DRY within reason; examples may duplicate code for clarity
📄 Source: CodeRabbit Inference Engine (.cursor/rules/examples.mdc)
List of files the instruction was applied to:
examples/amazon-sagemaker/README.md
`site/{docs,blog,src/pages}/**/*.{md,mdx}`: Prioritize minimal edits when updati...
site/{docs,blog,src/pages}/**/*.{md,mdx}: Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally
Structure content to reveal information progressively: begin with essential actions and information, then provide deeper context as necessary; organize information from most important to least important
Use action-oriented language: clearly outline actionable steps, use concise and direct language, prefer active voice over passive voice, and use imperative mood for instructions
Use 'eval' instead of 'evaluation' in all documentation; when referring to command line usage, use 'npx promptfoo eval' rather than 'npx promptfoo evaluation'; maintain consistency with this terminology across all examples, code blocks, and explanations
Use 'Promptfoo' (capitalized) at the beginning of sentences or in headings, and 'promptfoo' (lowercase) in code examples, terminal commands, or when referring to the package name; be consistent with the chosen capitalization within each document or section
Each markdown documentation page must include required front matter fields: 'title' (page title) and 'description' (concise summary, ideally 150-160 characters)
Optional front matter fields for markdown documentation pages: 'image', 'keywords', and 'sidebar_position' can be included to enhance SEO and navigation
Keep titles under 60 characters for optimal display in search results; write description meta tags between 150-160 characters; include relevant keywords naturally in both title and description; use unique titles and descriptions for each page; for image thumbnails, use 1200×630 pixels for optimal social media display
Only add a title attribute to code blocks that represent complete, runnable files; do not add titles to code fragments, partial examples, or snippets that aren't meant to be used as standalone files; applies to all code blocks regardless of language
Use special comment directives to highlight specific lines in code blocks: 'highlight-next-line', 'highlight-start'/'highlight-end', or line numbers in curly braces; never remove existing highlight directives when editing a document; preserve the original author's intent by maintaining their highlighting
Always specify the language for syntax highlighting in code blocks; use meaningful titles for code blocks when appropriate; keep code examples concise and focused; ensure code examples are correct and up-to-date; include comments in code examples where helpful
Use Docusaurus admonitions (note, tip, info, warning, danger) by wrapping content with three colons ':::' followed by the type; always include empty lines before and after the content inside admonitions, especially when using Prettier; you can include Markdown formatting within admonitions
When using Prettier to format Markdown files, always include empty lines around admonition content to prevent Prettier from reformatting admonitions to invalid syntax
Search the codebase to provide accurate and relevant backlinks within the documentation; include references to specific files, functions, or concepts; ensure backlinks are relevant to the current section
End each documentation section with a concise 'See Also' or 'Related Concepts' heading; link to related documentation or concepts; format the 'See Also' section consistently as a markdown list of links
Use clear, concise language; maintain a consistent tone throughout the documentation; write for an international audience (avoid idioms, colloquialisms, and culturally specific references); spell out acronyms on first use, followed by the acronym in parentheses
Include practical examples where appropriate; make examples realistic and applicable to common use cases; when showing configuration examples, include all necessary fields; demonstrate both basic and advanced usage when relevant
Use diagrams, screenshots, or illustrations to explain complex concepts; ensure images have appropriate alt text for accessibility; maintain a consistent visual style across all documentation; optimize images for web display
📄 Source: CodeRabbit Inference Engine (.cursor/rules/docusaurus.mdc)
List of files the instruction was applied to:
site/docs/providers/sagemaker.md
`**/*.{ts,tsx}`: Use TypeScript with strict type checking.
**/*.{ts,tsx}: Use TypeScript with strict type checking.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/providers/sagemaker.ts
`**/*.{js,jsx,ts,tsx}`: Follow established import order with @trivago/prettier-p...
**/*.{js,jsx,ts,tsx}: Follow established import order with @trivago/prettier-plugin-sort-imports.
Use consistent curly braces for all control statements.
Prefer const over let; avoid var.
Use object shorthand syntax whenever possible.
Use async/await for asynchronous code.
Use consistent error handling with proper type checks.
📄 Source: CodeRabbit Inference Engine (CLAUDE.md)
List of files the instruction was applied to:
src/providers/sagemaker.ts
`**/*.{ts,tsx}`: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
**/*.{ts,tsx}: Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
📄 Source: CodeRabbit Inference Engine (.cursor/rules/gh-cli-workflow.mdc)
List of files the instruction was applied to:
src/providers/sagemaker.ts
🧠 Learnings (4)
📓 Common learnings
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.304Z
Learning: Update examples when APIs or dependencies change
examples/amazon-sagemaker/README.md (14)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.304Z
Learning: Update examples when APIs or dependencies change
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/docusaurus.mdc:0-0
Timestamp: 2025-06-30T13:43:26.684Z
Learning: Applies to site/{docs,blog,src/pages}/**/*.{md,mdx} : Prioritize minimal edits when updating existing documentation; avoid creating entirely new sections or rewriting substantial portions; focus edits on improving grammar, spelling, clarity, fixing typos, and structural improvements where needed; do not modify existing headings (h1, h2, h3, etc.) as they are often linked externally
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Compare to similar providers where appropriate in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest model versions available in 2025 for model selection in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a working 'promptfooconfig.yaml' (or equivalent) file in each example
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/python.mdc:0-0
Timestamp: 2025-06-23T12:49:38.363Z
Learning: When implementing custom providers, prompts, or asserts in Python for the promptfoo project, follow the established API patterns to ensure compatibility and maintainability.
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Update model versions when new ones become available in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Format configuration files consistently
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Follow the specified field order in all configuration files: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always include the YAML schema reference at the top of configuration files: '# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json'
site/docs/providers/sagemaker.md (5)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/README.md : Compare to similar providers where appropriate in the README
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Include a mix of providers when comparing model performance in examples
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : Always use the latest available model versions for each provider in configuration files
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : When demonstrating specialized capabilities (vision, audio, etc.), use models that support those features
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/python.mdc:0-0
Timestamp: 2025-06-23T12:49:38.363Z
Learning: When implementing custom providers, prompts, or asserts in Python for the promptfoo project, follow the established API patterns to ensure compatibility and maintainability.
src/providers/sagemaker.ts (1)
Learnt from: CR
PR: promptfoo/promptfoo#0
File: .cursor/rules/examples.mdc:0-0
Timestamp: 2025-06-30T13:43:50.303Z
Learning: Applies to examples/*/promptfooconfig.yaml : For trivial test cases, make them quirky and fun to increase engagement
🪛 LanguageTool
examples/amazon-sagemaker/README.md
[uncategorized] ~24-~24: It seems likely that a singular genitive (’s) apostrophe is missing.
Context: ...ployed SageMaker AI endpoints with your models 3. AWS credentials configured locally 4...
(AI_HYDRA_LEO_APOSTROPHE_S_XS)
🪛 GitHub Actions: CI
src/providers/sagemaker.ts
[error] 663-663: TypeError: cache.get is not a function. This error occurs repeatedly in multiple tests indicating that the 'cache' object does not have a 'get' method implemented or accessible.
🔇 Additional comments (10)
examples/amazon-sagemaker/README.md (4)
1-1: LGTM: Consistent rebranding to Amazon SageMaker AIThe title update correctly reflects the rebranding from "Amazon SageMaker" to "Amazon SageMaker AI" and maintains consistency with the PR objectives.
3-3: LGTM: Description updated for SageMaker AIThe description correctly updates the terminology to reference "Amazon SageMaker AI endpoints" which aligns with the official AWS terminology as mentioned in the PR objectives.
15-20: LGTM: Purpose section updated consistentlyAll references in the purpose section have been correctly updated to use "SageMaker AI" terminology. However, I notice the reference to Anthropic models has been removed from the example configurations list, which aligns with the removal of Anthropic support.
66-66: LGTM: Testing section updated consistentlyThe testing section correctly references "SageMaker AI endpoint" maintaining consistency with the rebranding effort.
site/docs/providers/sagemaker.md (4)
2-5: LGTM: Front matter updated for SageMaker AI rebrandingThe sidebar label, title, and description have been consistently updated to reflect "Amazon SageMaker AI" terminology. The metadata changes improve clarity about the provider's specific focus on SageMaker AI endpoints.
7-7: LGTM: Main heading updated consistentlyThe heading correctly reflects the rebranding to "Amazon SageMaker AI" and maintains consistency with the front matter changes.
9-9: LGTM: Provider description updated for clarityThe description correctly specifies "Amazon SageMaker AI endpoints" and maintains the technical accuracy while clarifying the scope to SageMaker AI specifically, distinguishing it from the broader SageMaker platform.
13-13: LGTM: Setup instructions updated consistentlyThe setup instructions correctly reference "SageMaker AI endpoints" maintaining consistency with the overall rebranding effort throughout the documentation.
src/providers/sagemaker.ts (2)
51-53: LGTM: Model type comments and Anthropic removalThe added comments provide helpful clarification about model types, and the removal of 'anthropic' from the modelType union aligns with the PR objectives to remove Anthropic support from the SageMaker provider.
399-399: LGTM: Anthropic removed from supported model typesThe removal of 'anthropic' from the SAGEMAKER_MODEL_TYPES array is consistent with the removal from the TypeScript interface and aligns with the PR objectives to clean up Anthropic references.
| const cache = (await getCache) | ||
| ? await getCache() | ||
| : await import('../cache').then((m) => m.getCache()); | ||
| const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache()); |
There was a problem hiding this comment.
Fix duplicate cache handling issue
Same cache handling issue as line 660 - the conditional logic is incorrect and causing runtime errors.
- const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache());
+ const cache = getCache();This should be consistent with the fix applied to line 660.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache()); | |
| const cache = getCache(); |
🤖 Prompt for AI Agents
In src/providers/sagemaker.ts at line 761, the cache handling logic uses a
conditional that incorrectly mixes synchronous and asynchronous calls, causing
runtime errors. Refactor this line to consistently handle the cache retrieval by
checking if getCache is defined and then either calling it synchronously or
importing and awaiting the cache module asynchronously, matching the fix applied
at line 660 to avoid duplicate cache handling issues.
| const cache = (await getCache) | ||
| ? await getCache() | ||
| : await import('../cache').then((m) => m.getCache()); | ||
| const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache()); |
There was a problem hiding this comment.
Fix cache handling logic causing runtime errors
The current cache handling logic is problematic and causing the pipeline failure "cache.get is not a function". The conditional logic is incorrect.
- const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache());
+ const cache = getCache();The issue is that getCache is already imported and available, so the conditional check getCache ? is unnecessary and creates confusion. The function should be called directly.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const cache = getCache ? getCache() : await import('../cache').then((m) => m.getCache()); | |
| const cache = getCache(); |
🤖 Prompt for AI Agents
In src/providers/sagemaker.ts at line 660, the cache handling logic incorrectly
checks if getCache exists before calling it, causing runtime errors. Since
getCache is already imported and available, remove the conditional check and
call getCache() directly to obtain the cache instance.
Uh oh!
There was an error while loading. Please reload this page.