Skip to content

Conversation

@steven10a
Copy link
Collaborator

@steven10a steven10a commented Nov 3, 2025

Fixes reported [BUG #29 ]

  • topical-alignment check hard codes temperature=0 which is not valid for gpt-5 series models and causes an error

The fix:

  • All LLM based checks should be using llm-base which properly handles the temperature param
  • Additionally noticed user-defined-llms was not properly using llm-base, fixed that
  • Updated tests for both guardrails

@steven10a steven10a requested review from Copilot and gabor-openai and removed request for Copilot November 3, 2025 15:24
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the user-defined LLM and topical alignment guardrails to use a common createLLMCheckFn factory function, reducing code duplication and improving maintainability.

  • Replaces manual LLM check implementations with calls to createLLMCheckFn from llm-base.ts
  • Renames exported check functions from userDefinedLLMCheck/topicalAlignmentCheck to userDefinedLLM/topicalAlignment
  • Rewrites tests to verify the refactored implementation with multiple GPT model variants

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
src/checks/user-defined-llm.ts Refactored to use createLLMCheckFn factory, reducing ~150 lines of duplicate LLM handling code and renaming the exported check function
src/checks/topical-alignment.ts Refactored to use createLLMCheckFn factory, reducing ~80 lines of duplicate LLM handling code and renaming the exported check function
src/tests/unit/checks/user-defined-llm.test.ts Updated tests to verify the refactored implementation with model-specific temperature tests for gpt-4, gpt-5, gpt-4o, and gpt-3.5-turbo
src/tests/unit/checks/topical-alignment.test.ts Updated tests to verify the refactored implementation with model-specific temperature tests for gpt-4, gpt-5, gpt-4o, and gpt-3.5-turbo

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Collaborator

@gabor-openai gabor-openai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TY

@gabor-openai gabor-openai merged commit 50df54b into main Nov 3, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants