Skip to content

[Infra] Check for overfit in the scenario prompt as well#218

Merged
JanKrivanek merged 3 commits intomainfrom
dev/jankrivanek/overfit-fix
Mar 5, 2026
Merged

[Infra] Check for overfit in the scenario prompt as well#218
JanKrivanek merged 3 commits intomainfrom
dev/jankrivanek/overfit-fix

Conversation

@JanKrivanek
Copy link
Member

Motivation

Revealing the need to use skill (specific on or in general) in scenario prompt is a sign of overfitting as well.
Previously we've been checking anly the rubric graders

@JanKrivanek JanKrivanek requested a review from ViktorHofer as a code owner March 5, 2026 15:52
Copilot AI review requested due to automatic review settings March 5, 2026 15:52
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds prompt-level overfitting detection to the skill validator so scenario prompts that explicitly reference (or instruct using) a skill are treated as overfitting signals, alongside existing rubric/assertion checks.

Changes:

  • Extend overfitting schema/model to include prompt_assessments and incorporate them into score computation.
  • Add deterministic prompt scanning (DetectPromptOverfitting) and merge it with LLM-provided prompt assessments.
  • Update console reporting, dashboard data generation, and expand test coverage for prompt overfitting.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
eng/skill-validator/src/Services/OverfittingJudge.cs Adds deterministic prompt checks, parses/merges prompt_assessments, and updates scoring + LLM prompts/schema.
eng/skill-validator/src/Models/Models.cs Introduces PromptOverfitAssessment and adds PromptAssessments to OverfittingResult.
eng/skill-validator/src/Services/Reporter.cs Prints prompt-level overfitting signals in console output for moderate/high results.
eng/dashboard/generate-benchmark-data.ps1 Flags scenario overfitting when prompt assessments exist (supports new schema).
eng/skill-validator/tests/OverfittingJudgeTests.cs Adds coverage for prompt detection, parsing, and score effects; updates existing constructions for new result shape.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@JanKrivanek JanKrivanek merged commit 62d65dc into main Mar 5, 2026
4 of 5 checks passed
@JanKrivanek JanKrivanek deleted the dev/jankrivanek/overfit-fix branch March 5, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants