Skip to content

fix: code scanning alerts 1-3 in promptfoo verification flow#101

Draft
jbeckwith-oai wants to merge 3 commits intomainfrom
codex/fix-codeql-alert-1-promptfoo-eval
Draft

fix: code scanning alerts 1-3 in promptfoo verification flow#101
jbeckwith-oai wants to merge 3 commits intomainfrom
codex/fix-codeql-alert-1-promptfoo-eval

Conversation

@jbeckwith-oai
Copy link
Copy Markdown
Contributor

@jbeckwith-oai jbeckwith-oai commented Apr 13, 2026

Summary

  • separate user-facing generated config paths from the stable verification path
  • keep custom config filenames while also writing a stable promptfooconfig.yaml alias
  • run promptfoo eval via execFileSync with explicit argv instead of a shell command
  • preserve the existing verify timeout and diagnostic parsing
  • add regression tests for output-dir handling, custom filenames, and eval invocation

Root Cause

CodeQL split the promptfoo verification issue into three linked dataflow alerts:

  • alert 1: the verify path flowed into the later promptfoo eval shell command
  • alert 2: outputDir flowed into the generated config path
  • alert 3: filename flowed into the generated config path

This PR collapses that stacked series into one change set that fixes the full path from config generation through verification.

Validation

  • npm test -- src/agent/loop-eval.test.ts src/generator/config-filename.test.ts src/generator/config-outputdir.test.ts
  • npm run build

@jbeckwith-oai jbeckwith-oai changed the title [codex] Fix code scanning alert 1 in promptfoo eval execution [codex] Fix code scanning alerts 1-3 in promptfoo verification flow Apr 13, 2026
@jbeckwith-oai jbeckwith-oai changed the base branch from codex/fix-codeql-alert-3-config-filename to main April 13, 2026 20:33
@jbeckwith-oai jbeckwith-oai changed the title [codex] Fix code scanning alerts 1-3 in promptfoo verification flow fix: code scanning alerts 1-3 in promptfoo verification flow Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant