Skip to content

[No QA] Migrate ProposalPolice to Responses + Conversations API - #97199

Draft
roryabraham wants to merge 15 commits into
mainfrom
Rory-ProposalPoliceResponsesConversations
Draft

[No QA] Migrate ProposalPolice to Responses + Conversations API#97199
roryabraham wants to merge 15 commits into
mainfrom
Rory-ProposalPoliceResponsesConversations

Conversation

@roryabraham

@roryabraham roryabraham commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Explanation of Change

ProposalPolice currently runs on the OpenAI Assistants API: the system prompt is configured manually in the OpenAI dashboard (not checked into code), and duplicate-proposal detection makes one Assistants thread/run per prior proposal on the issue (each with up to 90s of polling), since Assistants are being deprecated in favor of the Responses API.

This migrates ProposalPolice to:

  • The Responses API (OpenAIUtils.promptResponses) on gpt-5.6-luna (replacing the Assistant's GPT-4o) for the template-check and edit-check calls — no more thread/run polling.
  • A persistent, per-issue OpenAI Conversation for duplicate-check, so the loop of one Assistants call per prior proposal becomes a single Responses call against the issue's Conversation history. The Conversation ID is tracked via a hidden marker in a bot comment on the issue.
  • A checked-in, composed system prompt (prompts/proposalPolice/*) split into small, focused instruction fragments per call type (template-check / edit-check / duplicate-check), replacing the single dashboard-configured prompt blob — each call only gets the instructions it actually needs.
  • JSON-schema structured outputs for each call's response shape, replacing manual "respond with JSON" prompt instructions.

Also extracted ProposalUtils (proposal/bot-detection helpers) as a shared, independently-testable utility.

No tests previously existed for ProposalPolice or OpenAIUtils; this PR adds coverage for all of the new logic (OpenAIUtilsTest, ProposalPoliceConversationTest, ProposalPoliceInputTest, proposalPoliceCommentTest).

Fixed Issues

$ #72725
PROPOSAL:

Tests

  • Verify that no errors appear in the JS console

This is a GitHub Actions bot with no UI, so "manual testing" means exercising the compiled action directly:

  1. Run npm run test -- OpenAIUtilsTest ProposalPoliceConversationTest ProposalPoliceInputTest proposalPoliceCommentTest GithubUtilsTest and confirm all pass.
  2. Run npm run typecheck-tsgo and npm run lint-changed and confirm both are clean.
  3. Run npm run gh-actions-build and confirm .github/actions/javascript/proposalPoliceComment/index.js is rebuilt with no further diff.
  4. On a disposable "Help Wanted" test issue, post a valid proposal (all mandatory sections filled in) and confirm no comment is posted (NO_ACTION).
  5. On the same issue, post a proposal missing a mandatory section and confirm ProposalPolice posts the "please follow the template" nag comment, tagging the correct author.
  6. Edit a valid proposal to substantially change its root cause/solution and confirm ProposalPolice prepends the "Edited by proposal-police" notice to the comment.
  7. From a second account, post a proposal whose root cause/solution closely matches an earlier one on the same issue, and confirm ProposalPolice withdraws it and posts a notice linking to the original.
  8. Confirm a tracking comment (containing a hidden Conversation-ID marker) appears on the issue after step 4, and that it's reused (not duplicated) across steps 5-7.

Steps 4-8 (live end-to-end verification against real GitHub/OpenAI API calls) have not been run yet as of opening this draft — only steps 1-3 have been verified locally. Marking as draft until 4-8 are confirmed on a real test issue.

Offline tests

None — this is a GitHub Actions bot with no client-side/offline behavior.

QA Steps

None — see [No QA] in the PR title. This is a backend-only GitHub Actions change with no user-facing app behavior.

  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android: Native
    • Android: mWeb Chrome
    • iOS: Native
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I followed proper code patterns (see Reviewing the code)
    • I verified that comments were added to code that is not self explanatory
    • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
    • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I followed the guidelines as stated in the Review Guidelines
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • If a new CSS style is added I verified that:
    • A similar style doesn't already exist
    • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))
  • If new assets were added or existing ones were modified, I verified that:
    • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
    • The assets load correctly across all supported platforms.
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Not applicable — this is a backend-only GitHub Actions change with no UI.

Android: Native

N/A

Android: mWeb Chrome

N/A

iOS: Native

N/A

iOS: mWeb Safari

N/A

MacOS: Chrome / Safari

N/A

roryabraham added 13 commits July 27, 2026 23:21
@roryabraham
Add OpenAI Conversations support to OpenAIUtils, drop Assistants API
2706ab3 
Replaces the deprecated Assistants thread/run polling helpers
(promptAssistant, parseAssistantResponse) with createConversation,
addConversationItems, and a generic parseJSONResponse<T>, so callers
can use the Responses API with persistent Conversations instead.
@roryabraham
Extract ProposalUtils and add GithubUtils.pinIssue
75fc837 
ProposalUtils holds the proposal/bot-detection helpers previously
private to proposalPoliceComment.ts, so the new conversation-tracking
utils can share them without a reverse import.

GithubUtils.pinIssue is a best-effort GraphQL pin (the REST API has no
equivalent) used to surface ProposalPolice's per-issue duplicate-check
tracking comment; failures are swallowed since pinning is a convenience,
not something the caller depends on.
@roryabraham
Replace ProposalPolice's prompt blob with composed instruction fragments
77e7871 
Splits the dashboard-configured Assistant prompt into small, focused
fragments (template definition, template/edit examples, decision tree,
bot actions, duplicate detection) with per-call-type assemblers, so
each Responses API call only gets the instructions it actually needs
instead of the whole prompt every time. Also drops a stale "re-state
the problem" section from the examples that no longer matches the
proposal template.

Adds JSON-schema definitions and type guards for the three response
shapes (template-check, edit-check, duplicate-check), replacing the
"respond with JSON" instructions previously baked into the prompt text.
@roryabraham
Add ProposalPoliceConversation utils for per-issue duplicate tracking
394de19 
Pure, independently-testable helpers for the duplicate-check
Conversation flow: finding a tracked Conversation ID from a hidden
marker on a bot-authored comment, building the tracking comment body,
seeding items for prior proposals, and chunking items to OpenAI's
20-per-call Conversation limit.
@roryabraham
Migrate ProposalPolice action off the Assistants API
f556c62 
Swaps promptAssistant for promptResponses on gpt-5.6-luna (replacing
the Assistant's GPT-4o) for template-check and edit-check, and
replaces the per-prior-proposal Assistants loop with a single
Responses call against a persistent per-issue Conversation for
duplicate-check.

Removes the now-unused PROPOSAL_POLICE_ASSISTANT_ID input/secret, and
exports `run` (guarded behind a JEST_WORKER_ID check on the
auto-invocation) so it can be unit tested directly.

Fixes #72725
@roryabraham
Add tests for the ProposalPolice Responses/Conversations migration
a401c23 
Covers OpenAIUtils (promptResponses, createConversation,
addConversationItems, parseJSONResponse), the ProposalPoliceConversation
tracking helpers, and proposalPoliceComment's run() end-to-end
(NO_ACTION, ACTION_REQUIRED, ACTION_EDIT, duplicate withdrawal,
bot-author skip, and the Conversation create/reuse flow).

No tests previously existed for either of these.
@roryabraham
Rebuild compiled GitHub Action bundles
3bac9ac 
Regenerates every action's ncc bundle via npm run gh-actions-build.
Most of these only pick up the new GithubUtils.pinIssue method (a
shared lib bundled into every action); proposalPoliceComment's bundle
reflects its full migration off the Assistants API.
@roryabraham
Merge branch 'main' into Rory-ProposalPoliceResponsesConversations
75375ef
@roryabraham
Address branch-reviewer feedback on duplicate detection
95a12aa 
- Require the model's action to be ACTION_HIDE_DUPLICATE, not just a
  high similarity score, before withdrawing a proposal as a duplicate
  (guards against the two fields disagreeing).
- Skip the duplicate-check Responses call entirely when an issue has
  no prior proposals to compare against, instead of always spending an
  API call on a comparison that can't find anything.
- Add a concurrency group (scoped by issue number) to the workflow so
  two comments posted close together on the same issue can't each
  create their own tracking Conversation.
- Add tests for both behavior changes.
@roryabraham
Fix: record a proposal directly when its duplicate-check call is skipped
cd373e2 
Round-2 branch-reviewer catch: the duplicate-check Responses call was
the only mechanism appending items to a Conversation (via its
`conversation` param's auto-append behavior), so skipping that call
for an issue's first proposal (introduced in the previous commit) also
skipped ever recording it - permanently hiding it from every future
duplicate check on that issue. Now the proposal is recorded directly
via addConversationItems when the call is skipped.

Adds a regression test that runs the action twice in sequence (first
proposal, then a near-duplicate second one) to prove the first
proposal is actually comparable.
@roryabraham
Address round-3 branch-reviewer feedback
e34ba16 
- Post the tracking comment (and pin) immediately after creating a
  Conversation, before sending any remaining seed-item batches, so a
  failure mid-seeding can't leave the Conversation permanently
  untracked and fragment duplicate-detection history across issues
  with a large pre-existing proposal backlog.
- Validate the model-reported duplicateCommentId actually matches a
  real proposal comment before using it to build the withdrawal
  notice's link.
- Escape angle brackets in untrusted comment/proposal text before
  interpolating it into our XML-style wrapper tags, so a comment
  containing a literal closing tag can't be mistaken by the model for
  the end of our own wrapper.
- Switch the auto-invocation guard from checking JEST_WORKER_ID to the
  require.main === module pattern already used by every other action
  in .github/actions/javascript/*.
- Add tests: multi-batch seeding (>20 prior proposals) with an
  assertion on tracking-comment-before-remaining-seed ordering, and
  escaping/tagging coverage for all four prompt input builders.
@roryabraham
Address round-4 branch-reviewer feedback
f47d9e7 
- Exclude the new proposal's own comment ID from the duplicate-check
  originalProposal lookup, guarding against a model self-match
  hallucination linking the withdrawal notice to itself.
- Add tests for GithubUtils.pinIssue (correct GraphQL call, and that
  errors are swallowed rather than thrown), the one piece of new logic
  from this migration that didn't yet have coverage.
@roryabraham
Add test coverage for the PROPOSAL_POLICE_MODEL constant
06d99d2 
Round-5 branch-reviewer note: no test asserted the actual model value
passed to promptResponses, so a future accidental edit to
PROPOSAL_POLICE_MODEL wouldn't be caught by the suite. Exports the
constant and asserts it's used for all three call types (duplicate,
template, and edit check).

(The model ID itself, gpt-5.6-luna, was independently confirmed to be
a real, GA OpenAI model as of 2026-07-09 before this migration began.)
@roryabraham
Remove GithubUtils.pinIssue
3859dec 
Pinning is a scarce, repo-wide resource (max 3 pinned issues for the
entire repo), used by maintainers for things like contributing guides
or roadmap items. ProposalPolice would have attempted to consume one
of those slots on the first proposal of every "Help Wanted" issue,
which could fire constantly on an active repo and conflict with real
pins - for zero functional benefit, since the hidden marker text in
the tracking comment (not pin status) is the actual mechanism used to
find a tracked Conversation.

Reverts the corresponding +1 eslint-seatbelt allowance bump for
GithubUtilsTest.ts back to its original value now that the pinIssue
test (and its one unsafe-type-assertion) is gone too.
@roryabraham roryabraham mentioned this pull request Jul 28, 2026
Open
@codecov

codecov Bot commented Jul 28, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
see 20 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@roryabraham roryabraham

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@roryabraham