Document structured choice UI; ban free-text multi-choice walls - #7
Conversation
When agent hosts expose pickers, free-text multi-option menus are a choice wall (contract R2/R4). Surface pattern, known-bad/known-good fixtures, and rule heuristics make the anti-pattern falsifiable. Closes #5
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fbf13c6849
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| numbered_opts = len(re.findall(r"^\s*\d+[.)]\s+\S+", agent, re.M)) | ||
| free_text_pick = bool( | ||
| re.search( | ||
| r"\b(pick|choose|reply with|type one of|which (would you like|do you want))\b", | ||
| agent_l, |
There was a problem hiding this comment.
Because numbered_opts counts every numbered line in the whole agent block and free_text_pick is set by any choose, a valid explicit human act like “Choose the account you want” followed by a 3-step browser login checklist is scored as an R2/R4 choice wall even though no multi-option menu was offered. This can make known-good setup or login fixtures fail; tie the choice prompt to choosing among those numbered items, or distinguish procedural steps from option lists.
Useful? React with 👍 / 👎.
| if vague_physical or (batched and re.search(r"\bbrowser\b|\bplug\b|\bhold\b", agent_l)): | ||
| p["R4"] = False | ||
| reasons.append("R4: vague or batched human act") | ||
| if free_text_pick and numbered_opts >= 3 and not structured_ui: |
There was a problem hiding this comment.
This R4 check reuses the numbered_opts >= 3 threshold, so a yes/no gate such as “choose 1 to delete / 2 to cancel” leaves R4 passing. The new contract and surface text explicitly call out yes/no gates as cases that should use the structured picker, so known-bad irreversible or scary two-choice gates can pass the eval unless R4 has a separate gate pattern or lower threshold.
Useful? React with 👍 / 👎.
Resolve eval/README.md fixture table: keep choice-wall/structured-choice from #7 and ask/step operator-gate fixtures from this branch.
1 participant
Summary
choice-walland known-goodstructured-choicefixtures; R2/R4 heuristics score free-text multi-option menus.Closes #5
Test plan
pytest -qbedside eval(all fixtures match expect)