Repository navigation
Using Grilling between two agent sessions: a PRD review workflow and its limitations #1433
elfish2025-crypto
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Recently, while designing a plugin called SimGame 3D Studio, we tried using grilling to support an ongoing cycle of questions, responses, and revisions between two independent Codex sessions working from the same product requirements document (PRD).
I was responsible for product direction and key tradeoffs. One session handled design and writing; the other independently reviewed the document from a user's perspective. The experience suggested that Grilling can be useful for clarifying designs between agents. It also revealed a blind spot worth sharing.
We divided the responsibilities as follows:
The sessions exchanged questions and responses through cross-session messages and by reading conversation records. The PRD file was their shared reference. I did not need to relay each question and answer manually.
The loop was:
Reviewer asks questions → author answers or revises the PRD → reviewer rereads the new version → checks whether the issues are resolved → proceeds to the next round.
We kept stable identifiers for the questions. The author could point to an answer already present in the document, or acknowledge a gap and update the design. The reviewer had to check the actual document. An explanation that existed only in chat could still leave a future developer with an ambiguous PRD, so that was not enough to close the issue. The final review conclusion was also tied to a specific document version.
Grilling supplied a structure for the questioning: treat the design as a tree of decisions with dependencies, ask the questions whose prerequisites are currently settled, and use the answers to open the next branches. We adapted this method so that the primary interviewee was the agent responsible for the PRD.
For example, "the asset has been generated" led to further questions: Can the user actually use it? Is it interactive once placed in the game? What state is it in if its quality is unacceptable? Will further revisions incur additional costs? If the user stops the process, what happens to generation results that arrive afterward?
Facts that agents could investigate were investigated by agents. Questions covered by existing product decisions were answered by the author session. Questions requiring new product tradeoffs still came back to me.
In the first phase, the PRD moved from v0.2 to v0.3. Two rounds of questions raised and resolved 11 issues, followed by a final verification pass. These covered costs and attempt limits, user choices and delegated authority, what counts as completed game integration, interactions involving multiple characters and props, delivery of editable source files, and stopping work partway through.
Here, "resolved" means the issue had been addressed in the document. It does not mean the corresponding functionality had been implemented or validated through actual use.
Then I noticed that both sessions had missed something fundamental:
Asset creation should be an ongoing process of co-creation between the user and Codex.
The design and review had focused heavily on workflows, states, deliverables, and acceptance criteria. They had not sufficiently explored how users would try things during creation, express reactions, change direction, and work with Codex to adjust animation, split assets into components, and revise the assets themselves.
So, even after that review had passed, there was still a structural gap in the intended product experience.
Once I introduced this perspective, the author rewrote the relevant parts of the PRD. The reviewer then examined v0.4 from the perspective of co-creation and raised and resolved three more issues: how work continues after the first sample is adopted, how users can actually operate and try the result, and how real feedback leads to an asset revision that the user can assess again.
This exposed a limit of the approach: separate contexts can help with independent review, but two agents can still share the same mistaken assumptions. If an important branch is missing from the initial design tree, working through all the existing branches will not automatically uncover the missing user experience.
We therefore interpret a passed review as: given the current requirements, review perspective, and document version, the questions raised have been resolved. Human experience and new understanding can reopen the design tree at any time.
This is a single-project experience, with no controlled comparison or post-implementation outcome data. What it has produced is something we can inspect: explicit questions, concrete document revisions, and a complete example of a human identifying a shared blind spot and restarting the review.
Having read the discussions around pre-implementation review, including #633 and #1051, I thought this was worth sharing as a way to compose the existing Grilling skill into a workflow.
I would be interested in others' experiences: when using Grilling for design review between agents, how do you uncover the questions that neither side thought to ask?
All reactions