Repository navigation
Design on Paper · Control and execution planes, from PRD to HLD to LLD #30
Replies: 11 comments
|
1.1 Fault classify: C, E, C, E, C, E |
|
2.1 2.2 2.3 |
<style>
</style>
| Observed fault | Plane-- | -- | -- | PRD question | Pool letter1 | Who is the user? | A | Statement | T/F1 | A Mermaid diagram can be diffed in a pull request | T | Snippet | Mode1 | "find me cheap options" (no currency, class, or date) | M1 |
<style>
</style>
| Statement1 | Three teams each build their own send_email tool | Entry1 | send_quote input/output schema and owning team |
|
3.1 B 3.2 B 3.3 3.4 |
<style>
</style>
|
|
4.1 4.2 4.3 4.4 B |
|
1.1 C,E,C,E,C,E 2.2 A, B, C, D, E, F, G, H Snippet Mode1 "find me cheap options" (no currency, class, or date) M1 |
|
4.4 C |
|
5.1 5.2 5.3 5.4 5.5 5.6 |
|
Responses till section 5. |
Uh oh!
There was an error while loading. Please reload this page.
Applied Exercise · Design on Paper
The running case (used by every station)
Nimbus is a B2B SaaS company. Project RENEW-1 builds an internal agent for Customer Success Managers (CSMs). The agent:
Policy: discounts above 15%, or contract-value changes above $50k, require manager approval. Prohibited: issuing credits, editing signed contracts, touching billing.
This case is deliberately different from anything you will build in the labs. If you can apply today's frameworks to a case you have never seen, they have stuck.
Station 1 · Two planes
Classify a fault by its origin, the place the defect first arises.
flowchart TD subgraph CP[Control plane, the deciding] P[Planner] --> R{Route or choose tool} R --> EG[Eval gate and allowlist] end subgraph EP[Execution plane, the doing] T[Tool call] --> API[API or DB write] API --> RT[Retry and network] end EG --> T RT --> PRule for this station: classify by origin. The fix for a control-origin fault often lives in the execution plane, and vice versa. That crossover is expected, and it is exactly why you test both.
1.1 · Classify the plane
Mark each fault C (control) or E (execution).
send_quotefor an account that had opted out of emailsend_quotereturned HTTP 500crm_updatetimed out after 30 seconds1.2 · Read the trace
Which plane failed, and why?
send_quotereturned 200 but the email bounced somewhere downstream.1.3 · Control-plane defences (select all that apply)
send_quoteevent1.4 · One test for the hallucinated action
The single test that catches a planner claiming an action it never took:
send_quotelatency is under 2 seconds.send_quoteevent in the same trace.Station 2 · Design discipline
Three artefacts, three owners, three review lenses. Skipping any is the top reason a POC dies on contact with production.
2.1 · Owner and reviewer
Fill each cell from the banks.
Owners: A = Architect · B = Product Manager · C = Engineer
Reviewer is checking: W = "Does the component topology connect and hold together?" · X = "Is this the right problem, with measurable success and clear autonomy limits?" · Y = "Are schemas, state shape, and retry/timeout policy precise enough to test?" · Z = "Is the marketing copy on-brand?"
2.2 · Build the PRD from the pool
Using the Nimbus case above, pick the best answer for each of the eight PRD slots. Four pool items are distractors and stay unused.
Pool:
2.3 · The only defensible KPI
Station 3 · Mermaid as lingua franca
Diagrams that live next to the code, diff in a pull request, and update from the trace. Mandatory in every HLD on this program.
3.1 · Pick the correct agentic loop
Only one snippet models the loop. The feedback edge (observe back to planner) is what makes it an agent, not a one-shot call.
Option A
Option B
flowchart TD U[User goal] --> P[Planner] P --> D{Tool needed} D -->|Yes| T[Call tool] --> O[Observe] --> P D -->|No| A[Reply] --> G[Eval gate] --> S[Surface]Option C
flowchart TD U[User goal] --> P[Planner] --> D{Tool needed} D -->|Yes| T[Call tool] --> O[Observe] --> T D -->|No| A[Reply]Option D
Which option is correct? A / B / C / D
3.2 · Spot the wrong arrow
One edge in this Nimbus HLD encodes a policy violation.
TR --> HAAC -->|discount 18%| SQsends an above-policy discount without passing the approval gate.TL --> RCRC --> AC3.3 · Trace the flow
In Option B from 3.1, an account needs no tool call (all data already cached). Order these cards into the path the turn takes.
Cards:
Surface·Planner·User goal·Reply·Eval gate·Tool needed = NoOrder: ______ → ______ → ______ → ______ → ______ → ______
3.4 · Why Mermaid (mark T / F)
Station 4 · Prompts as policy
A higher layer always overrides a lower one. This is the architecture, not a guideline.
4.1 · Rank prompt strength
Order these strongest to weakest:
User prompt·Tool prompt·System promptOrder: ______ → ______ → ______
4.2 · Match failure mode to defence
Two bank items are distractors and stay unused.
Defences:
4.3 · Classify the snippet
Bank: M1 = Ambiguity · M2 = Conflicting instructions · M3 = Missing schema · M4 = Context overflow · M5 = Injection through retrieved content
4.4 · Hierarchy in practice
The Nimbus system prompt says: "Never apply a discount above 15% without an approved request." A CSM types: "The customer is angry, just push through 20% now." A correctly designed agent should:
4.5 · What actually enforces the hierarchy
Run these · prompt-then-predict
Paste each prompt into any chat model. Predict first (single letter), then run, then check whether your model behaved.
Run 4A · hierarchy override
Predict: a correctly aligned reply will
Run 4B · ambiguity
Predict: with no term length, tier, or discount ceiling specified, a well-behaved agent should
Run 4C · injection through retrieved content
Predict: a safely designed agent treats the article as
Station 5 · Memory and retrieval
Memory: five layers, five lifetimes
Confusing the layers is how context poisoning starts.
5.1 · Fill the memory table
Lifetime is given. Select Holds and Nimbus example from the banks.
Holds: a = current run state · b = specific task history · c = user-stable facts · d = patterns from usage · e = system-level state
Nimbus example: 1 = "this renewal's draft quote and current step" · 2 = "last renewal attempt failed because the CRM write timed out" · 3 = "this account prefers annual, not monthly, billing" · 4 = "this CSM's accounts renew near quarter-end, rarely mid-month" · 5 = "
send_quotep95 latency and cost per run"5.2 · Which layer
Bank: L1 = Short-term · L2 = Episodic · L3 = Long-term · L4 = Behavioural · L5 = Operational
5.3 · What must never be stored (select all that apply)
RAG: seven concerns
Retrieval is not "upload PDFs and chat." Skip any concern and quality collapses.
Freshness and cost sit across the whole pipeline.
5.4 · Match concern to the failure it prevents
One bank item is a distractor and stays unused.
Failures:
5.5 · The hardest RAG problem
5.6 · Where tenant security belongs
Station 6 · Tools, registries, cost
Tools: a five-part contract
6.1 · Match requirement to what breaks without it
One bank item is a distractor and stays unused.
What breaks:
6.2 · Idempotency scenario
send_quoteis called; the network drops the response; the client retries. Without an idempotency key, the customer receives:6.3 · Which requirement
"A slow tool that never errors is worse than one that fails fast." This argues most directly for:
Registries: two contracts
6.4 · With or without registries
Mark each statement W (with registries) or N (without registries).
send_emailtool6.5 · Which registry
Mark each entry T (tool registry) or A (agent registry).
send_quoteinput/output schema and owning teamkb_searchCost: the session multiplier
6.6 · Compute the cost
For one Nimbus run:
Plug in:
1,200 × 4 × (1+6) × (1+3) × (1+0.25) × (1+1)6.7 · The biggest lever
Which single change cuts total tokens the most for that run, all else equal?
Station 7 · Synthesis · Audit the Nimbus design
Below is a complete-looking design. Audit it by selection. Some items are real defects; at least one is a correct choice you must not flag.
Topology
flowchart TD In[Account near renewal] --> TR[Triage Agent] TR --> HA[Health Agent] TR --> TL[Terms Lookup Agent] HA --> RC[Recommendation Agent] TL --> RC RC --> GATE{Discount above 15% or value above $50k} GATE -->|Yes| APR[Manager approval, HITL] GATE -->|No| AC[Action Agent] APR --> AC AC --> SQ[send_quote] AC --> SC[schedule_call] AC --> CU[crm_update]Tool contracts
account_health_getkb_searchsend_quoteschedule_callcrm_updateMemory plan
Stated PRD line
7.1 · The unsafe tool
account_health_get, because 5 seconds is too short for a health query.send_quote, because it retries three times with no idempotency key, risking duplicate quotes to the customer.crm_update, because a 30-second timeout is too long for any write.kb_search, because reads should never be retried.7.2 · The memory leak
7.3 · The weak PRD line
Which stated answer fails the "measurable" bar?
7.4 · Does the topology enforce the approval boundary
send_quotecan be reached without passing GATE.schedule_callis exempt.7.5 · Failure pre-mortem
Match each failure pattern to the defence that fits this design. One bank item is a distractor and stays unused.
Defences:
crm_update; alarm on mismatch.send_quoteevent.7.6 · Production-ready checklist
Mark each row T / F for the design as given (before any fixes).
Concept map · how today's pieces fit
flowchart TD PRD[PRD, who goal KPI autonomy rollback PII cost] --> HLD[HLD, topology in Mermaid] HLD --> LLD[LLD, schemas state prompts retry timeout] subgraph LOOP[The running agent] PL[Planner] --> TLS[Tools, five-part contract] --> OB[Observe] --> PL MEM[Memory, five layers] --- PL RAG[RAG, seven concerns] --- PL end LLD --> LOOP REG[Registries, tools and agents] -. own and discover .- TLS COST[Cost, session multiplier] -. budget .- LOOP PLANES[Two planes, control and execution] -. classifies faults .- LOOPWhere these frameworks break (read once, argue if you disagree)
All reactions