Replies: 2 comments
|
The thing worth isolating is the invisible decision. Provenance separation is one fix for it, and an expensive one. You already flag that C1 might be impractical, or might build a world that never exercises the feature. C1 does two things at once. Worth splitting them:
C2: before implementing, the agent writes down which ambiguities it found, which branch it took, what it assumed about states the requirement left open. Versioned next to the acceptance criteria. Reading:
The objection: an agent logs the ambiguities it noticed. The dangerous ones are the ones it never registered as questions. Countable the same way as the rest of your protocol. Cheap enough to run before the expensive arm. |
|
Thanks Toni. Good impulse, and exactly the kind of contribution I was hoping for. C2 is cheap enough that I should test it before the more expensive provenance-separated setup. I’ll add it to the protocol and see whether simply making the hidden decisions explicit already gives most of the effect. The interesting part for me is the failure mode you point out: the agent can only log ambiguities it actually noticed. The dangerous assumptions may be the ones it never recognized as decisions at all. I’ll investigate this and test the thesis. If C2 performs close to C1, that would materially change my current hypothesis. |
Uh oh!
There was an error while loading. Please reload this page.
Why?
Disclosure first: I work on DATAMIMIC at rapiddweller. The field observations below came out of the Community Edition. The hypothesis has to be tool independent and reproducible with SQL, Python or model based tooling. The tool is the instrument in the observation, not evidence for the claim.
When you implement a feature yourself, implementation forces you to decide. You can't write a branch on a question you haven't answered. Ambiguity in the requirement gets resolved at the keyboard, whether or not anyone writes the resolution down.
When an agent implements, that's gone. The agent picks a branch. The choice looks deliberate. Nothing surfaces the fact that a decision happened at all.
The decision doesn't disappear. It moves. And by default it moves into the artifacts nobody reads, which in a stateful system means the fixtures. So one interpretation ends up producing all three of these:
Green then means internally consistent. It doesn't mean correct.
#39 and the Direction 3 thread ask who authors the criterion. I want to ask who authors the conditions the criterion gets exercised under.
This matters most for stateful relational systems, where the test world is large, expensive to construct, and almost never reviewed by anyone. It matters least for stateless pure functions, where the input space is small enough that fixtures are readable. Teams shipping multi state business processes with coding agents are the ones who benefit if the answer turns out to be actionable. Teams doing CRUD probably don't need any of this, and finding that boundary is part of the research.
Expected results
H1. For new behaviour in stateful systems, at least one acceptance critical state combination is absent from production history, or too rare to provide usable test coverage.
Measurable directly. Take shipped features, enumerate the state combinations their acceptance criteria require, query production for how often those combinations occur. I expect this to fail for simple CRUD changes and hold for multi state process changes. Where the boundary sits is the useful output. If it fails broadly, the whole argument is scoped to a niche and I want to know that early.
H2, open. Constructing the test world from business intent, risk and acceptance needs, without access to the implementation under test, surfaces more interpretation errors before merge than constructing it in the same context as the implementation.
The claim is not that separate artifacts help. Two files written from one interpretation are one interpretation. The variable I suspect matters is provenance: what information was available when each artifact was authored. Which makes the interesting intervention something other than "put your test data in a versioned file". It's closer to "build the world out of something other than the thing you're checking".
What would make me reject H2. If provenance separated construction finds no more pre merge interpretation errors than same context construction while costing more human time. Also if everything it finds would have been found by a fixed realistic snapshot anyway, because then the populated state is doing all the work and the specification adds nothing.
Three questions I want answered, in this order:
Does the world matter? Does a preconstructed deterministic connected world beat implementation derived local fixtures at all?
Does the specification matter? Holding the materialised world constant, does having the executable specification that constructed it change what a reviewer can find?
Does provenance matter? Does authoring the test world from pre implementation artifacts only beat authoring it in the same context as the implementation?
Question 1 tests the boring explanation. If it comes back flat, questions 2 and 3 are not worth running and my thesis is staging environments with new vocabulary.
Collaboration
Concretely, what would help:
Tear down H2. Particularly if you think correlated failure kills it outright, or if you think question 1 will come back flat. I'd rather hear that now.
Experience reports. If you've run agentic development on a stateful relational system, what did you do about the test world, and what broke? I'm looking for cases where a green suite hid a shared misinterpretation, and for cases where it didn't.
Review my experiment designs before I run them. Especially the provenance boundary in question 3. I already got that wrong once, see Background.
Run question 2 yourself. It needs no new features and no second implementation. A populated scenario you already have, its specification, and two reviewers who haven't seen either. Anyone with an existing generated or seeded environment can produce counts in a day.
Prior art. I see overlap with property based generators, stateful and model based testing, and the oracle problem literature. What I haven't found is work treating the provenance of the generated test world as a variable when the system under test is itself authored by an autonomous coding agent. If that exists, that's the reference I'm looking for and it saves me a lot of time.
Target artifact
An article, and a reproducible experiment protocol others can run against their own codebase. If the results hold up, a talk. If question 1 comes back flat, a short honest writeup saying so, because that's also worth publishing.
Background and additional information
Greenfield is trivial, brownfield is the real case
Greenfield has no production data. Obvious, and not the interesting case.
Brownfield is where the sloppy version of this argument usually gets made, so let me be careful. It's not true that production data doesn't exist. It's not true that you can't use it. What's true is narrower:
New behaviour has to hold in the world as it was, composed with states or state combinations that may never have occurred, or occur too rarely to serve as test material.
A production subset gives you historical structure. It can't guarantee the novel combinations the new behaviour is supposed to survive.
Take a new payment recovery path across seven states. The customers are real, the accounts are real, the history is real. The specific combination the new rule has to survive may have a handful of instances, or none. Somebody has to build it. Right now that somebody is usually the same agent pass that wrote the code.
Two field observations
We built a stateful relational application with coding agents. Two cases stuck with me.
Cross role contradiction. A field employee saw four appointments. The dispatcher had no corresponding tour. Each role view was locally plausible. Looking at either projection on its own didn't reveal the violated cross role invariant, because the contradiction only existed once the two were related through the same business state.
Being precise about what did and didn't happen: the data specification didn't diagnose the root cause, and we had no automated cross role invariant that caught it. What the deterministic scenario gave us was a reproducible environment, which is what let us turn the contradiction into a permanent regression property afterwards.
A green test exposed a missing requirement. In a deterministic planning scenario, two scheduling algorithms placed the same 1,320 services with zero unplanned appointments. The candidate cut our approximated drive time metric by roughly 90 percent and cleared the acceptance threshold comfortably.
Workload distribution across employees got substantially worse.
The scheduler hadn't violated the specification. The specification had never said anything about acceptable workload balance. So the executable state didn't find an implementation defect. It made a gap in our own acceptance criteria measurable.
The percentage comes from a seeded deterministic scenario, so it's reproducible rather than averaged over runs. The drive time metric is an approximation. Don't read precision into it.
The boring explanation I can't rule out
Both observations are fully explained by something much less interesting than my hypothesis. A large populated connected deterministic environment reveals defects that role local fixtures hide. That's the argument for integration and staging environments, and it's twenty five years old.
Neither case shows that the separateness of the specification did any work. A sufficiently realistic fixed snapshot would plausibly have surfaced both findings. I can't discriminate between these with what I have.
There's a second problem I'd rather say myself than have found. Parts of our data specification were authored in the same working context as the implementation. Separate files, one interpretation. If the mechanism I care about is real, my own setup didn't test it.
Why separate artifacts aren't enough
Knight and Leveson showed that independently developed program versions did not fail independently, which rejected the independence assumption those reliability models were built on. Separate authorship alone gives me no basis for assuming separate interpretation.
My own reading, not their finding, is that the shared input is the likely culprit. If the data world and the implementation both derive from the same requirement text, the correlation survives the file boundary. The analogy to my case is imperfect. It's enough to kill the naive assumption that separate production yields independent evidence.
Executable feedback, and where I'm unsure
Agents get dense executable feedback about code. Compilers, type checkers, linters and test runners constrain what they can get away with. Domain state feedback is much weaker. Unless we encode invariants explicitly, almost nothing tells an agent that the business world its implementation produced is incoherent.
A materialised relational world may supply a different kind of signal. Not because it sits outside the model's context, since every observation eventually becomes context. Because the observation is causally downstream of executing the implementation against state that need not have been derived from that implementation.
The distinction I'm reaching for is between giving a model more information and giving it externalised executable feedback. I don't know whether that amounts to more than good integration testing, and I'd like to be argued out of it.
The three experiments
I originally had a single four arm design. It confounded world content, generation method, representation and reviewability, so it couldn't have told me which one mattered. Splitting it costs more total work, but each question becomes answerable on its own.
Question 1, does the world matter. A0, agent authors implementation plus local fixtures plus tests. A1, agent authors implementation plus tests against a preconstructed deterministic connected world. This tests the boring explanation directly.
Question 2, does the specification matter. Hold the world constant. B0, a reviewer gets only the populated snapshot. B1, the same reviewer gets identical populated state plus the executable specification that constructed it. Measure review minutes, incorrect assumptions found, whether they can explain why a given condition exists, and time to correctly modify the scenario. This isolates review surface from world content. It's the cheapest of the three and needs no features built.
Question 3, does provenance matter. C0, implementation and test world authored from the same agent context. C1, test world authored from pre implementation artifacts only, meaning requirements, accepted domain and schema contracts, acceptance criteria and identified risks, with no access to the implementation, its fixtures or its generated tests. Same feature specification, comparable world complexity. This is the one that actually tests H2.
Note the boundary in C1. In an earlier draft I wrote "no access to the schema", which was wrong. Without agreed domain and interface contracts the author can't build a structurally relevant world at all. The line runs at the implementation and everything derived from it, not at all shared information.
Counted throughout: interpretation errors surviving to human review, incorrect green acceptances caught later, cross boundary defects found before merge, time to first contradiction, human review minutes, correction iterations to green. Eight to twelve features, raw counts, no statistics at that sample size.
Where I expect disagreement
Is this integration testing with realistic data, renamed for the agent era? If you think yes, I want the argument, because I keep failing to rule it out myself.
My second observation may cut against me. The workload finding came from a human deciding to look at a metric nobody had specified. No artifact produced that. If the scarce resource is human attention pointed at the right question, then adding another specification surface might consume exactly the attention that produced the finding.
Is provenance separation achievable at all? Authoring a test world from pre implementation artifacts only sounds right and might be impractical, or might produce a world that never exercises the feature.
All reactions