Curious Questions #14
|
Thanks for sharing this repo. I've a few curious questions but they are not urgent so feel free to skip/address then as you please. Real failure that created Foremerge? I might add to them as I explore the repo further. |
Replies: 8 comments 10 replies
|
What real failure made you build this? Was it a specific multi-agent wreck, or a bet that fleets would need a whiteboard above Git? |
|
Initially it was just becoming a PITA resolving collisions and conflicts from multiple agents working in parallel on multiple work trees at the same time and duplicating code/functions. We had it on our roadmap to find a better solution but with no success as it didnt seem to exist at this layer of the stack. What pushed us to focus on Foremerge was a specific incident where the team was using multiple agents to replace a payment service class and one of our automated agents picked up a ticket from the backlog to extend it with an additional provider. Different worktrees, no overlapping lines, both merged clean, and the second change depended on an extension point the first had deleted. We caught it in review and it would have failed in the CI pipeline but something like this could quite easily slip through into production without proper tests in place. Thankfully it only cost us a day of refactoring but the gap was obvious. After using the tool successfully during our beta launch since Jan and feeling that this gap needed to be filled for the rest of the community I extracted the tool from the internal tools monorepo and open-sourced it in late August. Why intent before code rather than post-hoc diff analysis: timing and determinism. Post-hoc catches the collision after both agents have spent their tokens, so the best you can do is discard or refactor work. Comparing plans and intent catches it before those headaches and time/cost. A declared operation is a fact you can assert HIGH on, while an operation inferred from a diff is a guess. A coordination layer that guesses wrong at HIGH severity gets ignored within a week, which is why prose-inferred matches cap below HIGH in 0.4.0. Primary user today for this open source project is solo power users running 2+ agents on one machine. We tested it up to 98 agents running in parallel with zero conflicts (it was supposed to be 100 but 2 failed due to system resources and everything else ran slow as heck partly because of rate limits on the providers). Fleets and teams are the later story and honestly needs further validation on our cloud solution before we share that. What 1.0 has that 0.4.0 does not: published coordinated-vs-uncoordinated benchmarks with raw data, versioned JSON Schemas for the full protocol surface hopefully with community agreement and feedback, at least one language adapter so symbol scopes resolve structurally instead of as strings, and calibration evidence that findings correlate with real conflicts. Protocol stability is the promise 1.0 makes. |
|
Why advisory-only and never hard locks? How do you think about the tradeoff of “agent ignored the warning and shipped damage”? |
|
A lock assumes the declarer is right and will finish, and agents abandon work constantly. One stale lock halts could halt and gridlock the entire fleet. Deeper than that, we cannot enforce a semantic lock without intercepting filesystem writes, which we deliberately do not do, so a "lock" would be an advisory pretending to be a lock. On "agent ignored the warning and shipped damage", the warning is not the enforcement point, acceptance is. Accepting a ChangeSet requires no unresolved HIGH findings plus verification against the exact candidate fingerprint, and the ledger records that a finding was overridden, which makes ignoring it a visible decision instead of a silent one. Deterministic rules, no LLM judge: the conflict detector is the trust core, so it has to be reproducible, testable, free, offline, and identical for every user. A judge that changes its mind between runs erodes exactly the trust the layer exists to provide and an LLM judge is only as good as the context it is given. The real limits of rules show up when meaning, not exact text, matters. semantic equivalence (synonyms, undeclared renames, cross-file contract drift). The plan is to keep rules authoritative and let fuzzier matching only propose findings below HIGH. We put SQLite in the git common dir because its zero infrastructure, transactional, queryable, and the common dir is the one location every worktree already shares, so isolation plus shared state comes at no cost. CRDTs solve convergence of concurrent edits but our problem is whether two plans can both be true, and convergence is not correctness. Git notes are append-only but query poorly and entangle coordination state with push semantics. CRDT's are engineered to make conflicts disappear where as Foremerge exists to make them visible. They're excellent for what they're built for but the wrong tool at the layer we would use it. I would push back hard on a new storage engine until SQLite produces evidence one is necessary. One binary rather than a library: agents integrate through protocol surfaces (MCP, CLI, JSON API), not linkage, and runtimes already speak MCP. A library per runtime means N bindings with version skew and one binary keeps the deterministic core single-sourced. |
|
One of yours I skipped: non-goals people keep asking for anyway. Hard locks are the most requested, then a merge queue, then editor-buffer sync. They share a theme, each would make Foremerge claim authority it cannot honestly enforce. A lock it cannot physically hold, a merge decision that belongs to your Git host, a synchronization layer that would confuse convergence with correctness. The non-goals list is the tool refusing to pretend. |
|
How much do agents actually publish intents in the wild? Compliance vs tool friction, what have you seen? |
|
With the MCP skill or project template in context, compliance is good, because declaring intent is just another tool call in the loop. Without it, compliance is whatever your system prompt enforces. Getting those hard numbers on wild compliance rates is one of the things we most want real traces for (outside of our own use) Vocabulary stability: the seven operations are versioned with the protocol. I think expansion needs to happen conservatively through discussions with the communutity like this one. Same with removals. If you hit a real change the vocabulary cannot express, please report that or add it as a contribution. Claim vs intent cadence: publish intent when the plan forms, claim when you are about to touch the scope. Claims are leases, so long sessions renew them, and a materially changed plan is a new intent. Detecting drift between the declaration and what is actually being edited is on the 0.5.0 roadmap based on other feedback we received from reddit users. "None yet" vs "none ever": correct, and another source reviewer made the same point. 0.4.1 adds an as_of sequence to conflict queries so a client can tell how stale its view is instead of mistaking an early empty answer for a verdict. |
|
Also missed your last one: two agents claim the same scope and both continue. Both can, deliberately, because claims are advisory leases. The second claimant sees the finding at claim time and its options are all first-class moves: proceed (recorded), record an assessment (conflicts or depends_on), or re-scope. If both continue anyway, acceptance is the backstop: the first ChangeSet through accepts clean, the second faces an unresolved finding that needs explicit resolution or an operator override, all of it in the ledger. For HIGH findings my advice is a human makes the call. The tool's job is making sure the call happens with both plans visible, not at merge. |
|
How do you measure false positives vs false negatives on real multi-agent traces? |
|
We are measuring using paired runs, the same tasks with and without coordination on different claude and GPTree Graft accounts, then labeling discarded work, post-integration failures, and warnings dismissed as noise. False negatives are the hard half because of breaks that nothing detected, which means human labeling. We will publish raw results before making any quantitative claim, and until then we made sure to note this in the documentation. Synonyms: worth splitting into three layers. Declared renames are in the operation vocabulary and detected deterministically. Across different labels (PaymentService vs BillingService) the deterministic core refuses to guess on purpose so scope matching is token-normalized but never speculative. Speculation about synonymy erodes the trust the HIGH tier depends on. Instead the protocol hands semantic similarity to the layer that actually has semantic understanding, the agents. An agent that recognizes two differently-named intents overlap records it with an assessment (conflicts, compatible, duplicate, or depends_on, with rationale), and that verdict becomes durable, queryable coordination state that the rest of the fleet sees. Language adapters shrink the alias problem near-term by resolving symbols to canonical paths instead of strings. The research track I think is narrower than it sound and is really whether the engine itself should ever propose cross-label matches below HIGH with calibrated confidence. Suggestions stay heuristic suggestions in core. The policy layer being designed in the community threads is where a team encodes what must happen when a finding fires, the resolution itself stays a recorded human or agent decision. Probably the least confident rule is "divergent_rewrite". Two broad modifications of one scope are sometimes genuinely compatible, which is why it fires below HIGH. We keep it because the cost favors the warning i.e. dismissing a wrong one costs a sentence, missing a right one costs a rewrite. |
|
Validation runs as trusted OS commands with no sandbox, what’s the intended threat model for agent-supplied check names? |
|
Threat model: checks are registered by name, by a human, once and agents reference the name and cannot supply arbitrary command lines. The residual risk is real and documented: a hostile agent can write malicious test code that a legitimate check executes but our threat model is fallible agents, not adversarial ones. If your agents are adversarial, you need OS-level sandboxing underneath everything and no coordination layer replaces that. An agent that has shell access could run "foremerge checks set" itself but that agent could already run anything directly, so the boundary is meaningful for MCP-scoped agents. Also, agents still write the code that a legitimate check executes, so a malicious test file runs with your privileges regardless. I think mutate and restore gaming is probably accepted residual risk today, stated in limitations. Verification binds to the exact fingerprint and failed validation leaves refs untouched, but a test that lies about itself can win. Re-hashing the tree after the run is a plausible 0.5+ hardening. Accept vs merge: core stays above Git permanently. Not letting this turn into merge queue is a listed non-goal. The roadmap path is status checks that surface acceptance state to your Git host and let your branch protection do the gating. Git gates, Foremerge informs. State not in clone/push is intentional. Git is the wrong transport for it as coordination state is live, expiring, operational data, and Git moves snapshots of history. A claim is a lease is pointless if you're hoping someone pushes in time, and a stale synced copy is worse than none because it looks authoritative. Provenance is machine-local operational data right now, not repo history. Export/import tooling is on the roadmap, and team-shared state, when teams need one "whiteboard", they'll get it by pointing their agents at a shared Foremerge service which comes later on the roadmap with optional shared coordination. The trust model is the hard part. What's actually at risk: The store is .git/foremerge/state.sqlite3 in the git common dir, running in WAL mode. Losing it loses zero code (Git has all of that) but it loses coordination state and history: open intents and claims (transient, agents can re-declare), and the durable stuff: the hash-chained event ledger, assessments, verification evidence, acceptance decisions. For a solo user, that's telemetry and audit trail, not source of truth and probably dosnt matter too much. For someone who cares about the provenance ledger as a record, it's real data and backup is wise. The two loss scenarios that actually happen: re-cloning a repo (fresh .git, state gone, the most common one by far) and disk death. Time Machine/ etc already covers the second for most people since it's just a file, the catch is that some backup configs exclude .git directories. The correct temp solution: never cp the file since WAL mode means a copy taken mid-write can be corrupt or missing recent commits. Instead, use SQLite's online backup, which is safe while the service is running: |
|
Multi-machine shared daemon is later. What’s the hardest unsolved piece auth, consistency or threat model? |
|
Hardest piece is the trust model more than consistency. Consistency is boundable (one shared daemon first, no distributed SQLite pretensions). But a verification result asserted from another machine is hearsay unless re-executed or attested, and designing auth and attestation without turning a local-first tool into a server product is the genuinely hard part. That tension is why it sits in "later" right now. First language adapters will probably be TypeScript and Python, following where agent usage concentrates. Manual scopes stay first-class forever: APIs, schemas, config, infrastructure, and environment variables live in no AST. Many of the things agents collide on aren't code symbols and don't appear in any syntax tree. Benchmarks: pilot after the 0.5.0 tooling settles. Windows: release CI builds and tests x86_64-pc-windows-msvc today, including Windows-specific validation-timeout behavior, and cargo install works. The curl installer is unix-only, a winget or scoop manifest would be a very welcome contribution. |
|
And your business-plan question: the core stays Apache-2.0 and local-first, and the local tool will never require a server. Hopefully some sponsors will cover maintenance eventually. Right now we are covering all maintenance and costs at GPTree. If a sustainable business ever grows around this, it will be around optional team infrastructure, the shared coordination service in the roadmap's later section, never around gating the local tool. The word optional in that roadmap entry is load-bearing and I intend to keep it that way. |
|
Best path to stickiness: MCP skill, foremerge setup or CI/status checks? |
|
The best path to stickiness is the MCP skill that works with any coding tool (Claude, Codex, Cursor etc) because it makes coordination another tool call instead of a discipline no matter which one you use. Setup takes seconds and just works. Templates second. Status checks third, because that converts an individual habit into a team default. I think what makes runtimes default to it will be community adoption and the fact that the problem is being boring and neutral lol. Versioned schemas, no vendor coupling, Apache-2.0, published evidence. Runtimes adopt what their users already run, so the path is bottom-up through MCP, not partnerships. Locks and worktrees manage files, we compare plans and intent. Worktrees are complementary and we assume them. If someone ships actual semantic plan comparison, we compete on determinism, the verification gate, and the provenance ledger, and if they do it better, it is Apache-2.0 all the way down and users win. Most wanted contribution right now is conflict evidence. Real traces where detection was right, wrong, or missed, even a paragraph describing the collision. Second: scope vocabulary proposals grounded in a real codebase. Third: client adapter recipes for specific agent stacks. You clearly read the repo closely, so whichever matches how you explore, the issues are open. It took some time to go through all these and answer thoroughly so I appreciate your patience and can't thank you enough for the time you have taken to review this so carefully. Nick |
|
Really good batch of questions. I will answer each of your batches in its own thread, including the four from your opening post (the real failure story is under your first comment, hard locks under the second, conflict quality under the fourth, multi-machine under the sixth). This was a genuinely useful set of questions, keep them coming as you dig. |
Really good batch of questions. I will answer each of your batches in its own thread, including the four from your opening post (the real failure story is under your first comment, hard locks under the second, conflict quality under the fourth, multi-machine under the sixth). This was a genuinely useful set of questions, keep them coming as you dig.