Ah honest feedback about the first use experience

[rimmer]

I just wanted to give a quick feedback since I believe the project worth it.
It is going to be not a comprehensive review but rather a note. But since I noticed that the project perhaps lacks some feedback from the community maybe it will be helpful.

If you find it inappropriate feel free to just erase/hide it.

First and foremost: the project has a great potential and solves a problem with a good approach: a lot of control and configurability. It also plugs other tools instead of creating another harness.

But despite spending around an hour and a half to configure it I wasn't able to achieve my goal.
I had a bunch of PDRs and related tickets waiting for implementation. I wanted to find an orchestrator that can help me quickly implement them in parallel in separate git trees.
Despite Bernstein seems easy to configure and start using, it is not.

1. The documentation looks complete, but if you try to follow, there are several points of entry which are outdated. An example: it seems the agents concept have been renamed to adapters at some point but it is not reflected.
2. Therefore I just could not do initial planning with my agents (I have goose and pi). When I switched to codex, I still could not do it because I was trying to do bernstein -g but without a parameter to specify an adapter because it was complaining that the adapter is not set. Then I just experimentally found out if I configure it in my seed file it works.
3. After running it, it did not produce a plan despite it probably was supposed to.
   At this point I stopped.

What I see needs to be done to make this project usable by other people without consulting the author.

1. Dev experience interview: just give the project to someone new and give them several tasks. See how they do it.
2. Bots to update outdated docs automatically when doing changes.
3. Review docs (perhaps with someone else) to identify points of entry and the general flow -- how people navigate them. Remove all redundant and outdated stuff.

And a highly subjective comment that is controversial: the app is trying to do a lot of stuff and it does not do it well sometimes. The TUI is hard to understand at the first glance, some issues like the scrolling does not work. Perhaps reducing a number of features and the dev UI may reduce the surface of work and make the project maintenance easier.

Sorry again for being perhaps a bit more negative and not focusing on good, but I just couldn't get the "good" working so I can't objectively review it.
I'd be happy to help with all mentioned after mid September when I am done with my stuff, I think the project is worth it.

Best regards

[chernistry]

Thanks for writing this up. An hour and a half of setup that ends in nothing is the report I can't produce myself. I know where everything is, so I can't see what only makes sense once you already know it. Nothing here is inappropriate. It stays.

Point by point, and what I did about each.

1. Docs drift (agents → adapters). Correct, and the rename is only the visible half. The real problem is that nothing fails when a concept gets renamed and the prose keeps the old word: the reference docs are generated and checked, the narrative pages people actually land on first are not. Filed as #3527. The fix there is a gate, not a find-and-replace pass; a pass drifts again by the next release.

2. bernstein -g told you to pass a flag that does not exist. Real bug, and it's fixed: #3526, PR #3530. The message named --adapter, which is an argparse flag belonging to the internal orchestrator subprocess, not an option of the bernstein command you typed. It now names --cli, and a test resolves every flag and command in that message against the live CLI so the text can't drift back. The part that made it worse: the other two remedies it listed (BERNSTEIN_ADAPTER, cli: in bernstein.yaml) did work, which is how you eventually landed on the seed file. One wrong option out of three is more confusing than three wrong ones.

For what it's worth, goose and pi are both registered adapters and go through the same planning path as codex, so nothing about your first attempt was wrong. From your description it was the same fatal all three times — the switch to codex changed nothing because the adapter was never the problem.

3. Run started, produced no plan. Filed as #3528. I can't reproduce it from what I have. If the run directory still exists, bernstein debug bundle zips the last run's traces, logs, config and a doctor snapshot with secrets redacted first. That would show in one look whether the adapter returned nothing or returned something the planner rejected. No hurry on it.

4. TUI scrolling and first-glance layout. #3529.

On the controversial point, a straight answer instead of a balanced one, since a balanced one is no use to you. You're right about the size. bernstein --help currently lists 196 top-level commands, which is not a number one person can maintain honestly. But I'd cut somewhere other than where you might expect. The adapters aren't the weight: each one is a thin translation layer with conformance tests, and they're the reason this plugs into your tools instead of replacing them. The weight is the dev-facing and operator-facing command surface that grew one command at a time, each addition individually reasonable. That's the part that gets pruned.

What your note actually changes is ordering. A first run that dies on a wrong error message costs more than any new feature is worth, so the first-use path goes before anything else.

On September — offer taken, if it still stands. The most useful shape is the one you already wrote in your own list: two of the tickets you had queued, a clean machine, and a note at every point where you stopped, including the ones that feel too small to write down. Those are the ones I can't see from here. I'll leave #3527 and #3528 open until then rather than guess at them.

[chernistry]

v3.14.159 is out, and most of what it contains started here.

The error that sent you after a flag that doesn't exist is fixed (#3526) — it names --cli now, and a test resolves every command and flag in that message against the live CLI, so it can't drift back. Your third point turned out to be a class of bug rather than one bug: the plan loader was accepting malformed fields as defaults, which is exactly how a run starts, produces nothing, and says nothing about it. It now raises a named error with the field (#3534, #3539). I still don't have your specific reproduction, so #3528 stays open until I can confirm it was that and not something else.

The reference docs are now gated in CI against the registered CLI — if a command or flag exists in only one of the two places, the build fails (#3511, #3536). Twenty-three commands were documenting flags the parser rejects. That's the "bots to update outdated docs" idea, though as a gate rather than a bot; a pass would drift again by the next release. The narrative pages you actually land on first are not covered yet, which is #3527 and the harder half.

Still open for the next one: #3527 (narrative drift), #3528 (your no-plan run), #3529 (TUI scrolling and first-glance layout). The README now says beta and solo-maintained, which it should have said a while ago.

One more thing, unrelated: we're putting the repo on Google Code Wiki so there's a browsable, always-current map of the codebase — a second entry point that can't go stale the way handwritten pages do. The request is pending on their side and appears to be demand-driven, so if you feel like nudging it along, it's about ten seconds: #3553 has the steps.

Thanks again. This one was worth the hour and a half you lost.