[General] Is v0.8.1 usable at all?

[tkellogg]

Area

Development

Topic

It seems like the stability of the harness has progressively gotten worse. It's to the point where the harness won't stay alive more than 5 minutes. People are creating feature requests as if this isn't a problem for them. How?

What have you tried or considered?

No response

Additional context

No response

[alvinycheung]

nah, yeah it's pretty borked. It's not very usable. I think the v0.7.x is more stable.

[alexanderkjeldaas]

if you mean the ipython kernel, yes it crashes regularly for me. you continuously get the question whether to kill and restart it, or wait. that it received ctrl+c after a minute or so. is that what you mean or something else?

the tui interface itself stays up for me. also the daemon is stable and stays up for me. I do have my own nix flake packaging of it as installing things through npm these days is a bit crazy security wise though.

[tkellogg]

yeah, the TUI stays up, but the sessions get corrupted until you restart the kernel. Like globally, they all lock up and get corrupted at the same time.

[alexanderkjeldaas]

I have not experienced that exact failure mode, but I have experienced:

1. Lots of empty sessions. A session is created every time you start, even if you leave that session for one of the existing ones immediately. This is spamming the sessions list.
2. Empty sessions that are not possible to delete because they are somehow activated or 'idle' even if they are in the 'inactive' state.
3. Individual sessions that are lost and cannot be opened until you restart the cli.

[tkellogg]

I fixed the stability issues, I think. I rearchitected and rewrote a lot of how communication happens between sessions, TUIs and the supervisor, also the agent view.

https://github.com/tkellogg/prime-agent/tree/feat/direct-worker-transport

Fix 1: Supervisor only supervises

The supervisor was previously doing more than supervision, it was a whole-ass message broker. Basically anything that happened in any of the sessions went through this one single supervisor. But also, the supervisor had critical responsibilities, like ensuring that sessions are cleaned up, dead sessions are replaced, etc. The combo of both critical control plane traffic + less valuable data plane traffic meant that one single burst anywhere in any of the sessions could ripple through and take down all sessions.

My fix is to take a Tailscale-inspired approach, where the global supervisor becomes merely a registry where sessions can find TUIs and other sessions, while actual data plane traffic goes direct session->TUI without the global choke point.

Fix 2: SQLite for agent view

Previously, the agents view was very slow because it had to ping each session and get a response back in realtime in order to populate. If a session was slow/hung, then that item might not even populate. Worse, this all aggravated the supervisor-as-choke-point problem illustrated above.

My fix was to store the agents view in a combo of SQLite & jsonl files. The SQLite quickly lists all sessions, while the JSONL files hold more detailed information about the history of the session. Upon agents view read, it just scans the SQLite table, checks JSONL timestamps, and only reads JSONL files that changed.

The benefit of this is that slowness in any of the sessions doesn't ripple into any other session or even the agents view. They're isolated from each other.

Testing

I load tested it with 40 agents, flawless, fast. It used to be I could barely run 3 subagents. I did another load test with 16 sessions, each with 4 subagents, all communicating between each other. No issues at all.

idk, i think i just made it stable. Really hoping we can get this into the core.

Related issues

Some issues that are either directly or partially addressed by this change: #1870 #1641 #1381 #648

Issues that are related, but maybe not addressed: #821 #1654 #1229 #1199 #1200 #1297 #1374 #1417 #1571 #1643 #1902 #1905

I'm not an authorized contributor, so I can't make a pull request directly, but I'll definitely be using my fork until this gets merged & distributed.