Plan commitment

[DanEble]

The todos tool is too credulous. The LLM can do whatever it wants to the list—even replace it with an entirely different list without completing any of the tasks. I find myself watching it to make sure that it doesn't mark five tasks done at once without actually doing them. Or I return after some time and have to bring its sins to remembrance and tell it to backtrack.

What if the LLM could invoke a tool to protect the to-do list (and other planning tools that might be added in the future)? Once the plan is protected, removing todos and changing the status in suspicious ways would require approval.

[DanEble]

FYI @vadiminshakov. I noticed that you have been contributing in the area of planning and interaction, and you mentioned using weak models, which was part of my situation when I posted this idea.

[vadiminshakov]

This feature #2948

[DanEble]

I don't see changes to the todos tool in that PR. As the goal tool is described in the PR, it's not obvious to me that having a goal would do any good if the LLM hallucinated completion of a large number of tasks in one shot. What are you thinking in more detail? Would you phrase a goal like <Do my task> and if you ever finish > 5 tasks in one shot, review them to make sure you actually did them and didn't hallucinate?

[vadiminshakov]

I’m simply saying that, by design, the goal runtime forces the model to satisfy some condition. It’s extremely difficult to come up with a mechanism that would make it execute todo items just as meticulously.

If changing the todo list is prohibited, that still doesn’t protect against simply marking all items as completed and opening a new list. I just don’t see any way to make TODO a stricter mechanism without creating problems for the user

[DanEble]

If changing the todo list is prohibited, that still doesn’t protect against simply marking all items as completed and opening a new list.

I think we're talking past each other. If the harness pushes back when the LLM tries to mark > $n$ items done in one invocation of the tool, then it makes it more likely that the LLM will "notice" that it has hallucinated and rectify the situation, doesn't it? It seems that you are defining "protection" as elimination of the slightest mistake. Perfection would be a straw man for what I have actually suggested.

[vadiminshakov]

@DanEble yes, I misunderstood the original point. So you’re suggesting that if the model marks more than a certain number of list items as completed in a single call, we should treat that as invalid and tell the model that it needs to proceed step by step?

That would probably work, yes.