Anti hallucination measures

[michaelneale]

from #9720

moving to a discussion as this doesn't have a clear solution yet and need to throw around some ideas. Thanks for raising this @agault45

[agault45]

"In goose we have an adversarial model/agent which is to prevent bad actions against policy (and runs behind the scenes if configured)."

Are you referring to Sandpit here, @michaelneale ?

[agault45]

A couple possibilities: https://docs.google.com/document/d/1csZS1Ue8nqXWXVwjcSfdu9hEhnJDudeqhbyGcpBNCeM/edit?tab=t.0#heading=h.vsahjl6ua43a

[agault45]

I don't have strong opinions about the actual mechanism that gets implemented, so long as we consider some mechanism. The cost of a couple hundred tokens or a couple extra seconds of compute time is tiny compared to the cost of making a million dollar planning decision on fabricated data.

[michaelneale]

@agault45 with hooks and so on - would be interesting to have something native, the key is to sense if there is something factual or data driven, and then look back to see if a tool call was made, if the tool call was successful (I think that is critical, as what can happen is it fails to get the data and kind of gives up but then forgets it needs to be honest). There will be no deterministic solution here but layers of non deterministic things.

[agault45]

Caught Goose lying to me again this morning @michaelneale. Can't help but wonder how many fake numbers are making their way into business strategy/performance docs out there. Any more ideas on some kind of native safeguard?

[michaelneale]

yeah some of this is really fundamental to LLMs, not agents though, and varies a LOT with models and how they are trained and post trained etc. Usually I don't think it is a great idea to take the output at face value - ideally it is output that van be validated (NP problems) vs just qualitative output (sadly that is not what a lot of slop machines like openclaw are doing to the world).

In this case there really could be ways to know that tool calls were done or not at least which provides some backpressure that it did work? as in that case, it just failed to get anything back (which could be detected)