Commits on Jul 21, 2023

1. assist: Refactor token counting …

   With the actor model, tokens can be used in multiple ways (picking
   tools, invoking them, ...), which don't necessarily end up in a final
   action (sometimes we return a nextStep instead). Streaming responses
   were another challenge: the agent returned without the completion being
   over (it returned a routine streaming the deltas sent by the model).
   
   This PR introduces a TokenCounter interface that abstracts synchronous
   and asynchronous token counting. All token-consuming operations must
   return a TokenCounter. TokensCounters are stored in the agent state and
   returned once the agent exists. Finally, the token counters are
   evaluated asynchronously to give the streaming completion requests
   enough time to finish.

   

   hugoShaka authored and github-actions committed Jul 21, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 26a4bfc
   • Browse repository at this point

   Copy the full SHA

   26a4bfc View commit details

   Browse the repository at this point in the history