responseCallback when loading the model

[DarkSorrow]

Description

Hello,

I'm trying to use the responseCallback from the module

        await LLMModule.load({
          modelSource: getFilePath(modelURI),
          tokenizerSource: getFilePath(tokenizerURI),
          tokenizerConfigSource: getFilePath(tokenizerConfigURI),
          responseCallback: (token: string) => {
            console.log('token ***', token);
            console.log('*******')
            if (!didCancel && internalIsGenerating ) { 
              onTokenReceived(token);
            }
          },
        });

This is what i have and the log im getting is as such

{...}
 LOG  *******
 LOG  token *** <think>
Okay, the user
 LOG  *******
 LOG  token *** <think>
Okay, the user is
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking,
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking, "
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking, "Ok
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking, "Ok how
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking, "Ok how old
 LOG  *******
 LOG  token *** <think>
Okay, the user is asking, "Ok how old are
 LOG  *******

i thought the responseCallback was actually giving token per token answers but not accumulating it like the normal useLLM function was doing.
My goal is to try to parse the token on the fly so that i can change the UI, i'm depending on token so i can make direct equality like token === "" since i will know in advance the tag.

Also this is more a question but couldn't that JS callback be on the generate itself?
When i use the API of oai compatible cloud model i get the streaming of token on calling the completion endpoint and it is the same for the llamacpp libraries. The token callback is on the "completion" call which correspond to this library generate.
I'm saying this because the token parsing is often link to the display we are doing on the app and its often a function that depend on various state that can change and it seems that even when i use a useCallback im not getting the token sent to that function

Steps to reproduce

1. Load model with module
2. console.log the token

Snack or a link to a repository

No response

React Native Executorch version

0.4.2

React Native version

0.79

Platforms

iOS

JavaScript runtime

Hermes

Workflow

None

Architecture

None

Build type

None

Device

None

Device model

Simulator

AI model

No response

Performance logs

No response

Acknowledgements

Yes

[DarkSorrow]

So actually it's my mistake i guess the response callback is the same as when we use the useLLM, there is no way to get the result in a token per token to parse things ourselves right?

[pweglik]

Hey, right now there's no way without changing libraries internals. If you know what tokens you are expecting you can simply check the end of response field like this (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith):

const response = "Cats are the best!";

console.log(response.endsWith("best!"));

We will discuss whether we keep current API or make it per token callback and allow users to accumulate it themselves.

[DarkSorrow]

Yeah i was thinking of checking the end as well but since there could be different token and variant of "think", "tool_call" it can impact performance a lot. The solution i was thinking right now to have a minimum impact was more to just check if the last token added has a closing '>'. Then i can compare to a list of tag but this still makes me feel a bit odd and coding is about feelings right :D.

But i think it could be good to have the per token so that people can customize their ui more based on what they expect and in a way its a bit more standard with what other libraries / remote endpoint give. Also i think it should be done on the generate endpoint too and maybe this way it wouldn't impact the current library interface that much as it could be an additional arguments of the generate :x

And we can close this ticket if you are fine or keep it open to discuss the subject with more people maybe? (tho there is the discussion part for this)

[pweglik]

Thanks for your input! Let's keep it open, until we decide how to continue. There's high chance we go with what you propose in 0.5.0 and change responseCallback into tokenCalbback, or simply add tokenCallback that will return separate tokens after they are created

[DarkSorrow]

I think having the responseCallback accumulate the token can be fine for people that don't want to parse the content in real time to display things on the UI

But for the real time parsing i still think the callback shouldn't be given at the initialization of the model because it gets a bit messy to make the UI and the model interact well due to race condition with the events. Right now i have to create some extra state to make sure i put the streamed token in the correct message array but it would be a lot more simple to have the streamed function be related to the 'generation' being called. Maybe also some of the options we give the LLM could be passed to that function in the parameters like the temperature and such since i was reading that for some model when you pass tools or even give the option to '/not_think' there are some parameters that can be better suited to pass down to the llm

[pweglik]

Hey, we've decided to add onTokenCallback to LLMModule and hook instead of ResponseCallback. We will depracate responseCallback in 0.4.x and remove it in 0.5.0

[pweglik]

Hey, we've added tokenCallback in v0.4.5. Feel free to test and see if it works for you! We've also added a way to set new callback in LLMModule. See docs: https://docs.swmansion.com/react-native-executorch/docs/typescript-api/LLMModule