Skip to content

Make token and stat callback optional in multimodal runner #13872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 4, 2025
Merged

Make token and stat callback optional in multimodal runner #13872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
33aa591
Make token and stat callback optional in multimodal runner
jackzhxng Sep 2, 2025
13409cc
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Sep 3, 2025
84527ff
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Sep 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions extension/llm/runner/multimodal_runner.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Error MultimodalRunner::load() {
Error MultimodalRunner::generate(
const std::vector<MultimodalInput>& inputs,
const GenerationConfig& config,
std::function<void(const std::string&)>& token_callback,
std::function<void(const Stats&)>& stats_callback) {
std::function<void(const std::string&)> token_callback,
std::function<void(const Stats&)> stats_callback) {
if (inputs.empty()) {
ET_LOG(Error, "MultimodalInput vector cannot be empty");
return Error::InvalidArgument;
Expand Down
4 changes: 2 additions & 2 deletions extension/llm/runner/multimodal_runner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@ class ET_EXPERIMENTAL MultimodalRunner {
virtual ::executorch::runtime::Error generate(
const std::vector<MultimodalInput>& inputs,
const GenerationConfig& config,
std::function<void(const std::string&)>& token_callback,
std::function<void(const Stats&)>& stats_callback);
std::function<void(const std::string&)> token_callback = {},
std::function<void(const Stats&)> stats_callback = {});

inline void stop() {
text_token_generator_->stop();
Expand Down
6 changes: 3 additions & 3 deletions extension/llm/runner/text_token_generator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ class ET_EXPERIMENTAL TextTokenGenerator {

/**
* Token generation loop.
* @param tokens prompt tokens as well as the first token generated by
* prefill.
* @param start_pos the start position of the new tokens, based on how many
* @param tokens The first token generated by prefill, if using kv cache. Else
* the prompt tokens + the first token generated by prefill.
* @param start_pos The start position of the new tokens, based on how many
* prompt tokens is prefilled.
* @param max_new_tokens Maximum number of new tokens to generate.
* @param temperature controls the randomness of predictions by scaling the
Expand Down
Loading