Skip to content

[llava][14/N] Refactor runner prefill() and run_model_step() #4556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
facebook-github-bot merged 9 commits into from
Aug 9, 2024
Merged

[llava][14/N] Refactor runner prefill() and run_model_step() #4556

facebook-github-bot merged 9 commits into from
Aug 9, 2024

Conversation

@larryliu0820
Copy link
Contributor

@larryliu0820 larryliu0820 commented Aug 6, 2024 •
edited
Loading

Stack from ghstack (oldest at bottom):

This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

  • prefill():
    For the case where parallel prefill is enabled or not using kv cache,
    the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

  • run_model_step():
    This function should not update the input. Instead it should run the
    model differently, depending on whether kv cache is enabled. This should
    return the next token directly. All the input update needs to happen in
    the generation loop.

Differential Revision: D60840327

@larryliu0820
[llava][14/N] Refactor runner prefill() and run_model_step()
2dad738 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]
@larryliu0820 larryliu0820 mentioned this pull request Aug 6, 2024
Closed
@pytorch-bot
Copy link

pytorch-bot bot commented Aug 6, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4556

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4ff3a90 with merge base 92edd04 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][14/N] Refactor runner prefill() and run_model_step()
ee5d0bb 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

ghstack-source-id: e0f4f74
Pull Request resolved: #4556
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 6, 2024
@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
94290d1 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][14/N] Refactor runner prefill() and run_model_step()
f88f7fe 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

ghstack-source-id: 62eb4f6
Pull Request resolved: #4556
@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
188a000 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][14/N] Refactor runner prefill() and run_model_step()
ac738ad 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

ghstack-source-id: b4a1220
Pull Request resolved: #4556
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 6, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
aa08c52 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][15/N] Extract out text decoder runner
d963403 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

[ghstack-poisoned]
@larryliu0820 larryliu0820 mentioned this pull request Aug 6, 2024
Merged
larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][15/N] Extract out text decoder runner
53a5494 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

ghstack-source-id: 72cdfbc
Pull Request resolved: #4567
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 6, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
7b3c6d8 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 6, 2024
@larryliu0820
[llava][15/N] Extract out text decoder runner
ae65051 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

ghstack-source-id: 2628a3e
Pull Request resolved: #4567
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 6, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
dd5b370 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
@larryliu0820 larryliu0820 mentioned this pull request Aug 7, 2024
Merged
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
5a43186 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
a1e6c5e 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 7, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
6423450 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
aa929d6 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
de38ad7 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 7, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
c213247 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
46fbce1 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820 larryliu0820 mentioned this pull request Aug 7, 2024
Merged
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
18a92cc 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
94903ed 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 7, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
be1bab9 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
306b656 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 7, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
0cf1345 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 7, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
742b102 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 8, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
24eb005 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 8, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
ad1647d 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 8, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@larryliu0820
Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"
4ff3a90 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 8, 2024
@larryliu0820
Update base for Update on "[llava][15/N] Extract out text decoder run…
8a2b3f1 
…ner"

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this pull request Aug 8, 2024
@larryliu0820
Update on "[llava][15/N] Extract out text decoder runner"
e8da443 
Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]
@larryliu0820
Copy link
Contributor Author

larryliu0820 commented Aug 8, 2024

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot facebook-github-bot merged commit 886ccce into gh/larryliu0820/43/base Aug 9, 2024
larryliu0820 added a commit that referenced this pull request Aug 9, 2024
@larryliu0820
Gh/larryliu0820/46/base (#4643)
82608bf 
* [llava][13/N] Move metadata util to a separate header for reuse

As titled.

[ghstack-poisoned]

* Update on "[llava][13/N] Move metadata util to a separate header for reuse"


As titled.

[ghstack-poisoned]

* Update on "[llava][13/N] Move metadata util to a separate header for reuse"


As titled.

[ghstack-poisoned]

* Update on "[llava][13/N] Move metadata util to a separate header for reuse"


As titled.

[ghstack-poisoned]

* Update on "[llava][13/N] Move metadata util to a separate header for reuse"


As titled.

[ghstack-poisoned]

* [llava][14/N] Refactor runner prefill() and run_model_step()

This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]

* Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"


This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]

* Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"


This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

[ghstack-poisoned]

* [llava][15/N] Extract out text decoder runner

Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

[ghstack-poisoned]

* Update on "[llava][14/N] Refactor runner prefill() and run_model_step()"


This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

Differential Revision: [D60840327](https://our.internmc.facebook.com/intern/diff/D60840327)

[ghstack-poisoned]

* Update on "[llava][15/N] Extract out text decoder runner"


Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]

* Update base for Update on "[llava][15/N] Extract out text decoder runner"


Last PR #4556 refactored run_model_step() so that it is suitable to be
extracted out as a separate class.

This new `TextDecoderRunner` provides 2 APIs:

* step(tokens, start_pos)
This API takes one or more tokens with start_pos and feed them into Module. Return a tensor of logits.

* logits_to_token(logits)
This API samples the result and returns a token. We don't expect this
logic to change across different runners.

Differential Revision: [D60856571](https://our.internmc.facebook.com/intern/diff/D60856571)

[ghstack-poisoned]

* [llava][16/N] Extract out prefill logic into a new class

Depends on whether parallel or sequential prefill is chosen, prefill()
calls `TextDecoderRunner.step()` to prefill prompt tokens to LLM.

[ghstack-poisoned]

* Update base for Update on "[llava][16/N] Extract out prefill logic into a new class"


Depends on whether parallel or sequential prefill is chosen, prefill()
calls `TextDecoderRunner.step()` to prefill prompt tokens to LLM.

Differential Revision: [D60927756](https://our.internmc.facebook.com/intern/diff/D60927756)

[ghstack-poisoned]

* Update base for Update on "[llava][16/N] Extract out prefill logic into a new class"


Depends on whether parallel or sequential prefill is chosen, prefill()
calls `TextDecoderRunner.step()` to prefill prompt tokens to LLM.

Differential Revision: [D60927756](https://our.internmc.facebook.com/intern/diff/D60927756)

[ghstack-poisoned]

* Update base for Update on "[llava][16/N] Extract out prefill logic into a new class"


Depends on whether parallel or sequential prefill is chosen, prefill()
calls `TextDecoderRunner.step()` to prefill prompt tokens to LLM.

Differential Revision: [D60927756](https://our.internmc.facebook.com/intern/diff/D60927756)

[ghstack-poisoned]

* Update base for Update on "[llava][17/N] Move util.h into /e/llm/runner"


So that it can be reused

Differential Revision: [D60938984](https://our.internmc.facebook.com/intern/diff/D60938984)

[ghstack-poisoned]

* Update base for Update on "[llava][17/N] Move util.h into /e/llm/runner"


So that it can be reused

Differential Revision: [D60938984](https://our.internmc.facebook.com/intern/diff/D60938984)

[ghstack-poisoned]

* [llava][17/N] Move util.h into /e/llm/runner

Differential Revision: D60938984

Pull Request resolved: #4588
kedarnath03 pushed a commit to kedarnath03/executorch that referenced this pull request Jun 25, 2025
@larryliu0820
[llava][14/N] Refactor runner prefill() and run_model_step()
04c7998 
This refactoring is needed in order to extract out prefill() and
run_model_step() out from runner so that these APIs become replaceable
and easy to plugin and use.

* prefill():
For the case where parallel prefill is enabled or not using kv cache,
the model is able to accept a large block (more than 1) of tokens.

For the other case where we have kv cache but parallel prefill is not
enabled, we can only feed in 1 token every time.

* run_model_step():
This function should not update the input. Instead it should run the
model differently, depending on whether kv cache is enabled. This should
return the next token directly. All the input update needs to happen in
the generation loop.

ghstack-source-id: 303c242
Pull Request resolved: pytorch/executorch#4556
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@helunwencser helunwencser helunwencser approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@larryliu0820 @helunwencser @facebook-github-bot