Do not let chunked prefill generate decode logprobs#3777
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
santhnm2
left a comment
santhnm2
left a comment
There was a problem hiding this comment.
Can we add a unit test for this?
tdene
added
the
Expert Review
| request.prompt_log_probs = [] | ||
| request.generated_log_probs = [] | ||
| is_chunked_prefill = request_id == self.context.chunked_prefill_request_id | ||
| is_prefill = len(request_log_probs) > 1 |
There was a problem hiding this comment.
I believe we have scheduling code to prevent prefill requests from having a single token, but would be good to double check that there's no edge case here
There was a problem hiding this comment.
We guarantee prefill requests have more than one token.
We do not, in fact, guarantee that chunked prefill requests have more than one token. But that does not affect our logic here.
I left a comment for future code editors to understand.
There was a problem hiding this comment.
Comment has been invalidated by a new merge with main.
Everything is good and makes sense after merging with main.
Phlip79
removed
Expert Review
|
@tdene FYI we don't use the "Expert Review" label anymore |
|
/claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
tdene
force-pushed
the
tde/fix_chunked_prefill_bug
branch
from
d779897 to
a404777
Compare
tdene
changed the title
tdene
force-pushed
the
tde/fix_chunked_prefill_bug
branch
from
a404777 to
9db621d
Compare
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/23009338011 |
5 participants
What does this PR do ?
Contribution process
Pre-checks
Code review
Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!
All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.
Step 1: Mark PR as "Ready for Review"
.github/CODEOWNERS.Final Review might get declined if these requirements are not fulfilled.
Step 2: Final Review
For PRs that change
megatron/core, once all expert reviewers have approved, theFinal Reviewlabel is applied automatically and final reviewers are assigned.For PRs outside
megatron/core, this step is skipped.Step 3: Approved
Once all required reviewers have approved, the
Approvedlabel is applied automatically.Merge
Any member of mcore-engineers will be able to merge your PR.
For MRs into `dev` branch
The proposed review process for `dev` branch is under active discussion.MRs are mergable after one approval by either
eharper@nvidia.comorzijiey@nvidia.com.