[Intel HPU] enable chunked prefill by fmiao2372 · Pull Request #5903 · PaddlePaddle/FastDeploy

fmiao2372 · 2026-01-06T05:18:04Z

Motivation

enable chunked prefill on intel hpu

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick，PR标题需遵循格式，在最开始加上[Cherry-Pick]标签，以及最后面加上原PR ID，例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

depend on PaddlePaddle/PaddleCustomDevice#2324

Modifications

hpu attention backend
hpu forward metadata
hpu model runner

Usage or Command

use these parameters to enable chunked prefill
--enable-chunked-prefill
--max-num-batched-tokens 4096

Accuracy Tests

ERNIE-4.5-21B-A3B-Paddle
Accuracy: 0.920
Invalid: 0.001
Latency: 370.744 s

Checklist

[Done] Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
[Done] Format your code, run pre-commit before commit.
[Done] Add unit tests. Please write the reason in this PR if no unit tests.
conducted by local tests
[Done] Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2026-01-06T05:18:10Z

Thanks for your contribution!

Copilot

Pull request overview

This PR enables chunked prefill support for Intel HPU platform, allowing prefill operations to be split into smaller chunks when processing long sequences alongside decode operations.

Key changes:

Enhanced HPU attention backend to support mixed encoder/decoder execution modes
Modified forward metadata structure to separate encoder and decoder state management
Added chunked prefill warmup and resource allocation logic

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
fastdeploy/worker/hpu_model_runner.py	Implements chunked prefill logic, adds mixed batch warmup, separates encoder/decoder metadata handling
fastdeploy/model_executor/layers/backends/intel_hpu/attention/hpu_attn_backend.py	Adds forward_mixed method to handle concurrent encoder/decoder batches, updates forward_extend and forward_decode to use separated metadata
fastdeploy/model_executor/forward_meta.py	Restructures HPUForwardMeta to maintain separate encoder/decoder state with dedicated fields for rotary embeddings, block metadata, and batch information
fastdeploy/engine/sched/resource_manager_v1.py	Adds HPU-specific token budget alignment logic to ensure chunk sizes are multiples of block_size

Copilot · 2026-01-06T05:18:55Z

fastdeploy/worker/hpu_model_runner.py

                self.share_inputs["prompt_lens"][idx : idx + 1] = len(input_ids)
                self.share_inputs["is_block_step"][idx : idx + 1] = False
-                # self.share_inputs["is_chunk_step"][idx : idx + 1] = prefill_end_index < len(input_ids)
+                self.share_inputs["is_chunk_step"][idx : idx + 1] = prefill_end_index < len(input_ids)


This line was previously commented out. Please ensure all corresponding references to is_chunk_step are properly updated and tested throughout the codebase, particularly in functions that consume this metadata.

not an issue

Copilot · 2026-01-06T05:18:55Z

fastdeploy/worker/hpu_model_runner.py

The variable name len shadows the built-in Python function. Consider renaming to seq_chunk_size or encoder_chunk_len for clarity and to avoid shadowing the built-in.

Suggested change

len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)

position = 0

for i in range(batch_ids_encoder.shape[0]):

encoder_id = batch_ids_encoder[i].item()

seq_len = seq_lens_encoder[encoder_id].item()

output_data[position] = tmp_out[i * len + seq_len - 1]

encoder_chunk_len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)

position = 0

for i in range(batch_ids_encoder.shape[0]):

encoder_id = batch_ids_encoder[i].item()

seq_len = seq_lens_encoder[encoder_id].item()

output_data[position] = tmp_out[i * encoder_chunk_len + seq_len - 1]

Copilot · 2026-01-06T05:18:56Z

fastdeploy/worker/hpu_model_runner.py

The index calculation i - total_batch_decoder appears incorrect. When i iterates from 0 to batch_ids_decoder.shape[0] - 1, this will produce negative indices for early iterations. The correct index should likely be total_batch_encoder * len + i.

Suggested change

len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)

position = 0

for i in range(batch_ids_encoder.shape[0]):

encoder_id = batch_ids_encoder[i].item()

seq_len = seq_lens_encoder[encoder_id].item()

output_data[position] = tmp_out[i * len + seq_len - 1]

position += 1

for i in range(batch_ids_decoder.shape[0]):

output_data[position] = tmp_out[i - total_batch_decoder]

block_len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)

position = 0

for i in range(batch_ids_encoder.shape[0]):

encoder_id = batch_ids_encoder[i].item()

seq_len = seq_lens_encoder[encoder_id].item()

output_data[position] = tmp_out[i * block_len + seq_len - 1]

position += 1

decoder_start = total_batch_encoder * block_len

for i in range(batch_ids_decoder.shape[0]):

output_data[position] = tmp_out[decoder_start + i]

not an issue

Copilot · 2026-01-06T05:18:56Z

fastdeploy/model_executor/layers/backends/intel_hpu/attention/hpu_attn_backend.py

In the forward_mixed method's measurement mode branch for decoder, forward_meta.rotary_embs is used instead of forward_meta.rotary_embs_decoder. This inconsistency with the non-measurement branch (line 683) will cause incorrect rotary embeddings to be applied during measurement mode.

Suggested change

forward_meta.rotary_embs,

forward_meta.rotary_embs_decoder,

Copilot · 2026-01-06T05:18:56Z

fastdeploy/model_executor/forward_meta.py

Line 376 sets forward_mode to ForwardMode.MIXED, then line 377-378 conditionally sets it again to the same value. The initial assignment on line 376 is redundant.

Suggested change

forward_mode = ForwardMode.MIXED

codecov-commenter · 2026-01-06T07:25:21Z

Codecov Report

❌ Patch coverage is 63.33333% with 11 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@ab553b3). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/engine/sched/resource_manager_v1.py	0.00%	6 Missing ⚠️
fastdeploy/model_executor/forward_meta.py	79.16%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #5903   +/-   ##
==========================================
  Coverage           ?   66.70%           
==========================================
  Files              ?      347           
  Lines              ?    44426           
  Branches           ?     6823           
==========================================
  Hits               ?    29636           
  Misses             ?    12609           
  Partials           ?     2181

Flag	Coverage Δ
GPU	`66.70% <63.33%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

zoooo0820

LGTM

EmmonsCurse

LGTM for skipping coverage.

[Intel HPU] enable chunked prefill

0bac5c6

Copilot AI review requested due to automatic review settings January 6, 2026 05:18

fmiao2372 temporarily deployed to Metax_ci January 6, 2026 05:18 — with GitHub Actions Inactive

paddle-bot bot added the contributor External developers label Jan 6, 2026

Copilot AI reviewed Jan 6, 2026

View reviewed changes

fix bug by copilot comments

63f0380

fmiao2372 had a problem deploying to Metax_ci January 6, 2026 05:49 — with GitHub Actions Failure

fmiao2372 mentioned this pull request Jan 6, 2026

[Intel HPU] enable chunked prefill PaddlePaddle/PaddleCustomDevice#2324

Merged

zoooo0820 approved these changes Jan 6, 2026

View reviewed changes

EmmonsCurse approved these changes Jan 6, 2026

View reviewed changes

EmmonsCurse merged commit 1ee285c into PaddlePaddle:develop Jan 6, 2026
15 of 20 checks passed

-    len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)
-    position = 0
-    for i in range(batch_ids_encoder.shape[0]):
-        encoder_id = batch_ids_encoder[i].item()
-        seq_len = seq_lens_encoder[encoder_id].item()
-        output_data[position] = tmp_out[i * len + seq_len - 1]
+    encoder_chunk_len = int((tmp_out.shape[0] - total_batch_decoder) / total_batch_encoder)
+    position = 0
+    for i in range(batch_ids_encoder.shape[0]):
+        encoder_id = batch_ids_encoder[i].item()
+        seq_len = seq_lens_encoder[encoder_id].item()
+        output_data[position] = tmp_out[i * encoder_chunk_len + seq_len - 1]

Conversation

fmiao2372 commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot bot commented Jan 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

fmiao2372 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

fmiao2372 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

fmiao2372 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

fmiao2372 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

fmiao2372 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jan 6, 2026

Codecov Report

Uh oh!

zoooo0820 left a comment

Choose a reason for hiding this comment

Uh oh!

EmmonsCurse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

fmiao2372 commented Jan 6, 2026 •

edited

Loading