Skip to content

[FDConfig]Turn on the CUDAGraph + RL switch #4508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 23, 2025
Merged

[FDConfig]Turn on the CUDAGraph + RL switch #4508

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fdba041
Turn on the CUDAGraph + RL switch
gongshaotian Oct 21, 2025
0a94046
merge develop
gongshaotian Oct 21, 2025
e945d4e
reduce max_num_seqs and number of request
gongshaotian Oct 22, 2025
bd28e8b
Merge branch 'develop' of https://github.com/PaddlePaddle/FastDeploy …
gongshaotian Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions fastdeploy/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1510,9 +1510,7 @@ def postprocess(self):
self.structured_outputs_config.guided_decoding_backend = "xgrammar"

# Adjustment GraphOptConfig
if (self.scheduler_config.splitwise_role != "mixed") or (
self.load_config is not None and self.load_config.dynamic_load_weight is True
):
if self.scheduler_config.splitwise_role != "mixed":
self.graph_opt_config.use_cudagraph = False
logger.info(
"CUDAGraph does not support to be started together with PD Disaggregation temporarily, but has been automatically closed!"
Expand Down Expand Up @@ -1630,11 +1628,12 @@ def check(self):
self.scheduler_config.check()

# Check graph optimization config
if self.graph_opt_config.graph_opt_level > 0 or self.graph_opt_config.use_cudagraph:
if self.graph_opt_config.graph_opt_level > 0:
if self.load_config is not None:
assert (
self.load_config.dynamic_load_weight is False
), "Static graph cannot be used in RL scene temporarily"

if int(envs.ENABLE_V1_KVCACHE_SCHEDULER) == 1:
assert (
int(envs.FD_DISABLED_RECOVER) == 0
Expand Down
2 changes: 1 addition & 1 deletion tests/ce/stable_cases/launch_model.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ python -m fastdeploy.entrypoints.openai.api_server \
--cache-queue-port ${FD_CACHE_QUEUE_PORT} \
--quantization wint8 \
--max-model-len 32768 \
--max-num-seqs 256 \
--max-num-seqs 1 \
--gpu-memory-utilization 0.9 \
--model "$MODEL_PATH" \
--load-strategy ipc_snapshot \
Expand Down
2 changes: 1 addition & 1 deletion tests/ce/stable_cases/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ PORT="${FD_API_PORT}" # 这里需要配合启动脚本那个URL PORT
BASE_URL="http://$HOST:$PORT"

TOTAL_ROUNDS=30
CHAT_REQUESTS_PER_ROUND=5
CHAT_REQUESTS_PER_ROUND=1
export CUDA_VISIBLE_DEVICES=0,1
MAX_MEMORY_MB=10240 # 10GB

Expand Down