Problem serving nvidia/DeepSeek-V3-0324-FP4 on 8xH200

[0xshawn]

Hi,

I'm trying to deploy huggingface nvidia/DeepSeek-V3-0324-FP4 on 8xH200 server. Both trtllm-serve, vLLM, and sglang are not able to serve this model because of the GPU arch not support this model.

I noticed that the supported device in README of huggingface is:

Inference:
Engine: TensorRT-LLM
Test Hardware: B200

So, my questions is:

1. how could I run this model on NVIDIA H200?
2. If not, is there any document that I can convert model of deepseek v3 0324 myself to run FP4 on H200?

Thank you!

[troycheng]

Supported Hardware Microarchitecture Compatibility:NVIDIA Blackwell

H200 is based on hopper architecture

[0xshawn]

@troycheng Thanks for your reply

[0xshawn]

Hi @troycheng,

Can I quantize the deepseek v3 0324 to FP4 by myself with TensorRT to support NVIDIA hopper architecture ?

Thank you!

[troycheng]

Hi @troycheng,

Can I quantize the deepseek v3 0324 to FP4 by myself with TensorRT to support NVIDIA hopper architecture ?

Thank you!

You can refer to this answer, "Hopper doesn't have HW-supported FP4"

https://huggingface.co/nvidia/DeepSeek-R1-FP4/discussions/8