1.5.0.dev20260812001218

Installation

Via PyPI

pip install tt-forge==1.5.0.dev20260812001218 --extra-index-url https://pypi.eng.aws.tenstorrent.com/

Via Docker

docker pull ghcr.io/tenstorrent/tt-forge-slim:1.5.0.dev20260812001218

Dependency commits

tt-xla commit: 3cfcfcfdefac5e310f9864c82548021636e18771
tt-mlir commit: da6f1ce081af95630f5d679392e50a4bafa7c6ea
tt-metal commit: 5beed318d0f0d1c6212e605947fb0be80c9e0a1d

What's Changed

• Uplift third_party/tt_forge_models to 7e486b570e422aa5da81fc10074abd68e7d3b8cb 2026-08-08 by @vvukomanTT in #1024

Full Changelog: 1.5.0.dev20260808000817...1.5.0.dev20260812001218

---

LLM Performance

Model	Token/sec/user	Batch	Token/sec	ttft (ms)	Hardware
pytorch_Falcon_3_10B_Base_nlp_causal_lm_huggingface	41.0	32	1312.0	870.12	p150
pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface	54.0	32	1728.0	696.51	n150
pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface	101.0	32	3232.0	300.04	p150
pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface	36.0	32	1152.0	851.15	n150
pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface	67.0	32	2144.0	382.85	p150
pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface	18.0	32	576.0	1242.77	n150
pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface	35.0	32	1120.0	514.09	p150
pytorch_GPT-OSS_120B_nlp_causal_lm_huggingface	4.0	8	32.0	1418.64	p150
pytorch_GPT-OSS_20B_nlp_causal_lm_huggingface	8.0	64	512.0	7026.75	n150
pytorch_GPT-OSS_20B_nlp_causal_lm_huggingface	21.0	1	21.0	305.39	p150
pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface	38.0	32	1216.0	642.27	n150
pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface	74.0	32	2368.0	239.31	p150
pytorch_Llama_3.1_70B_Instruct_nlp_causal_lm_huggingface	7.0	32	224.0	9366.03	n150
pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface	21.0	32	672.0	1280.98	n150
pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface	51.0	32	1632.0	661.59	p150
pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface	64.0	32	2048.0	572.82	n150
pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface	124.0	32	3968.0	242.2	p150
pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface	30.0	32	960.0	618.76	n150
pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface	54.0	32	1728.0	283.61	p150
pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface	20.0	32	640.0	1242.85	n150
pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface	35.0	32	1120.0	585.28	p150
pytorch_Mistral_Small_24B_INSTRUCT_2501_nlp_causal_lm_huggingface	29.0	32	928.0	912.01	p150
pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface	20.0	32	640.0	668.72	n150
pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface	37.0	32	1184.0	321.18	p150
pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface	20.0	32	640.0	655.9	n150
pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface	37.0	32	1184.0	326.37	p150
pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface	8.0	32	256.0	1505.44	n150
pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface	20.0	32	640.0	682.5	p150
pytorch_Qwen 2.5 Coder_32B_Instruct_nlp_causal_lm_huggingface	17.0	32	544.0	1492.51	p150
pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface	69.0	32	2208.0	416.33	n150
pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface	126.0	32	4032.0	170.84	p150
pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface	36.0	32	1152.0	487.75	n150
pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface	63.0	32	2016.0	201.74	p150
pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface	30.0	32	960.0	684.4	n150
pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface	58.0	32	1856.0	293.35	p150
pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface	16.0	32	512.0	832.8	n150
pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface	28.0	32	896.0	352.74	p150
pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface	48.0	32	1536.0	1177.48	n150
pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface	96.0	32	3072.0	553.07	p150
pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface	36.0	32	1152.0	739.98	n150
pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface	65.0	32	2080.0	337.18	p150
pytorch_Qwen 3_32B_nlp_causal_lm_huggingface	17.0	32	544.0	1852.34	p150
pytorch_Qwen 3_4B_nlp_causal_lm_huggingface	23.0	32	736.0	983.15	n150
pytorch_Qwen 3_4B_nlp_causal_lm_huggingface	41.0	32	1312.0	453.98	p150
pytorch_Qwen 3_8B_nlp_causal_lm_huggingface	16.0	32	512.0	1619.65	n150
pytorch_Qwen 3_8B_nlp_causal_lm_huggingface	30.0	32	960.0	738.18	p150

Non-LLM Performance

Model	Batch	Sample/sec	Hardware
Wan2.2-I2V-A14B-DiT	1	0.0	p150
Wan2.2-I2V-A14B-UMT5-Text-Encoder	1	12.0	p150
Wan2.2-I2V-A14B-VAE-Decoder	1	0.0	p150
Wan2.2-I2V-A14B-VAE-Encoder	1	1.0	p150
flux1-dev	1	0.0	p150
flux2	1	0.0	p150
glm-image	1	0.0	p150
hunyuan-image-2.1	1	0.0	p150
janus-pro-1b	1	0.0	n150
janus-pro-1b	1	0.0	p150
janus-pro-7b	1	0.0	p150
playground-v2.5	1	0.0	n150
playground-v2.5	1	0.0	p150
pytorch_BERT_emrecan/bert-base-turkish-cased-mean-nli-stsb-tr_nlp_embed_gen_huggingface	8	159.0	n150
pytorch_BGE-M3_Base_nlp_embed_gen_custom	4	9.0	n150
pytorch_BGE-M3_Base_nlp_embed_gen_custom	4	17.0	p150
pytorch_EfficientNet_Timm_B0_cv_image_cls_timm	8	346.0	n150
pytorch_EfficientNet_Timm_B0_cv_image_cls_timm	8	793.0	p150
pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom	32	14154.0	n150
pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom	32	29631.0	p150
pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub	12	1168.0	n150
pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub	12	2785.0	p150
pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface	32	46.0	n150
pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface	32	97.0	p150
pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface	8	1310.0	n150
pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface	8	2764.0	p150
pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface	1	37.0	n150
pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface	1	83.0	p150
pytorch_Swin_S_cv_image_cls_torchvision	1	10.0	n150
pytorch_Swin_S_cv_image_cls_torchvision	1	22.0	p150
pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface	1	5.0	n150
pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface	1	9.0	p150
pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github	1	137.0	n150
pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github	1	237.0	p150
pytorch_VGG19-UNet_base_cv_image_seg_custom	1	153.0	n150
pytorch_VGG19-UNet_base_cv_image_seg_custom	1	309.0	p150
pytorch_ViT_Base_cv_image_cls_huggingface	8	234.0	n150
pytorch_ViT_Base_cv_image_cls_huggingface	8	563.0	p150
pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm	8	744.0	n150
pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm	8	1549.0	p150
sdxl-lightning	1	0.0	n150
sdxl-lightning	1	0.0	p150
zimage	1	0.0	p150

Model coverage

Info: Full list of supported models is available in the assets section.

Model task	Model architecture	Model variant	Model framework	Inference	Training	n150	n300	p150	Single device	Data parallel	Tensor parallel	Model source
cv image cls	DLA	169	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv keypoint det	HRNet	W30	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	MLP-Mixer	Base	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	RegNet	Y 160	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	SegFormer	Mit B1	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
mm masked lm	ViLT Masked LM	Mlm	pytorch	✅	❌	✅	❌	❌	✅	❌	❌	View Source
cv object det	CenterNet	ResNet18 Backbone COCO	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	EfficientDet	D6	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp qa	Fuyu	8B	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv keypoint det	HRNet	W18 Small v2	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv keypoint det	HRNet	W40	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	Llama	3.0 8B	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image fe	MGP-STR Base	base	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	ResNeXt	14 32x4d Osmr	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	EfficientNet	B4	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	EfficientNet	B5	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	HarDNet	hardnet68	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	MobileNetV1	Mobilenet v1	pytorch	✅	❌	❌	✅	❌	❌	✅	❌	View Source
cv image cls	ResNet	ResNet50 TIMM High Resolution	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image seg	Ultra-Fast Lane Detection	TuSimple ResNet34 Backbone	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	VGG	Timm Vgg19 Bn	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	YOLOX	S	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	GPT-Neo	2 7B	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	VoVNet	57	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	YOLO-World	Medium 1280	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	DeiT	Tiny	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp text cls	GPT-Neo	2 7B	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	MLP-Mixer	Mixer B16 224	pytorch	✅	❌	❌	✅	✅	✅	✅	❌	View Source
cv panoptic seg	Panoptic Segmentation	ResNet101 Backbone 3x COCO	pytorch	✅	❌	✅	❌	❌	✅	❌	❌	View Source
cv image cls	RegNet	Y 128gf	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	T5	Large	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	VoVNet	39	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	Llama	1B Tiny	jax	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	OpenELM	450M	jax	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	DINOv2	Large	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	EfficientDet	D4	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv keypoint det	HRNet	v2 W40 Osmr	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	Mistral	7B	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image cls	MobileNetV3	Mobilenet v3 Large	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv depth est	MonoDepth2	Mono+stereo 640x192	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp text cls	OPT	1.3b	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	RegNet	Y 064	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	RetinaNet	ResNet50 Backbone with FPN	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image seg	U-Net	Torchhub Brain Unet	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	VoVNet	39 Th	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp masked lm	BERT	Base Uncased	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp qa	OPT	1.3b	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
mm image ttt	Stable Diffusion UNet	Base	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv keypoint det	HRNet	W48	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp text cls	Llama	3.2 1B Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp text cls	Llama	3.2 3B Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	OPT	350M	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp token cls	Phi-3	Mini 4K Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	RegNet	X 800mf	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	VGG	Torchvision Vgg19	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	OpenELM	1 1B	jax	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image cls	AlexNet	Default	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	EfficientDet	D7	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Llama	Huggyllama 7B	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image cls	MobileNetV3	Small 100	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp text cls	Phi-4	Phi 4	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	14B Instruct 1M	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image cls	RegNet	Y 3 2gf	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
mm image text similarity	SigLIP	So400m Patch16 256 I18n	pytorch	✅	❌	✅	❌	❌	✅	❌	❌	View Source
nlp masked lm	ALBERT	Base v2	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	PerceiverIO Vision	Vision Perceiver Learned	pytorch	✅	❌	✅	❌	❌	✅	❌	❌	View Source
cv image cls	VGG	Torchvision Vgg13	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	YOLO-World	Large 1280	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv img to img	Autoencoder	conv	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp text cls	Llama	3.3 70B Instruct	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv image cls	MLP-Mixer	Mixer S16 224	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	RegNet	Y 400mf	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	RetinaNet	ResNet18 Backbone with FPN	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	VGG	16	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	CodeGen	350M Nl	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	MobileNetV2	Mobilenet v2 0.75 160	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp masked lm	RoBERTa	Xlm Base	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	VoVNet	Ese Vovnet19b Dw	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	Wide ResNet	101.2	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv object det	YOLOv6	L	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source

---

LLM Performance

Model	Token/sec/user	Batch	Token/sec	ttft (ms)	Hardware
pytorch_Falcon_3_10B_Base_nlp_causal_lm_huggingface	41.0	32	1312.0	870.12	p150
pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface	54.0	32	1728.0	696.51	n150
pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface	101.0	32	3232.0	300.04	p150
pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface	36.0	32	1152.0	851.15	n150
pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface	67.0	32	2144.0	382.85	p150
pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface	18.0	32	576.0	1242.77	n150
pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface	35.0	32	1120.0	514.09	p150
pytorch_GPT-OSS_120B_nlp_causal_lm_huggingface	4.0	8	32.0	1418.64	p150
pytorch_GPT-OSS_20B_nlp_causal_lm_huggingface	8.0	64	512.0	7026.75	n150
pytorch_GPT-OSS_20B_nlp_causal_lm_huggingface	21.0	1	21.0	305.39	p150
pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface	38.0	32	1216.0	642.27	n150
pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface	74.0	32	2368.0	239.31	p150
pytorch_Llama_3.1_70B_Instruct_nlp_causal_lm_huggingface	7.0	32	224.0	9366.03	n150
pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface	21.0	32	672.0	1280.98	n150
pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface	51.0	32	1632.0	661.59	p150
pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface	64.0	32	2048.0	572.82	n150
pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface	124.0	32	3968.0	242.2	p150
pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface	30.0	32	960.0	618.76	n150
pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface	54.0	32	1728.0	283.61	p150
pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface	20.0	32	640.0	1242.85	n150
pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface	35.0	32	1120.0	585.28	p150
pytorch_Mistral_Small_24B_INSTRUCT_2501_nlp_causal_lm_huggingface	29.0	32	928.0	912.01	p150
pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface	20.0	32	640.0	668.72	n150
pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface	37.0	32	1184.0	321.18	p150
pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface	20.0	32	640.0	655.9	n150
pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface	37.0	32	1184.0	326.37	p150
pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface	8.0	32	256.0	1505.44	n150
pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface	20.0	32	640.0	682.5	p150
pytorch_Qwen 2.5 Coder_32B_Instruct_nlp_causal_lm_huggingface	17.0	32	544.0	1492.51	p150
pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface	69.0	32	2208.0	416.33	n150
pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface	126.0	32	4032.0	170.84	p150
pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface	36.0	32	1152.0	487.75	n150
pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface	63.0	32	2016.0	201.74	p150
pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface	30.0	32	960.0	684.4	n150
pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface	58.0	32	1856.0	293.35	p150
pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface	16.0	32	512.0	832.8	n150
pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface	28.0	32	896.0	352.74	p150
pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface	48.0	32	1536.0	1177.48	n150
pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface	96.0	32	3072.0	553.07	p150
pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface	36.0	32	1152.0	739.98	n150
pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface	65.0	32	2080.0	337.18	p150
pytorch_Qwen 3_32B_nlp_causal_lm_huggingface	17.0	32	544.0	1852.34	p150
pytorch_Qwen 3_4B_nlp_causal_lm_huggingface	23.0	32	736.0	983.15	n150
pytorch_Qwen 3_4B_nlp_causal_lm_huggingface	41.0	32	1312.0	453.98	p150
pytorch_Qwen 3_8B_nlp_causal_lm_huggingface	16.0	32	512.0	1619.65	n150
pytorch_Qwen 3_8B_nlp_causal_lm_huggingface	30.0	32	960.0	738.18	p150

Non-LLM Performance

Model	Batch	Sample/sec	Hardware
Wan2.2-I2V-A14B-DiT	1	0.0	p150
Wan2.2-I2V-A14B-UMT5-Text-Encoder	1	12.0	p150
Wan2.2-I2V-A14B-VAE-Decoder	1	0.0	p150
Wan2.2-I2V-A14B-VAE-Encoder	1	1.0	p150
flux1-dev	1	0.0	p150
flux2	1	0.0	p150
glm-image	1	0.0	p150
hunyuan-image-2.1	1	0.0	p150
janus-pro-1b	1	0.0	n150
janus-pro-1b	1	0.0	p150
janus-pro-7b	1	0.0	p150
playground-v2.5	1	0.0	n150
playground-v2.5	1	0.0	p150
pytorch_BERT_emrecan/bert-base-turkish-cased-mean-nli-stsb-tr_nlp_embed_gen_huggingface	8	159.0	n150
pytorch_BGE-M3_Base_nlp_embed_gen_custom	4	9.0	n150
pytorch_BGE-M3_Base_nlp_embed_gen_custom	4	17.0	p150
pytorch_EfficientNet_Timm_B0_cv_image_cls_timm	8	346.0	n150
pytorch_EfficientNet_Timm_B0_cv_image_cls_timm	8	793.0	p150
pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom	32	14154.0	n150
pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom	32	29631.0	p150
pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub	12	1168.0	n150
pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub	12	2785.0	p150
pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface	32	46.0	n150
pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface	32	97.0	p150
pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface	8	1310.0	n150
pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface	8	2764.0	p150
pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface	1	37.0	n150
pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface	1	83.0	p150
pytorch_Swin_S_cv_image_cls_torchvision	1	10.0	n150
pytorch_Swin_S_cv_image_cls_torchvision	1	22.0	p150
pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface	1	5.0	n150
pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface	1	9.0	p150
pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github	1	137.0	n150
pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github	1	237.0	p150
pytorch_VGG19-UNet_base_cv_image_seg_custom	1	153.0	n150
pytorch_VGG19-UNet_base_cv_image_seg_custom	1	309.0	p150
pytorch_ViT_Base_cv_image_cls_huggingface	8	234.0	n150
pytorch_ViT_Base_cv_image_cls_huggingface	8	563.0	p150
pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm	8	744.0	n150
pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm	8	1549.0	p150
sdxl-lightning	1	0.0	n150
sdxl-lightning	1	0.0	p150
zimage	1	0.0	p150

Model coverage

Info: Full list of supported models is available in the assets section.

Model task	Model architecture	Model variant	Model framework	Inference	Training	n150	n300	p150	Single device	Data parallel	Tensor parallel	Model source
cv image cls	MNIST	Cnn Dropout	jax	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5 Coder	3B Instruct	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv image cls	ResNet	ResNet50 HuggingFace High Resolution	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-2	Phi 2	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Mistral	Ministral 8B Instruct	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
mm action prediction	OpenVLA-OFT	Finetuned Libero 10	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
mm tts	xtts_v2	speaker encoder	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-1 LoRA	Phi 1	pytorch	❌	✅	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-1	Phi 1	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
nlp causal lm	Qwen 3	1 7B	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv image cls	VoVNet	Ese Vovnet19b Dw.ra In1k	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	Qwen 2.5 Coder	32B Instruct	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv object det	EfficientDet	D0	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Falcon	3 10B Base	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
nlp causal lm	Phi-1.5	Phi 1 5	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
cv image cls	VGG	HF Vgg19	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	olmo_3	3 1125 32b	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
mm tts	xtts_v2	gpt prefill	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	ALLaM	7B Instruct	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
nlp causal lm	Llama	3.2 3B	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	YOLO-World	Small 640	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	0.5B	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
nlp causal lm	Falcon	3 3B Base	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	Swin	S	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5 Coder	1.5B Instruct	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
nlp causal lm	Qwen 2.5 Coder	3B	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Mistral	7B INSTRUCT v03	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
nlp causal lm	Qwen 2.5	32B Instruct	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Qwen 3	0 6B	pytorch	✅	❌	✅	✅	✅	✅	❌	✅	View Source
nlp causal lm	Falcon	3 1B Base	pytorch	✅	❌	✅	✅	✅	✅	❌	✅	View Source
cv object det	PointPillars	pointpillars	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
mm tts	xtts_v2	gpt decode	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	3B Instruct	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv image cls	MobileNetV1	Mobilenet v1	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	ssd512	ssd512	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	olmo_3	3 1025 7b	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-3	Mini Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp embed gen	FLUX	ClipTextEncoder	pytorch	✅	❌	❌	❌	❌	✅	❌	❌	Source not available
nlp causal lm	Mistral	Magistral Small 2506	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
cv image cls	MNIST	Cnn Dropout	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	olmo_3	3 7b think	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	14B Instruct	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
nlp causal lm	Phi-2	Phi 2	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
mm image text similarity	CLIP	Base Patch16	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	DETR	ResNet50 Backbone	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv image cls	SegFormer	Mit B0	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	DINOv2	Small	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 3	32B	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Qwen 2.5 Coder	1.5B	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
cv object det	YOLOS Small	Small	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
cv object det	YOLOv4	Base	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
cv image cls	MNIST	Mlp Custom	jax	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	3B Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	0.5B Instruct	jax	✅	❌	✅	✅	✅	✅	❌	✅	View Source
cv image cls	EfficientNet	B0	pytorch	✅	❌	✅	✅	✅	✅	✅	❌	View Source
nlp causal lm	Gemma	2 27B IT	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Mistral	Devstral Small 2505	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Gemma LoRA	1.1 2B IT	pytorch	❌	✅	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-4	Phi 4	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
cv object det	YOLOP	Default	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Llama LoRA	3.2 1B	pytorch	❌	✅	✅	❌	✅	✅	❌	❌	View Source
cv image cls	MNIST	Mlp Custom 1x2	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
mm tts	xtts_v2	hifigan decoder	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Falcon	3 7B Base	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
mm visual qa	Llama	3.2 11B Vision Instruct	pytorch	✅	❌	❌	❌	✅	✅	❌	❌	View Source
cv image seg	Ultra-Fast Lane Detection	TuSimple ResNet18 Backbone	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Qwen 2.5	7B Instruct	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
nlp token cls	BiLSTM-CRF	Default	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Llama	3.1 70B	pytorch	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Llama	3.2 1B	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp embed gen	FLUX	T5TextEncoder	pytorch	✅	❌	❌	❌	❌	✅	❌	❌	Source not available
cv image cls	AlexNet	Custom 1x2	jax	✅	❌	❌	✅	❌	❌	❌	✅	View Source
nlp causal lm	Gemma	2 9B IT	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
nlp causal lm	Qwen 3	8B	pytorch	✅	❌	❌	✅	✅	✅	❌	✅	View Source
cv img to img	Autoencoder	linear	pytorch	❌	✅	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Gemma	2 2B IT	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-3	Mini 128K Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
nlp causal lm	Phi-3	Mini 4K Instruct	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
conditional generation	U-Net for Conditional Generation	Base	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source
mm tts	xtts_v2	gpt latents	pytorch	✅	❌	✅	❌	✅	✅	❌	❌	View Source