Skip to content

1.0.0

Choose a tag to compare

@tenstorrent-github-bot tenstorrent-github-bot released this 27 Mar 10:39
· 60 commits to main since this release
058b278

Installation

Via PyPI

pip install tt-forge==1.0.0 --extra-index-url https://pypi.eng.aws.tenstorrent.com/

Via Docker

docker pull ghcr.io/tenstorrent/tt-forge-slim:1.0.0

What's Changed

New Contributors

Full Changelog: 0.9.0...1.0.0


LLM Performance

Model Token/sec/user Batch Token/sec ttft (ms)
pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface 25.65 32 820.85 314.36
pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface 19.17 32 613.57 391.92
pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface 12.05 32 385.64 602.8
pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface 13.19 32 421.95 650.54
pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface 12.12 32 387.91 551.13
pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface 28.23 32 903.24 335.67
pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface 17.77 32 568.61 499.55
pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface 13.56 32 434.06 474.07
pytorch_Mistral_Ministral_8B_Instruct_nlp_causal_lm_huggingface 11.07 32 354.32 567.19
pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface 17.99 32 575.69 232.6
pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface 17.92 32 573.44 230.1
pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface 8.66 32 277.02 463.28
pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface 22.43 32 717.67 331.34
pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface 19.41 32 621.24 436.16
pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface 16.32 32 522.29 520.05
pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface 12.23 32 391.38 862.02
pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface 19.64 32 628.63 372.12
pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface 18.62 32 595.84 407.61
pytorch_Qwen 3_4B_nlp_causal_lm_huggingface 13.8 32 441.47 584.59
pytorch_Qwen 3_8B_nlp_causal_lm_huggingface 9.71 32 310.84 735.82

Non-LLM Performance

Model Batch Sample/sec
pytorch_BERT_emrecan/bert-base-turkish-cased-mean-nli-stsb-tr_nlp_embed_gen_huggingface 8 64.6
pytorch_BGE-M3_Base_nlp_embed_gen_custom 4 7.97
pytorch_EfficientNet_Timm_B0_cv_image_cls_timm 8 286.58
pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom 32 21158.17
pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub 12 844.91
pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface 32 48.71
pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface 8 822.61
pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface 1 22.04
pytorch_Swin_S_cv_image_cls_torchvision 1 8.1
pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface 1 3.2
pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github 1 141.31
pytorch_VGG19-UNet_base_cv_image_seg_custom 1 141.72
pytorch_ViT_Base_cv_image_cls_huggingface 8 228.63
pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm 8 532.21

Model coverage

Info: Full list of supported models is available in the assets section.

Model task Model architecture Model variant Model framework Inference Training n150 n300 p150 Single device Data parallel Tensor parallel Model source
conditional generation U-Net for Conditional Generation Base pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls AlexNet Custom 1x2 jax ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
cv image cls DINOv2 Small pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls EfficientNet B0 pytorch ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ View Source
cv image cls MNIST Cnn Batchnorm jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls MNIST Cnn Dropout jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls MNIST Cnn Dropout pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls MNIST Cnn Nodropout pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls MNIST Mlp Custom jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls MNIST Mlp Custom jax ❌ ✅ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls MNIST Mlp Custom 1x2 jax ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
cv image cls MobileNetV1 Mobilenet v1 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls MobileNetV2 Mobilenet v2 pytorch ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ View Source
cv image cls ResNet ResNet18 jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls ResNet ResNet50 jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls ResNet ResNet50 HuggingFace High Resolution pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls SegFormer Mit B0 pytorch ✅ ❌ ✅ ✅ ❌ ✅ ✅ ❌ View Source
cv image cls Swin S pytorch ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ View Source
cv image cls VGG HF Vgg19 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image cls ViT Base pytorch ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ View Source
cv image cls ViT Base Patch16 224 jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv image cls VoVNet Ese Vovnet19b Dw.ra In1k pytorch ✅ ❌ ✅ ✅ ❌ ✅ ✅ ❌ View Source
cv image seg Ultra-Fast Lane Detection TuSimple ResNet18 Backbone pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv image seg VGG19-UNet base pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv img to img Autoencoder linear pytorch ❌ ✅ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv object det Attention DenseUNet Base pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv object det DETR ResNet50 Backbone pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det EfficientDet D0 pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv object det OWL-ViT Base Patch32 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det TransFuser None pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det YOLO-World Small 640 pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv object det YOLOP Default pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det YOLOS Small Small pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det YOLOv4 Base pytorch ✅ ❌ ✅ ✅ ❌ ✅ ✅ ❌ View Source
cv object det YOLOv7 Default pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
cv object det YOLOv9 T pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
cv object det ssd512 ssd512 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
mm action prediction OpenVLA 7B Finetuned Libero Object pytorch ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ View Source
mm image text similarity SigLIP Base Patch16 224 pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Falcon 3 10B Base pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Falcon 3 1B Base pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Falcon 3 3B Base pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Falcon 3 7B Base pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm GPT-2 Base jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm GPT-2 Xl jax ❌ ✅ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm GPT-Neo 125M jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm GPT-OSS 20B pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Gemma 1.1 2B IT pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Gemma 1.1 7B IT pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Gemma 2 27B IT pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Gemma 2 2B IT pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Gemma 2 9B IT pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Llama 3.1 70B pytorch ✅ ❌ ❌ ❌ ❌ ❌ ❌ ✅ View Source
nlp causal lm Llama 3.1 8B Instruct pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Llama 3.2 1B pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Llama 3.2 3B pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Mistral 7B INSTRUCT v03 pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Mistral Devstral Small 2505 pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Mistral Magistral Small 2506 pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Mistral Ministral 8B Instruct pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Mistral Nemo INSTRUCT 2407 pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Mistral Small 24B INSTRUCT 2501 pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Phi-1 Phi 1 jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-1 Phi 1 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
nlp causal lm Phi-1.5 Phi 1 5 jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-1.5 Phi 1 5 pytorch ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ View Source
nlp causal lm Phi-2 Phi 2 pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-3 Mini 128K Instruct pytorch ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-3 Mini 4K Instruct pytorch ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-3 Mini Instruct pytorch ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Phi-4 Phi 4 pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Qwen 2.5 0.5B jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Qwen 2.5 0.5B Instruct jax ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Qwen 2.5 0.5B Instruct pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Qwen 2.5 1.5B Instruct pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Qwen 2.5 14B Instruct pytorch ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ View Source
nlp causal lm Qwen 2.5 32B Instruct pytorch ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ View Source
nlp causal lm Qwen 2.5 3B Instruct pytorch ✅ ❌ ✅ ❌ ✅ ✅ ❌ ❌ View Source
nlp causal lm Qwen 2.5 72B pytorch ✅ ❌ ❌ ❌ ❌ ❌ ❌ ✅ View Source
nlp causal lm Qwen 2.5 72B Instruct pytorch ✅ ❌ ❌ ❌ ❌ ❌ ❌ ✅ View Source