Repository navigation
1.0.0
·
60 commits
to main
since this release
Installation
Via PyPI
pip install tt-forge==1.0.0 --extra-index-url https://pypi.eng.aws.tenstorrent.com/Via Docker
docker pull ghcr.io/tenstorrent/tt-forge-slim:1.0.0What's Changed
- Removed depricated tt-torch references and updated benchmark/README.md by @skhatriTT in #893
- Forge onnx release on python 3.12 by @vvukomanTT in #900
- Fix 3.11 references that were missed by @vvukomanTT in #903
- Uplift third_party/tt_forge_models to 06d8cb1ee812ab1b899647834acfd811bbd1a1bc 2026-02-28 by @vvukomanTT in #904
- Bump release version by @vvukomanTT in #902
- Add onnx demos: Set1 by @meenakshiramanathan1 in #892
- Set transformers to 4.57.1 for demo tests by @nsumrakTT in #907
- Uplift third_party/tt_forge_models to da5c79019035491321d6bd72b583ad2de0398c54 2026-03-07 by @vvukomanTT in #908
- Security: Add authorization check to claude.yml workflow by @jbakerTT in #906
- Add TT-Forge Quick Start Guide by @skhatriTT in #840
- Add pytorch hf demo and fix models-matrix by @gengelageTT in #910
- Overhaul README and getting started docs by @staylorTT in #911
- Uplift third_party/tt_forge_models to e0c5bca09838cbedbedfd3c9c6f9a3edfc961fe6 2026-03-14 by @vvukomanTT in #912
- Update README.md file and CI by @chandrasekaranpradeep in #913
- Fix demos: replace removed xla_backend with tt backend by @acicovicTT in #915
- Update README to remove TT-Torch information by @staylorTT in #916
- Uplift third_party/tt_forge_models to 9dad6bc9b61c38452663e4c4234809a5a0ee9d53 2026-03-21 by @vvukomanTT in #918
- Add model bring-up guide by @staylorTT in #919
New Contributors
Full Changelog: 0.9.0...1.0.0
LLM Performance
| Model | Token/sec/user | Batch | Token/sec | ttft (ms) |
|---|---|---|---|---|
| pytorch_Falcon_3_1B_Base_nlp_causal_lm_huggingface | 25.65 | 32 | 820.85 | 314.36 |
| pytorch_Falcon_3_3B_Base_nlp_causal_lm_huggingface | 19.17 | 32 | 613.57 | 391.92 |
| pytorch_Falcon_3_7B_Base_nlp_causal_lm_huggingface | 12.05 | 32 | 385.64 | 602.8 |
| pytorch_Gemma_1.1_2B_IT_nlp_causal_lm_huggingface | 13.19 | 32 | 421.95 | 650.54 |
| pytorch_Llama_3.1_8B_Instruct_nlp_causal_lm_huggingface | 12.12 | 32 | 387.91 | 551.13 |
| pytorch_Llama_3.2_1B_Instruct_nlp_causal_lm_huggingface | 28.23 | 32 | 903.24 | 335.67 |
| pytorch_Llama_3.2_3B_Instruct_nlp_causal_lm_huggingface | 17.77 | 32 | 568.61 | 499.55 |
| pytorch_Mistral_7B_INSTRUCT_v03_nlp_causal_lm_huggingface | 13.56 | 32 | 434.06 | 474.07 |
| pytorch_Mistral_Ministral_8B_Instruct_nlp_causal_lm_huggingface | 11.07 | 32 | 354.32 | 567.19 |
| pytorch_Phi-1.5_Phi_1_5_nlp_causal_lm_huggingface | 17.99 | 32 | 575.69 | 232.6 |
| pytorch_Phi-1_Phi_1_nlp_causal_lm_huggingface | 17.92 | 32 | 573.44 | 230.1 |
| pytorch_Phi-2_Phi_2_nlp_causal_lm_huggingface | 8.66 | 32 | 277.02 | 463.28 |
| pytorch_Qwen 2.5_0.5B_Instruct_nlp_causal_lm_huggingface | 22.43 | 32 | 717.67 | 331.34 |
| pytorch_Qwen 2.5_1.5B_Instruct_nlp_causal_lm_huggingface | 19.41 | 32 | 621.24 | 436.16 |
| pytorch_Qwen 2.5_3B_Instruct_nlp_causal_lm_huggingface | 16.32 | 32 | 522.29 | 520.05 |
| pytorch_Qwen 2.5_7B_Instruct_nlp_causal_lm_huggingface | 12.23 | 32 | 391.38 | 862.02 |
| pytorch_Qwen 3_0_6B_nlp_causal_lm_huggingface | 19.64 | 32 | 628.63 | 372.12 |
| pytorch_Qwen 3_1_7B_nlp_causal_lm_huggingface | 18.62 | 32 | 595.84 | 407.61 |
| pytorch_Qwen 3_4B_nlp_causal_lm_huggingface | 13.8 | 32 | 441.47 | 584.59 |
| pytorch_Qwen 3_8B_nlp_causal_lm_huggingface | 9.71 | 32 | 310.84 | 735.82 |
Non-LLM Performance
| Model | Batch | Sample/sec |
|---|---|---|
| pytorch_BERT_emrecan/bert-base-turkish-cased-mean-nli-stsb-tr_nlp_embed_gen_huggingface | 8 | 64.6 |
| pytorch_BGE-M3_Base_nlp_embed_gen_custom | 4 | 7.97 |
| pytorch_EfficientNet_Timm_B0_cv_image_cls_timm | 8 | 286.58 |
| pytorch_MNIST_Cnn_Dropout_cv_image_cls_custom | 32 | 21158.17 |
| pytorch_MobileNetV2_Mobilenet_v2_cv_image_cls_torch_hub | 12 | 844.91 |
| pytorch_Qwen 3_Embedding_4B_nlp_embed_gen_huggingface | 32 | 48.71 |
| pytorch_ResNet_ResNet50_HuggingFace_cv_image_cls_huggingface | 8 | 822.61 |
| pytorch_SegFormer_B0_Finetuned_Ade_512_512_cv_image_seg_huggingface | 1 | 22.04 |
| pytorch_Swin_S_cv_image_cls_torchvision | 1 | 8.1 |
| pytorch_U-Net for Conditional Generation_Base_conditional_generation_huggingface | 1 | 3.2 |
| pytorch_Ultra-Fast Lane Detection v2_TuSimple_ResNet34_Backbone_cv_image_seg_github | 1 | 141.31 |
| pytorch_VGG19-UNet_base_cv_image_seg_custom | 1 | 141.72 |
| pytorch_ViT_Base_cv_image_cls_huggingface | 8 | 228.63 |
| pytorch_VoVNet_Ese_Vovnet19b_Dw.ra_In1k_cv_image_cls_timm | 8 | 532.21 |
Model coverage
Info: Full list of supported models is available in the assets section.
| Model task | Model architecture | Model variant | Model framework | Inference | Training | n150 | n300 | p150 | Single device | Data parallel | Tensor parallel | Model source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| conditional generation | U-Net for Conditional Generation | Base | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | AlexNet | Custom 1x2 | jax | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| cv image cls | DINOv2 | Small | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | EfficientNet | B0 | pytorch | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | View Source |
| cv image cls | MNIST | Cnn Batchnorm | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Cnn Dropout | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Cnn Dropout | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Cnn Nodropout | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Mlp Custom | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Mlp Custom | jax | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MNIST | Mlp Custom 1x2 | jax | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| cv image cls | MobileNetV1 | Mobilenet v1 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | MobileNetV2 | Mobilenet v2 | pytorch | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | View Source |
| cv image cls | ResNet | ResNet18 | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | ResNet | ResNet50 | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | ResNet | ResNet50 HuggingFace High Resolution | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | SegFormer | Mit B0 | pytorch | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | View Source |
| cv image cls | Swin | S | pytorch | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | View Source |
| cv image cls | VGG | HF Vgg19 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image cls | ViT | Base | pytorch | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | View Source |
| cv image cls | ViT | Base Patch16 224 | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv image cls | VoVNet | Ese Vovnet19b Dw.ra In1k | pytorch | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | View Source |
| cv image seg | Ultra-Fast Lane Detection | TuSimple ResNet18 Backbone | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv image seg | VGG19-UNet | base | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv img to img | Autoencoder | linear | pytorch | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv object det | Attention DenseUNet | Base | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv object det | DETR | ResNet50 Backbone | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | EfficientDet | D0 | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv object det | OWL-ViT | Base Patch32 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | TransFuser | None | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | YOLO-World | Small 640 | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv object det | YOLOP | Default | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | YOLOS Small | Small | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | YOLOv4 | Base | pytorch | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | View Source |
| cv object det | YOLOv7 | Default | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| cv object det | YOLOv9 | T | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| cv object det | ssd512 | ssd512 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| mm action prediction | OpenVLA | 7B Finetuned Libero Object | pytorch | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| mm image text similarity | SigLIP | Base Patch16 224 | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Falcon | 3 10B Base | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Falcon | 3 1B Base | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Falcon | 3 3B Base | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Falcon | 3 7B Base | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | GPT-2 | Base | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | GPT-2 | Xl | jax | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | GPT-Neo | 125M | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | GPT-OSS | 20B | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Gemma | 1.1 2B IT | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Gemma | 1.1 7B IT | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Gemma | 2 27B IT | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Gemma | 2 2B IT | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Gemma | 2 9B IT | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Llama | 3.1 70B | pytorch | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Llama | 3.1 8B Instruct | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Llama | 3.2 1B | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Llama | 3.2 3B | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Mistral | 7B INSTRUCT v03 | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Mistral | Devstral Small 2505 | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Mistral | Magistral Small 2506 | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Mistral | Ministral 8B Instruct | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Mistral | Nemo INSTRUCT 2407 | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Mistral | Small 24B INSTRUCT 2501 | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Phi-1 | Phi 1 | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-1 | Phi 1 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-1.5 | Phi 1 5 | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-1.5 | Phi 1 5 | pytorch | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-2 | Phi 2 | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-3 | Mini 128K Instruct | pytorch | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-3 | Mini 4K Instruct | pytorch | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-3 | Mini Instruct | pytorch | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Phi-4 | Phi 4 | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Qwen 2.5 | 0.5B | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Qwen 2.5 | 0.5B Instruct | jax | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Qwen 2.5 | 0.5B Instruct | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Qwen 2.5 | 1.5B Instruct | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Qwen 2.5 | 14B Instruct | pytorch | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | View Source |
| nlp causal lm | Qwen 2.5 | 32B Instruct | pytorch | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Qwen 2.5 | 3B Instruct | pytorch | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | View Source |
| nlp causal lm | Qwen 2.5 | 72B | pytorch | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | View Source |
| nlp causal lm | Qwen 2.5 | 72B Instruct | pytorch | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | View Source |