diff --git a/docs/ADVANCED-USERS.md b/docs/ADVANCED-USERS.md index 8f66b8a29..b996bf202 100644 --- a/docs/ADVANCED-USERS.md +++ b/docs/ADVANCED-USERS.md @@ -251,6 +251,8 @@ To improve performance, you can compile the model with `--compile` trading off the time to first token processed with time per token. To improve performance further, you may also compile the prefill with `--compile-prefill`. This will increase further compilation times though. +For CPU, you can use `--max-autotune` to further improve the performance +with `--compile` and `compile-prefill`. See [`max-autotune on CPU tutorial`](https://pytorch.org/tutorials/prototype/max_autotune_on_CPU_tutorial.html). Parallel prefill is not yet supported by exported models, and may be supported in a future release. diff --git a/docs/model_customization.md b/docs/model_customization.md index 3c076fa71..7108b4ce2 100644 --- a/docs/model_customization.md +++ b/docs/model_customization.md @@ -34,6 +34,9 @@ prefill with `--compile_prefill`. To learn more about compilation, check out: https://pytorch.org/get-started/pytorch-2.0/ +For CPU, you can use `--max-autotune` to further improve the performance with `--compile` and `compile-prefill`. + +See [`max-autotune on CPU tutorial`](https://pytorch.org/tutorials/prototype/max_autotune_on_CPU_tutorial.html). ## Model Precision