From 2e8c6221602faf102fa18e37d222840a09660357 Mon Sep 17 00:00:00 2001 From: BujSet Date: Mon, 30 Jun 2025 15:51:00 +0000 Subject: [PATCH 1/5] Updating documentation for cmake dtype selective build --- docs/source/kernel-library-selective-build.md | 30 +++++++++++++------ examples/selective_build/README.md | 3 +- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/source/kernel-library-selective-build.md b/docs/source/kernel-library-selective-build.md index f9a991767a3..943fff9beb6 100644 --- a/docs/source/kernel-library-selective-build.md +++ b/docs/source/kernel-library-selective-build.md @@ -36,14 +36,16 @@ The basic flow looks like this: ## APIs -We expose a CMake macro `[gen_selected_ops](https://github.com/pytorch/executorch/blob/main/tools/cmake/Codegen.cmake#L12)`, to allow users specifying op info: +We expose a CMake macro [gen_selected_ops](https://github.com/pytorch/executorch/blob/main/tools/cmake/Codegen.cmake#L12), to allow users specifying op info: ``` gen_selected_ops( - LIB_NAME # the name of the selective build operator library to be generated - OPS_SCHEMA_YAML # path to a yaml file containing operators to be selected - ROOT_OPS # comma separated operator names to be selected - INCLUDE_ALL_OPS # boolean flag to include all operators + LIB_NAME # the name of the selective build operator library to be generated + OPS_SCHEMA_YAML # path to a yaml file containing operators to be selected + ROOT_OPS # comma separated operator names to be selected + INCLUDE_ALL_OPS # boolean flag to include all operators + OPS_FROM_MODEL # path to a pte file of model to select operators from + DTYPE_SELECTIVE_BUILD # boolean flag to enable dtye selection ) ``` @@ -62,6 +64,9 @@ Context: each kernel library is designed to have a yaml file associated with it. This API lets users pass in a list of operator names. Note that this API can be combined with the API above and we will create a allowlist from the union of both API inputs. +### Select ops from model + +This API lets users pass in a pte file of an exported model. When enable, the pte file will be parsed to generate a yaml file that enumerates the operators and dtypes. Flag `DTYPE_SELECTIVE_BUILD` must be set to `ON` for the flow to use the generated YAML file. Once enabled, a header file that specifies only the operators and dtypes used by the model is created and linked against a rebuild of the portable_kernels lib. This feature is not yet supported for other libs. ## Example Walkthrough @@ -69,18 +74,21 @@ In CMakeLists.txt we have the following logic: ```cmake set(_kernel_lib) if(SELECT_ALL_OPS) - gen_selected_ops("" "" "${SELECT_ALL_OPS}") + gen_selected_ops("" "" "${SELECT_ALL_OPS}" "" "") elseif(SELECT_OPS_LIST) - gen_selected_ops("" "${SELECT_OPS_LIST}" "") + gen_selected_ops("" "${SELECT_OPS_LIST}" "" "" "") elseif(SELECT_OPS_YAML) set(_custom_ops_yaml ${EXECUTORCH_ROOT}/examples/portable/custom_ops/custom_ops.yaml) gen_selected_ops("${_custom_ops_yaml}" "" "") +elseif(SELECT_OPS_MODEL && DTYPE_SELECTIVE_BUILD) + set(_model_path $(realpath model.pte)) + gen_selected_ops("" "" "" "${_model_path}" "ON") endif() ``` Then when calling CMake, we can do: ``` -cmake -D… -DSELECT_OPS_LIST="aten::add.out,aten::mm.out” +cmake -D… -DSELECT_OPS_LIST="aten::add.out,aten::mm.out" ``` Or @@ -89,4 +97,8 @@ Or cmake -D… -DSELECT_OPS_YAML=ON ``` -To select from either an operator name list or a schema yaml from kernel library. +To select from either an operator name list or a schema yaml from kernel library. To select operators and dtypes using the model API, we can run: + +``` +cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON +``` diff --git a/examples/selective_build/README.md b/examples/selective_build/README.md index 97706d70c48..2e8e1d58bc1 100644 --- a/examples/selective_build/README.md +++ b/examples/selective_build/README.md @@ -12,10 +12,11 @@ cd executorch bash examples/selective_build/test_selective_build.sh cmake ``` -Check out `CMakeLists.txt` for demo of 3 selective build APIs: +Check out `CMakeLists.txt` for demo of 4 selective build APIs: 1. `SELECT_ALL_OPS`: Select all ops from the dependency kernel libraries, register all of them into ExecuTorch runtime. 2. `SELECT_OPS_LIST`: Only select operators from a list. 3. `SELECT_OPS_YAML`: Only select operators from a yaml file. +3. `SELECT_OPS_FROM_MODEL`: Only select operators from a from an exported model pte. Other configs: - `MAX_KERNEL_NUM=N`: Only allocate memory for N operators. From 060fc142d3e88e6c18a2aed38924f594f381f8a3 Mon Sep 17 00:00:00 2001 From: BujSet Date: Mon, 30 Jun 2025 17:34:39 +0000 Subject: [PATCH 2/5] Updating docs based on some comments, unclear about one section --- docs/source/kernel-library-selective-build.md | 27 +++++++++++++++---- examples/selective_build/README.md | 7 +++-- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/source/kernel-library-selective-build.md b/docs/source/kernel-library-selective-build.md index 943fff9beb6..d01d8f2a4eb 100644 --- a/docs/source/kernel-library-selective-build.md +++ b/docs/source/kernel-library-selective-build.md @@ -66,13 +66,19 @@ This API lets users pass in a list of operator names. Note that this API can be ### Select ops from model -This API lets users pass in a pte file of an exported model. When enable, the pte file will be parsed to generate a yaml file that enumerates the operators and dtypes. Flag `DTYPE_SELECTIVE_BUILD` must be set to `ON` for the flow to use the generated YAML file. Once enabled, a header file that specifies only the operators and dtypes used by the model is created and linked against a rebuild of the portable_kernels lib. This feature is not yet supported for other libs. +This API lets users pass in a pte file of an exported model. When used, the pte file will be parsed to generate a yaml file that enumerates the operators and dtypes used in the model. + +### Dtype Selective Build + +Beyond pruning the binary to remove unused operators, the binary size can furhter reduced by enforcing checks on what dtypes are used by the specific operators. For example, if your model only uses floats for the `add` operator, then including variants of the `add` operators for `doubles` and `ints` is unnecessary. The flag `DTYPE_SELECTIVE_BUILD` can be set to `ON` to support this additional optimization. Currently, dtype selective build is only support with the model API described above. Once enabled, a header file that specifies only the operators and dtypes used by the model is created and linked against a rebuild of the `portable_kernels` lib. This feature is not yet supported for other libs. ## Example Walkthrough -In CMakeLists.txt we have the following logic: +In [CMakeLists.txt](https://github.com/pytorch/executorch/blob/main/examples/selective_build/CMakeLists.txt#L89-L123) we have the following logic: + ```cmake set(_kernel_lib) + if(SELECT_ALL_OPS) gen_selected_ops("" "" "${SELECT_ALL_OPS}" "" "") elseif(SELECT_OPS_LIST) @@ -80,9 +86,13 @@ elseif(SELECT_OPS_LIST) elseif(SELECT_OPS_YAML) set(_custom_ops_yaml ${EXECUTORCH_ROOT}/examples/portable/custom_ops/custom_ops.yaml) gen_selected_ops("${_custom_ops_yaml}" "" "") -elseif(SELECT_OPS_MODEL && DTYPE_SELECTIVE_BUILD) +elseif(SELECT_OPS_MODEL) set(_model_path $(realpath model.pte)) - gen_selected_ops("" "" "" "${_model_path}" "ON") + if(DTYPE_SELECTIVE_BUILD) + gen_selected_ops("" "" "" "${_model_path}" "ON") + else + gen_selected_ops("" "" "" "${_model_path}" "") + endif() endif() ``` Then when calling CMake, we can do: @@ -97,7 +107,14 @@ Or cmake -D… -DSELECT_OPS_YAML=ON ``` -To select from either an operator name list or a schema yaml from kernel library. To select operators and dtypes using the model API, we can run: +Or + + +``` +cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" +``` + +To select from either an operator name list, a schema yaml, or directly from an exported model's pte from kernel library. To further optimize the binary size for dtype selection, we can run: ``` cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON diff --git a/examples/selective_build/README.md b/examples/selective_build/README.md index 2e8e1d58bc1..888020ce770 100644 --- a/examples/selective_build/README.md +++ b/examples/selective_build/README.md @@ -12,13 +12,12 @@ cd executorch bash examples/selective_build/test_selective_build.sh cmake ``` -Check out `CMakeLists.txt` for demo of 4 selective build APIs: +Check out `CMakeLists.txt` for demo of selective build APIs: 1. `SELECT_ALL_OPS`: Select all ops from the dependency kernel libraries, register all of them into ExecuTorch runtime. 2. `SELECT_OPS_LIST`: Only select operators from a list. 3. `SELECT_OPS_YAML`: Only select operators from a yaml file. -3. `SELECT_OPS_FROM_MODEL`: Only select operators from a from an exported model pte. +4. `SELECT_OPS_FROM_MODEL`: Only select operators from a from an exported model pte. +5. `DTYPE_SELECTIVE_BUILD`: Enable rebuild of `portable_kernels` to use dtype selection. Currently only supported for `SELECTED_OPS_FROM_MODEL` API and `portable_kernels` lib. Other configs: - `MAX_KERNEL_NUM=N`: Only allocate memory for N operators. - -We have one more API incoming: only select from an exported model file (.pte). From 3bb664f7f23e970cadf67f69ccd6127768cc8fa8 Mon Sep 17 00:00:00 2001 From: BujSet Date: Mon, 30 Jun 2025 18:33:46 +0000 Subject: [PATCH 3/5] Updating doc to better reflect new changes to gen_selected_ops --- docs/source/kernel-library-selective-build.md | 54 +++++-------------- 1 file changed, 13 insertions(+), 41 deletions(-) diff --git a/docs/source/kernel-library-selective-build.md b/docs/source/kernel-library-selective-build.md index d01d8f2a4eb..0b60361ae4e 100644 --- a/docs/source/kernel-library-selective-build.md +++ b/docs/source/kernel-library-selective-build.md @@ -74,48 +74,20 @@ Beyond pruning the binary to remove unused operators, the binary size can furhte ## Example Walkthrough -In [CMakeLists.txt](https://github.com/pytorch/executorch/blob/main/examples/selective_build/CMakeLists.txt#L89-L123) we have the following logic: - -```cmake -set(_kernel_lib) - -if(SELECT_ALL_OPS) - gen_selected_ops("" "" "${SELECT_ALL_OPS}" "" "") -elseif(SELECT_OPS_LIST) - gen_selected_ops("" "${SELECT_OPS_LIST}" "" "" "") -elseif(SELECT_OPS_YAML) - set(_custom_ops_yaml ${EXECUTORCH_ROOT}/examples/portable/custom_ops/custom_ops.yaml) - gen_selected_ops("${_custom_ops_yaml}" "" "") -elseif(SELECT_OPS_MODEL) - set(_model_path $(realpath model.pte)) - if(DTYPE_SELECTIVE_BUILD) - gen_selected_ops("" "" "" "${_model_path}" "ON") - else - gen_selected_ops("" "" "" "${_model_path}" "") - endif() -endif() -``` -Then when calling CMake, we can do: - -``` -cmake -D… -DSELECT_OPS_LIST="aten::add.out,aten::mm.out" -``` +In [CMakeLists.txt](https://github.com/BujSet/executorch/blob/main/examples/selective_build/CMakeLists.txt#L48-L72), we have the following cmake config options: -Or - -``` -cmake -D… -DSELECT_OPS_YAML=ON -``` +1. `EXECUTORCH_SELECT_OPS_YAML` +2. `EXECUTORCH_SELECT_OPS_LIST` +3. `EXECUTORCH_SELECT_ALL_OPS` +4. `EXECUTORCH_SELECT_OPS_FROM_MODEL` +5. `EXECUTORCH_DTYPE_SELECTIVE_BUILD` -Or +These options allow a user to tailor the cmake build process to utilize the different APIs, and results in different invocations on the `gen_selected_ops` [function](https://github.com/BujSet/executorch/blob/main/examples/selective_build/CMakeLists.txt#L110-L123). The following table describes some examples of how the invocation changes when these configs are set: +| Example cmake Call | Resultant `gen_selected_ops` Invocation | +| :----: | :---:| +|
cmake -D… -DSELECT_OPS_LIST="aten::add.out,aten::mm.out"
|
gen_selected_ops("" "${SELECT_OPS_LIST}" "" "" "")
| +|
cmake -D… -DSELECT_OPS_YAML=ON
|
set(_custom_ops_yaml ${EXECUTORCH_ROOT}/examples/portable/custom_ops/custom_ops.yaml)
gen_selected_ops("${_custom_ops_yaml}" "" "")
| +|
cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out"
|
gen_selected_ops("" "" "" "${_model_path}" "")
| +|
cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON
 |
gen_selected_ops("" "" "" "${_model_path}" "ON")
| -``` -cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -``` - -To select from either an operator name list, a schema yaml, or directly from an exported model's pte from kernel library. To further optimize the binary size for dtype selection, we can run: - -``` -cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON -``` From 9ec7dc2ee809fbe06e9ec04823d8f0fbdd56a2c4 Mon Sep 17 00:00:00 2001 From: BujSet Date: Mon, 30 Jun 2025 19:36:57 +0000 Subject: [PATCH 4/5] Addressing comments --- docs/source/kernel-library-selective-build.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/kernel-library-selective-build.md b/docs/source/kernel-library-selective-build.md index 0b60361ae4e..01233d868f7 100644 --- a/docs/source/kernel-library-selective-build.md +++ b/docs/source/kernel-library-selective-build.md @@ -49,6 +49,7 @@ gen_selected_ops( ) ``` +The macro makes a call to gen_oplist.py, which requires a [distinct selection](https://github.com/BujSet/executorch/blob/main/codegen/tools/gen_oplist.py#L222-L228) of API choice. `OPS_SCHEMA_YAML`, `ROOT_OPS`, `INCLUDE_ALL_OPS`, and `OPS_FROM_MODEL` are mutually exclusive options, and should not be used in conjunction. ### Select all ops @@ -70,7 +71,7 @@ This API lets users pass in a pte file of an exported model. When used, the pte ### Dtype Selective Build -Beyond pruning the binary to remove unused operators, the binary size can furhter reduced by enforcing checks on what dtypes are used by the specific operators. For example, if your model only uses floats for the `add` operator, then including variants of the `add` operators for `doubles` and `ints` is unnecessary. The flag `DTYPE_SELECTIVE_BUILD` can be set to `ON` to support this additional optimization. Currently, dtype selective build is only support with the model API described above. Once enabled, a header file that specifies only the operators and dtypes used by the model is created and linked against a rebuild of the `portable_kernels` lib. This feature is not yet supported for other libs. +Beyond pruning the binary to remove unused operators, the binary size can further reduced by removing unused dtypes. For example, if your model only uses floats for the `add` operator, then including variants of the `add` operators for `doubles` and `ints` is unnecessary. The flag `DTYPE_SELECTIVE_BUILD` can be set to `ON` to support this additional optimization. Currently, dtype selective build is only supported with the model API described above. Once enabled, a header file that specifies only the operators and dtypes used by the model is created and linked against a rebuild of the `portable_kernels` lib. This feature is only supported for the portable kernels library; it's not supported for optimized, quantized or custom kernel libraries. ## Example Walkthrough From e9745f4269027103710f594d9604cb37a9cc55ae Mon Sep 17 00:00:00 2001 From: BujSet Date: Mon, 30 Jun 2025 19:51:35 +0000 Subject: [PATCH 5/5] Linting --- docs/source/kernel-library-selective-build.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/kernel-library-selective-build.md b/docs/source/kernel-library-selective-build.md index 01233d868f7..3e9400f4d6a 100644 --- a/docs/source/kernel-library-selective-build.md +++ b/docs/source/kernel-library-selective-build.md @@ -91,4 +91,3 @@ These options allow a user to tailor the cmake build process to utilize the diff |
cmake -D… -DSELECT_OPS_YAML=ON
|
set(_custom_ops_yaml ${EXECUTORCH_ROOT}/examples/portable/custom_ops/custom_ops.yaml)
gen_selected_ops("${_custom_ops_yaml}" "" "")
| |
cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out"
|
gen_selected_ops("" "" "" "${_model_path}" "")
| |
cmake -D… -DEXECUTORCH_SELECT_OPS_FROM_MODEL="model.pte.out" -DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON
 |
gen_selected_ops("" "" "" "${_model_path}" "ON")
| -