Skip to content

Ship the quantized kernels as their own library - #21595

Open
shoumikhin wants to merge 2 commits into
gh/shoumikhin/80/headfrom
gh/shoumikhin/90/head
Open

Ship the quantized kernels as their own library#21595
shoumikhin wants to merge 2 commits into
gh/shoumikhin/80/headfrom
gh/shoumikhin/90/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

A model quantized before export needs the quantized operator kernels
registered when it runs. Those kernels were only ever built into the Python
extension, so a C++ application had nothing to link and could not run such a
model from an installed package. The only way to get them was to build
ExecuTorch from source and link the target directly, which is what the
embedded examples in this repo do.

The registration library was built as a static archive. That is awkward to
ship, because the kernels register themselves from a static initializer and
an initializer in an archive is discarded unless the consumer opts into
whole-archive linking. Build it as a shared library instead, the same way the
merged CPU kernel set already is, and install it:

find_package(executorch CONFIG REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(
  app PRIVATE executorch::runtime executorch::kernels_quantized
)

The library is named after the component that selects it, so the pair reads
together: executorch::kernels_quantized resolves to
libexecutorch_kernels_quantized.so. The code generation helper would
otherwise name it after its internal target, which would not match.

Nothing else in the wheel links this library, so it also has to be named as a
build target explicitly, or it is declared but never built and packaging
looks for a file that does not exist.

Note that a quantized model delegated to XNNPACK does not need this: that
delegate claims the quantize and dequantize operators itself, so nothing is
left for the CPU kernels to run. These are for a quantized model running on
plain CPU, or on a backend that does not claim those operators.

Tested by building a wheel and installing it into a clean environment, with
imports confirmed to resolve to the installed package rather than a checkout:

  • The wheel ships libexecutorch_kernels_quantized.so.1 next to the runtime, the
    thread pool and the merged CPU kernels.
  • It is a shared library whose soname matches, and it resolves the runtime from
    libexecutorch.so.1 rather than carrying its own copy of the core.
  • Loading it adds 19 quantized_decomposed operators to the registry, including
    the out variants of quantize_per_tensor and dequantize_per_tensor. None were
    present beforehand.
  • A C++ application that asks for the component by name links it and reports
    the same 19 kernels registered at startup, which confirms the library is
    retained on the link line even though the application references no symbol
    from it directly.
  • The merged CPU kernel set is unaffected and builds to the same shape.
  • The existing aggregate library over these kernels still builds, and now
    resolves them through this shared library instead of a static archive.

@shoumikhin
Update
b8b0ddd 
[ghstack-poisoned]
@shoumikhin

shoumikhin commented Aug 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Stack from ghstack (oldest at bottom):

@pytorch-bot

pytorch-bot Bot commented Aug 5, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21595

Note: Links to docs will display an error until the docs builds have been completed.

❌ 164 New Failures, 19 Pending, 2 Unrelated Failures, 7 Unclassified Failures

As of commit 47c1a47 with merge base efd6b55 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

  • periodic / test-models-linux (buck2, mv3, portable, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 942E:B0105:B0EA87:24CC2A7:6A739036 and timestamp 2026-08-05 19:34:14 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 8106:39EA40:5ADB9A4:1310F38C:6A73902F and timestamp 2026-08-05 19:34:07 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • periodic / test-models-linux (cmake, mv3, portable, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID C78E:292886:501FA4E:10DC2176:6A739032 and timestamp 2026-08-05 19:34:10 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID E342:ACD32:A80FA8:22EBB20:6A73902D and timestamp 2026-08-05 19:34:05 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • periodic / test-models-linux (cmake, vit, portable, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 91E0:361F36:AB35F4:2399EFE:6A73902E and timestamp 2026-08-05 19:34:06 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 9344:31821C:B3B09E:2549934:6A73902F and timestamp 2026-08-05 19:34:07 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
  • trunk / test-models-windows (vit, xnnpack-q8) / windows-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
    API rate limit exceeded

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 5, 2026
This was referenced Aug 5, 2026
Open
Open
Open
Open
Open
Open
Open
@shoumikhin
Update
47c1a47 
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 is a code owner
@kirklandsign kirklandsign Awaiting requested review from kirklandsign kirklandsign is a code owner

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shoumikhin