Update default executor runner with new optional options #14017

mansnils · 2025-09-05T18:11:13Z

By default not all outputs are printed. Adds option for printing all output.
Adds option to save output to file.
In the executor runner all inputs get hard coded to ones. Adds optional input option, in which case tensor inputs will be written from supplied binary input files.

For Arm backend update the Arm VKML unit test runner as a user with real inputs and file output. Enables add/acos_unit tests to run on Vulkan runtime that depend on this.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

In the executor runner all inputs get hard coded to ones. Add optional input option, in which case tensor inputs will be written from supplied binary input files. Also update the Arm VKML runner unit test runner as a user with real inputs. GenAI used, Blackduck Scan OK Change-Id: Ie0363e04f0bbcb2342781f3c560ac30837d88a31

By default not all output is printed. Adds option for printing all output. Also adds option to print output to file. Also update the Arm VKML unit test runner as a user that prints output to file. Enables acos_unit test to run on Vulkan runtime that depends on this. Change-Id: If61c1fe89c9da004fa9db4524e1413893549abce

pytorch-bot · 2025-09-05T18:11:17Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14017

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 943493b with merge base 3c533aa ():

NEW FAILURE - The following job has failed:

Build Presets / apple (llm) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

The file was added by mistake.

zingo · 2025-09-08T09:55:11Z

Hi @digantdesai this is touching stuff outside Arm backend and we need help reviewing it.

digantdesai

LGTM, seems like BC changes for the runner and the extn/utils, make sure the CI is green.

digantdesai · 2025-09-09T10:28:49Z

backends/arm/test/ops/test_acos.py

+        run_on_vulkan_runtime=True,
    )
-    pipeline.run()
+    try:


Nit: can we do it like this -

executorch/backends/arm/test/tester/test_pipeline.py

Line 285 in 3c533aa

class TOSAPipelineMaker(BasePipelineMaker, Generic[T]):

Else I imagine we have to write this for every single test..

Yes indeed we shall not do a try/except in every unit test. Just wanted to enable a test that tested this. We will either do a central try/except or add xfails for remaining tests in coming patches.

digantdesai · 2025-09-09T10:38:18Z

examples/portable/executor_runner/executor_runner.cpp

    model_path,
    "model.pte",
    "Model serialized in flatbuffer format.");
+DEFINE_string(inputs, "", "Comma-separated list of input files");


JFYI you can also use bundled-io in the PTE.

Yes, perhaps add that in a separate patch

Outputs can also be saved with ETDump but that is maybe a bigger/separate patch this file should probably be re-synced with examples/devtools/example_runner/example_runner.cpp someday.

Or @digantdesai do you want to avoid adding "inputs" as an argument and only handle it via BundleIO?

E.g. is this something you prefere us to fix now in this PR and later after 1.0/GA?

Just spoted your "LGTM" so that answers my question :)

zingo · 2025-09-09T13:33:02Z

Test fails are unrelated.

zingo · 2025-09-09T13:53:36Z

We are pausing the merge a short bit, we want to make sure the testing also work if VGF tools/VulkanML is installed, and it currently isn't yet in the GitHub testing. But will be in a separate PR.

UPDATE: should be merged at the same time / together with #14116

)" This reverts commit 16aac24.

…4193) Reverts #14017 https://www.internalfb.com/diff/D82053273 Here's the error from internal failure ``` executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ [CONTEXT] | %u [CONTEXT] 118 | buffer_size, [CONTEXT] | ^~~~~~~~~~~ [CONTEXT] xplat/executorch/runtime/platform/log.h:181:13: note: expanded from macro 'ET_LOG' [CONTEXT] 180 | _format, \ [CONTEXT] | ~~~~~~~ [CONTEXT] 181 | ##__VA_ARGS__); \ [CONTEXT] | ^~~~~~~~~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ [CONTEXT] | %zu [CONTEXT] 118 | buffer_size, [CONTEXT] 119 | tensor_meta->nbytes()); [CONTEXT] | ^~~~~~~~~~~~~~~~~~~~~ [CONTEXT] xplat/executorch/runtime/platform/log.h:181:13: note: expanded from macro 'ET_LOG' [CONTEXT] 180 | _format, \ [CONTEXT] | ~~~~~~~ [CONTEXT] 181 | ##__VA_ARGS__); \ [CONTEXT] | ^~~~~~~~~~~ [CONTEXT] 2 errors generated. ```

…ytorch#14193) This reverts commit dc190f9. Original PR, pytorch#14017 was reverted by pytorch#14193. This reverts the revert. The only difference from the original PR is fixing a printf format mismatch in inputs.cpp: %ld is changed to %zu. This should compile on both 32- and 64-bit ABIs. Change-Id: If487e6c6bde313844a94db99a7431af33dfcdd0a

…torch#14193) Reverts pytorch#14017 https://www.internalfb.com/diff/D82053273 Here's the error from internal failure ``` executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | executorch/extension/runner_util/inputs.cpp:118:13: error: format specifies type 'long' but the argument has type 'std::tuple_element<1, std::pair<char *, unsigned int>>::type' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ [CONTEXT] | %u [CONTEXT] 118 | buffer_size, [CONTEXT] | ^~~~~~~~~~~ [CONTEXT] xplat/executorch/runtime/platform/log.h:181:13: note: expanded from macro 'ET_LOG' [CONTEXT] 180 | _format, \ [CONTEXT] | ~~~~~~~ [CONTEXT] 181 | ##__VA_ARGS__); \ [CONTEXT] | ^~~~~~~~~~~ xplat/executorch/extension/runner_util/inputs.cpp:119:13: error: format specifies type 'long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat] [CONTEXT] 117 | "input size (%ld) and tensor size (%ld) mismatch!", [CONTEXT] | ~~~ [CONTEXT] | %zu [CONTEXT] 118 | buffer_size, [CONTEXT] 119 | tensor_meta->nbytes()); [CONTEXT] | ^~~~~~~~~~~~~~~~~~~~~ [CONTEXT] xplat/executorch/runtime/platform/log.h:181:13: note: expanded from macro 'ET_LOG' [CONTEXT] 180 | _format, \ [CONTEXT] | ~~~~~~~ [CONTEXT] 181 | ##__VA_ARGS__); \ [CONTEXT] | ^~~~~~~~~~~ [CONTEXT] 2 errors generated. ```

mansnils added 2 commits September 5, 2025 19:54

mansnils added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk labels Sep 5, 2025

mansnils requested review from swolchok, larryliu0820, manuelcandales and digantdesai as code owners September 5, 2025 18:11

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 5, 2025

mansnils added 2 commits September 8, 2025 10:43

Merge remote-tracking branch origin/upstream/main

a0faaa0

Remove backends/arm/test/ops/.fuse_hidden00004ec100000001

3a345f3

The file was added by mistake.

zingo added this to the 1.0.0 milestone Sep 9, 2025

digantdesai approved these changes Sep 9, 2025

View reviewed changes

Merge remote-tracking branch origin/upstream/main

943493b

zingo added the release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava label Sep 9, 2025

zingo mentioned this pull request Sep 9, 2025

Arm backend: Remove _vgf_ op pytest xfails #14116

Merged

zingo modified the milestone: 1.0.0 Sep 9, 2025

mansnils merged commit 16aac24 into pytorch:main Sep 9, 2025
458 of 459 checks passed

mergennachin added a commit that referenced this pull request Sep 11, 2025

Revert "Update default executor runner with new optional options (#14017

6ddfcbe

)" This reverts commit 16aac24.

mergennachin mentioned this pull request Sep 11, 2025

Revert "Update default executor runner with new optional options" #14193

Merged

mansnils mentioned this pull request Sep 11, 2025

Reapply "Update default executor runner with new optional options" #14207

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update default executor runner with new optional options #14017

Update default executor runner with new optional options #14017

Uh oh!

mansnils commented Sep 5, 2025 •

edited by zingo

Loading

Uh oh!

pytorch-bot bot commented Sep 5, 2025 •

edited

Loading

Uh oh!

zingo commented Sep 8, 2025

Uh oh!

digantdesai left a comment

Uh oh!

digantdesai Sep 9, 2025

Uh oh!

mansnils Sep 9, 2025

Uh oh!

digantdesai Sep 9, 2025

Uh oh!

mansnils Sep 9, 2025

Uh oh!

zingo Sep 9, 2025

Uh oh!

zingo Sep 9, 2025

Uh oh!

zingo commented Sep 9, 2025

Uh oh!

zingo commented Sep 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Update default executor runner with new optional options #14017

Update default executor runner with new optional options #14017

Uh oh!

Conversation

mansnils commented Sep 5, 2025 • edited by zingo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14017

❌ 1 New Failure

Uh oh!

zingo commented Sep 8, 2025

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

digantdesai Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

mansnils Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

mansnils Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

zingo Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

zingo Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

zingo commented Sep 9, 2025

Uh oh!

zingo commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mansnils commented Sep 5, 2025 •

edited by zingo

Loading

pytorch-bot bot commented Sep 5, 2025 •

edited

Loading

zingo commented Sep 9, 2025 •

edited

Loading