Arm backend: Introduce support for a VGF runtime backend. #12426

robell · 2025-07-14T11:57:36Z

This is a first version of a VGF runtime with spport for simple VGF files containing inputs and outputs (no weights) and will prepare the appropriate Vulkan structures and dispatch the workload following the normal backend delegate interfaces. It's intended to be extended to take advantage of the existing Vulkan delegate by replacing the basic object creation, and by re-using the VgfRepr in the appropriate way in either a "direct" Arm backend for testing and simple deployment, or integrated with the Vulkan backend to have good memory, sync and performance interop with existing Vulkan delegate operators.

It re-uses the build-setup (headers, volk, etc) and vulkan_executor_runner and has been tested on linux only. This was on the simple S32 add kernel from the aot_arm_compiler, and a quantized and non-quantized mv2.

It depends on a number of components which are not yet released, and the script for these is not included, as our third party dependencies are still evolving.

Details:

Minor build fix for vulkan runtime.
Bump vulkan and volk headers to get tensor and graph extensions
First version of VGFBackend, dispatching on a vulkan layer driver
Will process the examples/models mv2 model and constants

Change-Id: I1f278cb98872ae8c0675c72995f0249c038d07d8

Testing

This change currently requires internal dependencies while a few pieces are upstreamed. The following is reproducable for those with full access to the ML SDK for Vulkan (https://github.com/arm/ai-ml-sdk-model-converter)

# test models
python3 -m examples.arm.aot_arm_compiler -t vgf --delegate --model_name="add" -i ./out_add -o out_add.pte
python3 -m examples.arm.aot_arm_compiler -t vgf --delegate --model_name="mv2" -i ./out_mv2 -o out_mv2.pte

#quantized test models
python3 -m examples.arm.aot_arm_compiler -t vgf --delegate --quantize --model_name=add -i ./out_add_quant -o out_add_quant.pte
python3 -m examples.arm.aot_arm_compiler --model_name=mv2 --target=vgf --quantize --delegate -i ./out_mv2_quant -o out_mv2_quant.pte

# commands to execute them using the vulkan executor runner
./cmake-out/backends/vulkan/vulkan_executor_runner -model_path out_add.pte
./cmake-out/backends/vulkan/vulkan_executor_runner -model_path out_mv2.pte
./cmake-out/backends/vulkan/vulkan_executor_runner -model_path out_add_quant.pte
./cmake-out/backends/vulkan/vulkan_executor_runner -model_path out_mv2_quant.pte

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

pytorch-bot · 2025-07-14T11:57:39Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12426

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 043be0b with merge base de0554d ():

NEW FAILURES - The following jobs have failed:

trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t 56b13906bcf3710c74b72dbdbf9ebab0f0d6405ebd537e4a450da079fd1b33e9 /exec failed with exit code 1
trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / test-phi-3-mini-runner-linux / linux-job (gh) (trunk failure)
AttributeError: 'StaticCacheConfig' object has no attribute 'get'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

backends/vulkan/runtime/graph/containers/Types.h

backends/vulkan/CMakeLists.txt

ArmRyan

Approved internally

robell · 2025-07-14T14:24:12Z

only failing case is an http 429 error from HF. Unless we want to kick the tests off again, this just needs a review for the vulkan backend build changes and the introduction of the EXECUTORCH_BUILD_VGF option. The VGF backend code has been reviewed internally, but comments welcome of course. Builds/tests will be added once the final dependencies are upstream, which i hope to be a few weeks now.

Sebastian-Larsson · 2025-07-15T06:35:38Z

@digantdesai Do you mind taking a look since this touches some files outside of the Arm backend?

digantdesai

Thanks @robell !

backends/arm/CMakeLists.txt

backends/vulkan/CMakeLists.txt

backends/vulkan/third-party/Vulkan-Headers

digantdesai · 2025-07-24T11:01:05Z

CMakeLists.txt

 if(EXECUTORCH_BUILD_VULKAN)
  add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/backends/vulkan)
 endif()
+if(EXECUTORCH_BUILD_VGF)


Add a readme under backends/arm?

We should to that! I will add one in a separate patch, I will have some updates for this soon

digantdesai · 2025-07-24T11:02:03Z

Please rebase and land once the CI is green (MMLU could be unrelated).

This is a first version of a VGF runtime with spport for simple VGF files containing inputs and outputs (no weights) and will prepare the appropriate Vulkan structures and dispatch the workload following the normal backend delegate interfaces. It's intended to be extended to take advantage of the existing Vulkan delegate by replacing the basic object creation, and by re-using the VgfRepr in the appropriate way in either a "direct" Arm backend for testing and simple deployment, or integrated with the Vulkan backend to have good memory, sync and performance interop with existing Vulkan delegate operators. It re-uses the build-setup (headers, volk, etc) and vulkan_executor_runner and has been tested on linux only. This was on the simple S32 add kernel from the aot_arm_compiler, and a quantized and non-quantized mv2. It depends on a number of components which are not yet released, and the script for these is not included, as our third party dependencies are still evolving. Details: * Minor build fix for vulkan runtime. * Bump vulkan and volk headers to get tensor and graph extensions * First version of VGFBackend, dispatching on a vulkan layer driver * Will process the examples/models mv2 model and constants Signed-off-by: Rob Elliott <robert.elliott@arm.com> Change-Id: I1f278cb98872ae8c0675c72995f0249c038d07d8

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

robell · 2025-07-28T12:10:55Z

I don't believe the failures are related to my changes.

The llama runner failure is present on trunk intermittently: b8fe100

The arm-backend test is for a path i've not modified and is also present intermittently on trunk: https://github.com/pytorch/executorch/actions/runs/16562716934/job/46835516691

the test-phi-3-mini case is present on trunk.

Sebastian-Larsson

Unrelated CI failures. Approved

robell added the release notes: arm Changes to the ARM backend delegate label Jul 14, 2025

robell requested review from SS-JIA, digantdesai, kirklandsign and larryliu0820 as code owners July 14, 2025 11:57

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 14, 2025

robell requested review from ArmRyan, Sebastian-Larsson, YufengShi-dudu and tom-arm July 14, 2025 11:58

robell added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk labels Jul 14, 2025

robell commented Jul 14, 2025

View reviewed changes

backends/vulkan/runtime/graph/containers/Types.h Show resolved Hide resolved

robell commented Jul 14, 2025

View reviewed changes

backends/vulkan/CMakeLists.txt Show resolved Hide resolved

ArmRyan approved these changes Jul 14, 2025

View reviewed changes

robell requested review from SS-JIA and removed request for SS-JIA July 14, 2025 14:22

digantdesai approved these changes Jul 24, 2025

View reviewed changes

robell force-pushed the vgf_t branch from 9099306 to 8aaaf4b Compare July 28, 2025 09:01

minor fix for cmake changes

043be0b

Signed-off-by: Rob Elliott <robert.elliott@arm.com>

Sebastian-Larsson approved these changes Jul 28, 2025

View reviewed changes

Sebastian-Larsson merged commit 1c72e0e into pytorch:main Jul 28, 2025
196 of 199 checks passed

Arm backend: Introduce support for a VGF runtime backend. #12426

Arm backend: Introduce support for a VGF runtime backend. #12426

Uh oh!

Conversation

robell commented Jul 14, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

pytorch-bot bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12426

❌ 2 New Failures, 1 Unrelated Failure

Uh oh!

Uh oh!

Uh oh!

ArmRyan left a comment

Choose a reason for hiding this comment

Uh oh!

robell commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sebastian-Larsson commented Jul 15, 2025

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digantdesai Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

ArmRyan Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai commented Jul 24, 2025

Uh oh!

robell commented Jul 28, 2025

Uh oh!

Sebastian-Larsson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

robell commented Jul 14, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 14, 2025 •

edited

Loading

robell commented Jul 14, 2025 •

edited

Loading