Arm Backend: Enable Pybindings for tosa_serialization lib

[ArmRyan]

These bindings should drastically increase serialization
performance, both speed wise and memory usage wise.

This patch also enables serialization of tosa flatbuffers
that are over 2GB in size.

• Ensure each tosa operator has an attribute.
• Replace TosaOp.Op() calls with Op enum
• Convert tosa clamp data to numpy int8 to allow c++ byte serialization
• Replace "import tosa_serializer.serializer" with "import tosa_serializer" for pybind
• Remove serialize to json.
• Remove Darwin support workaround in mlsdk dependencies
• Update mlsdk to latest tag, but overwrite the tosa_mlir_translator to a version that supports offset buffers
• Update tosa-tools to the monorepo
• Update slice to clamp end and start dim. This avoids the object not having a number field from being too large or small.

Change-Id: Id03582b6b7b9eef296480f9ea3a4744a08b22603

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15356

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 2 Cancelled Jobs, 3 Unrelated Failures

As of commit f350ce4 with merge base 788ef2f ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  The process '/usr/bin/sh' failed with exit code 255
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 54f156a6f2246f9ecc7fcb3af9ffdd75923d8dfab60b5df560dadbe791aa2297 /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 88c3908eacf6f390ec86da73968bf6ea58050ccfcb65c611dbeef222edd2174d /exec failed with exit code 1
• Test CUDA Builds / export-gemma3-cuda-non-quantized / linux-job (gh)
  RuntimeError: Command docker exec -t 44c5cf5e933d2ca340ed2ce183a82974165f5ef9955c092bcbed214c83dad1ab /exec failed with exit code 2
• Test CUDA Builds / export-voxtral-cuda-non-quantized / linux-job (gh)
  RuntimeError: Command docker exec -t 50c9bd44b4b2ebf6913f871347577ad55e869cac1e914e6136e4218808eb26c8 /exec failed with exit code 2
• Test CUDA Builds / export-voxtral-cuda-quantized-int4-tile-packed / linux-job (gh)
  RuntimeError: Command docker exec -t 9089e0115ae76bcefa51b91a0773d4cfbdf166eaaaab143e52a899cc8b4e7c4e /exec failed with exit code 2
• Test CUDA Builds / export-voxtral-cuda-quantized-int4-weight-only / linux-job (gh)
  RuntimeError: Command docker exec -t 8b24745d9bbb0c0cb3c9f97d8d9faf6b2a825e56bdb79c70f61f1030e1c5053e /exec failed with exit code 2
• Test Metal Backend / export-voxtral-metal-artifact / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• periodic (gh)
• trunk / test-models-macos-coreml (edsr) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest / linux / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ArmRyan]

failures unrelated

[robell]

Internal review completed so we're happy with this.

[digantdesai]

This was a huge change (esp in terms of tosa serializer changes), it is breaking internal CI. Let me try to fwd fix. In the future if we are touching to many files, we should also pull internally and test (at least until we improve OSS testing for buck :().

[digantdesai]

I guess tosa/tosa-reference-model repo on gitlab.arm.com is dead (but its there) and long live tosa/tosa-tools?

[ArmRyan]

Yeah individual repos probably wont change anymore, but should stay online. tosa-tools is a monorepo and is the future of the tooling as far as I'm aware

[digantdesai]

re. the tosa-serializer, IIUC we have both c++ (through pybind) and direct python APIs, i.e. tosa_serializer vs. _tosa_serializer why do we have two? Any differences? Are you going to deprecate the direct python ones over pybind ones for perf?

[ArmRyan]

I think its mostly just a quick of the reference model atm, when you pip install reference model it installs old python api ( import serializer.tosa_serializer ), but if you pip install serializer you get c++ ( import tosa_serializer ). Im not sure what the tosa teams plans are, but i think the pure python api is likely going to be frozen where it is, it already lags behind the c++ anyway. For us in ET we have no plans to use the old python version anymore.

Theres not really any difference that I can see, the graphs should serialize to the same flatbuffer and have the same graph, I dont think either one has more or less features right now, but that could change

[digantdesai]

Hmm I am having difficulty in buckify the C++ variant, and since this is breaking things internally, let me revert this for now, we can try to land this again later this week.

[robell]

it will likely impact quite a few follow on fixes which depend on this change if we have to revert this (and so a revert chain) - is there anything we can do to help get this through?

[digantdesai]

The problem is, because of the travel etc. this is blocking a bunch of changes internally since we can't land this due to failures. Do you know which PRs are depending on this? Besides the import name change, this PR (on its own) seems quite independent.

[ArmRyan]

I think its mostly the large revert chain upstream and internally that we were hoping to avoid

[digantdesai]

Any impact on BC esp on Vela 4.4.[12]? I see the CI is OK so should be low risk.

[ArmRyan]

Should be fine, tosa is supposed to be backwards compatible so unless we use new tosa features ( we dont atm ) then nothing should change for vela. The main changes are:
Pybind11 instead of pure python backend, so better perf when lowering. eg. Llama 3.2 1b in int8 goes from ~1h lowering on my laptop to like 3 minutes, and ram usage goes from over 30gb to ~9.
Support for >2GB flatbuffers, these modify the flatbuffer layout if it is >2GB, but there is a guard around ethos-u backend to try and block those from entering vela until its supported there

[ArmRyan]

This was a huge change (esp in terms of tosa serializer changes), it is breaking internal CI. Let me try to fwd fix. In the future if we are touching to many files, we should also pull internally and test (at least until we improve OSS testing for buck :().

It should just be as simple as a setup.sh run, front-end usage should be identical, its purely the internals of the backend that are api broken, but they shouldnt really be exposed so as long as the library exists there should be no change