ASR JNI bindings

[rohansjoshi]

Summary

Added JNI bindings to run Qualcomm Whisper Runner in Android

Test plan

CI tests

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13525

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 6afb221 with merge base 2640a86 ():

NEW FAILURE - The following job has failed:

• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 6c53d1cc10a2d1d815c3d1ac96cb8ed9a69d9d3c5d023c70010871c7d3179f47 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[GregoryComer]

I'm a little worried about maintaining model-specific APIs. Would it be possible to build it under examples or similar?

There is definitely a need to provide these sort of task-specific driver APIs, though it might be nice to have a more general AudioToTextRunner (or something similar). If we do land this as is, can we make sure it's clear that it's unstable and should not be used in production? Even if it is marked experimental in code, if it's under the Android extension, there is an implicit assumption that it will be supported.

My concern is that we'll eventually have a more general task-specific runner and will want to migrate to that without hitting BC guarantees. Having this Whisper-specific code under examples would potentially discourage production use until then.

@jackzhxng @larryliu0820 How would you expect Whisper to fit into our runner strategy in the long term?

[rohansjoshi]

I'm a little worried about maintaining model-specific APIs. Would it be possible to build it under examples or similar?

There is definitely a need to provide these sort of task-specific driver APIs, though it might be nice to have a more general AudioToTextRunner (or something similar). If we do land this as is, can we make sure it's clear that it's unstable and should not be used in production? Even if it is marked experimental in code, if it's under the Android extension, there is an implicit assumption that it will be supported.

My concern is that we'll eventually have a more general task-specific runner and will want to migrate to that without hitting BC guarantees. Having this Whisper-specific code under examples would potentially discourage production use until then.

@jackzhxng @larryliu0820 How would you expect Whisper to fit into our runner strategy in the long term?

Right now for LLMs the API is we have Java class LlmModule which wraps a C++ class which stores a pointer to a IRunner runner, and in the constructor we specify which runner to use. So one idea could be to have a ASRModule Java class wrap a C++ class which has a runner which implements some interface we could call AudioToTextRunner. Then the Whisper Runner in examples/qualcomm could implement this interface.

[jackzhxng]

Ideally we can fit Whisper into our multimodal runner strategy but it's not super high priority at the moment. I think it's fine to have an ASR module though, but I agree we shouldn't have anything Whisper specific in there

[mergennachin]

@kirklandsign @jackzhxng -- Why did we approve this PR? This should be reviewed carefully, as we're adding first-class citizen APIs to our extension.

Also, cc @larryliu0820 - please keep an eye on these kinds of PRs

[mergennachin]

Constructor for ASRRunner should contain configs such as temperature and other configs.

[mergennachin]

Also what about taking tokenizer and module as part of the constructor?

[rohansjoshi]

Most of the configs (temperature, echo, etc) which are encapsulated in the GenerationConfig struct for LLM runners aren't relevant for ASR. ASRRunner is supposed to be a general interface like IRunner so I didn't put any constructor here

[mergennachin]

top-level ASR component shouldn't depend on example Whisper

[mergennachin]

You should refactor out general purpose ASR logic out of this file and put into asr_runner.h/cpp

[larryliu0820]

Semi related to the diff, I think we started to recognize a plugin pattern here, it would be better if we can rely on static initialization to register these natives and just iterate through them in line 535. This way we don't need all these macros, we just link the necessary library and it register itself.

[meta-cla]

Hi @rohansjoshi!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!