Mobile / Edge / ARM / ONNX Use Cases #331

snakers4 · 2021-02-20T06:33:20Z

snakers4
Feb 20, 2021
Maintainer

While the VAD (especially the micro one) was explicitly designed for IOT / edge / mobile use cases, we do not have the resource or expertise to provide instructions for corresponding ARM / mobile builds for PyTorch and / or ONNX.

ONNX guides were refurbished recently and it is implied that ARM binaries will be made available (but they are not yet).

People from the community (see telegram chat) have also claimed successful builds and use of silero-models on pytorch replacing mkl with cblas.

In any case sharing such dockerized builds (e.g. based off debian / ubuntu / alpine) for your tested used cases will be of great value for the community, PRs greatly encouraged and appreciated.

Please see some examples here - https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/README.md#arm-32v7

If you feel like doing something like this - please provide a build in a dockerfile and provide some background info on which arch / device / processor you are running it, if this hardware is generally available, what is the end performance etc

snakers4 · 2021-04-22T04:00:41Z

snakers4
Apr 22, 2021
Maintainer Author

Some ARM builds here
snakers4/silero-models#70 (comment)

0 replies

Newo6174 · 2021-06-10T07:19:40Z

Newo6174
Jun 10, 2021

Builds for armv7

0 replies

snakers4 · 2021-06-10T07:21:59Z

snakers4
Jun 10, 2021
Maintainer Author

Nice, this guy bothered to create a Dockerfile!

0 replies

gstvg · 2021-07-29T19:59:14Z

gstvg
Jul 29, 2021

Tract, written in Rust, can easily be compiled to ARM devices, and streaming audio on edge devices is one of their use-cases. I tried running it but there's an unimplemented op. Support for it is being tracked on above mentioned issue.

0 replies

snakers4 · 2021-09-06T12:58:07Z

snakers4
Sep 6, 2021
Maintainer Author

Running silero-vad on Android - https://github.com/bgubanov/VadExample

You need to use this model - https://github.com/snakers4/silero-vad/blob/master/files/model_micro_mobile.jit

1 reply

john401528 May 26, 2023

How to deploy to iOS? Is use silero_vad.onnx?

snakers4 · 2021-09-16T03:45:40Z

snakers4
Sep 16, 2021
Maintainer Author

Will keep this issue pinned for everyone to see

0 replies

boxabirds · 2021-10-02T08:32:50Z

boxabirds
Oct 2, 2021

I’m looking to do an iOS version.

Can I assume the .jit model file is created by TorchScript so I’d follow the instructions for how to use PyTorch for iOS (which is currently requiring C++)? It does seem like there’s no convention for the file suffix but it seems .pt is actually preferred after a quick search.
How did you pretrain the models? I couldn’t immediately see any source code that produces that file.

0 replies

snakers4 · 2021-10-02T08:48:53Z

snakers4
Oct 2, 2021
Maintainer Author

Can I assume the .jit model file is created by TorchScript so I’d follow the instructions for how to use PyTorch for iOS (which is currently requiring C++)? It does seem like there’s no convention for the file suffix but it seems .pt is actually preferred after a quick search.

The .jit models are indeed created using TorchScript.
A common problem with running them on mobile was STFT (lack of Intel's MKL or something), which was fixed with this micro model - https://github.com/snakers4/silero-vad/blob/master/files/model_micro_mobile.jit. We are planning a large update soon, where all of the models would not have this problem.

People actually successfully managed to run the model on Android - #37 (comment)

Since PyTorch also has packages now and we created this project some time ago, we reserved the more obvious names - .jit for TorchScript, .onnx for ONNX. In hindsight probably none of this really matters.

How did you pretrain the models?

Using our internal datasets and algorithms.
The public models now feature 4 or 5 languages and have some slight issues with post-processing parameter fine-tuning.
Most likely, next version of the models will be simplified (only 1M param model and 10k param model, maybe for 16k and 8k, most likely in JIT and ONNX), will use ~100 languages and be easier to use.

I couldn’t immediately see any source code that produces that file.

We decided against sharing our pipelines since we consider our VAD as a domain agnostic solution as opposed to yet another toolkit. If some tweaks or optimizations are required - we do them as commercial projects.

0 replies

boxabirds · 2021-10-02T08:58:34Z

boxabirds
Oct 2, 2021

Great thanks. BTW your original web article’s source code link is broken.

0 replies

snakers4 · 2021-10-02T09:01:13Z

snakers4
Oct 2, 2021
Maintainer Author

Which of the articles? Can you send the link / tell which exact link is broken?

0 replies

boxabirds · 2021-10-02T09:06:05Z

boxabirds
Oct 2, 2021

This one https://towardsdatascience.com/modern-portable-voice-activity-detector-released-417e516aadde#:~:text=webrtc%20though%20starts%20to%20show%20its%20age%20and%20it%20suffers%20from%20many%20false%20positives.

And the original one this is syndicated from

0 replies

snakers4 · 2021-10-02T09:11:35Z

snakers4
Oct 2, 2021
Maintainer Author

For some reason the English articles has a broken link to TorchHub
Fixed it, many thanks

0 replies

dragen1860 · 2021-10-19T02:28:01Z

dragen1860
Oct 19, 2021

HI, @snakers4 as you pointed, the .jit actully same with .pt. But when i change the extension file name from .jit to .pt, I found the Netron can not visualize the model.pt file. Very confused? thank you.

0 replies

snakers4 · 2021-10-19T05:23:32Z

snakers4
Oct 19, 2021
Maintainer Author

HI, @snakers4 as you pointed, the .jit actully same with .pt. But when i change the extension file name from .jit to .pt, I found the Netron can not visualize the model.pt file. Very confused? thank you.

There is no real convention for these new things, i.e. TorchScript models (jit) or Torch packages (pt).
Since this piece of software (Netron) was probably envisaged and written 1-2 years ago, most likely they faced the same question and decided to keep this distinction as well.

0 replies

snakers4 · 2021-12-07T10:26:29Z

snakers4
Dec 7, 2021
Maintainer Author

The new VAD model in v3.0 release should be compatible with all versions of PyTorch (mobile, ARM, x86, etc) since it does not:

Use built-in FFT
Due to small model size we decided to publish a single non-quantized version of the model

0 replies

snakers4 · 2021-12-10T05:18:02Z

snakers4
Dec 10, 2021
Maintainer Author

Some interesting comments on this topic worthy of sharing with the general public:

0 replies

snakers4 · 2021-12-17T15:30:15Z

snakers4
Dec 17, 2021
Maintainer Author

The new release has an ONNX model, albeit only for 16 kHz - https://github.com/snakers4/silero-vad/releases/tag/v3.1

0 replies

uloveqian2021 · 2021-12-21T06:45:48Z

uloveqian2021
Dec 21, 2021

Can I assume the .jit model file is created by TorchScript so I’d follow the instructions for how to use PyTorch for iOS (which is currently requiring C++)? It does seem like there’s no convention for the file suffix but it seems .pt is actually preferred after a quick search.

The .jit models are indeed created using TorchScript. A common problem with running them on mobile was STFT (lack of Intel's MKL or something), which was fixed with this micro model - https://github.com/snakers4/silero-vad/blob/master/files/model_micro_mobile.jit. We are planning a large update soon, where all of the models would not have this problem.

People actually successfully managed to run the model on Android - #37 (comment)

Since PyTorch also has packages now and we created this project some time ago, we reserved the more obvious names - .jit for TorchScript, .onnx for ONNX. In hindsight probably none of this really matters.

How did you pretrain the models?

Using our internal datasets and algorithms. The public models now feature 4 or 5 languages and have some slight issues with post-processing parameter fine-tuning. Most likely, next version of the models will be simplified (only 1M param model and 10k param model, maybe for 16k and 8k, most likely in JIT and ONNX), will use ~100 languages and be easier to use.

I couldn’t immediately see any source code that produces that file.

We decided against sharing our pipelines since we consider our VAD as a domain agnostic solution as opposed to yet another toolkit. If some tweaks or optimizations are required - we do them as commercial projects.

Hello, when I use the new models Running silero-vad on Android - https://github.com/bgubanov/VadExample , there are still errors about "RuntimeError: stft: ATen not compiled with MKL support". Is there a solution here?

0 replies

snakers4 · 2021-12-21T06:52:15Z

snakers4
Dec 21, 2021
Maintainer Author

errors about "RuntimeError: stft: ATen not compiled with MKL support". Is there a solution here?

@uloveqian2021, you are using the new V3 model, right? This one - https://github.com/snakers4/silero-vad/blob/master/files/silero_vad.jit ?

The old model works?

This model should not contain torch.stft, unless we made a mistake during export

@adamnsandle

Can you check this please?

0 replies

adamnsandle · 2021-12-21T08:58:39Z

adamnsandle
Dec 21, 2021
Collaborator

Hello, when I use the new models Running silero-vad on Android - https://github.com/bgubanov/VadExample , there are still errors about "RuntimeError: stft: ATen not compiled with MKL support". Is there a solution here?

Our current jit and onnx models do not contain torch.stft, although initial 3.0 release was with torch.stft, then we replaced it with mobile supported analogue

Please try latest models, it should work

0 replies

snakers4 · 2021-12-21T09:19:02Z

snakers4
Dec 21, 2021
Maintainer Author

@uloveqian2021
Can you please check that with the latest model the problem goes away

0 replies

wangbq18 · 2021-12-21T14:08:17Z

wangbq18
Dec 21, 2021

@uloveqian2021 Can you please check that with the latest model the problem goes away

Yes, It's work now, Thanks!

0 replies

wangbq18 · 2021-12-21T14:08:39Z

wangbq18
Dec 21, 2021

Hello, when I use the new models Running silero-vad on Android - https://github.com/bgubanov/VadExample , there are still errors about "RuntimeError: stft: ATen not compiled with MKL support". Is there a solution here?

Our current jit and onnx models do not contain torch.stft, although initial 3.0 release was with torch.stft, then we replaced it with mobile supported analogue

Please try latest models, it should work

Yes, It's work now, Thanks!