[docs] Two audio backends are available, not four as stated in docs

[vadimkantorov]

📚 The doc issue

Is ffmpeg backend also implemented? Something else?

https://pytorch.org/audio/stable/backend.html#backend :

[vadimkantorov]

ffmpeg backend would be useful, even just for info function

• a feature request: info function should have an option to demux all packets (without actual audio decoding) and to return latest pts_time. Certain webm+opus files may not have a duration in headers. This can certainly be done with ffprobe/ffmpeg in-process

[vadimkantorov]

Any comments about ffmpeg backend in general? (it would also be nice to accept an explicit backend= argument to dispatching torchaudio.load to not rely on multiprocessing-fishy global backend variables)

It also would be nice if docs clarified that Python's pysoundfile bindings to libsndfile is used, and not libsndfile directly (right?)

[nateanl]

Hi @vadimkantorov, thanks for sharing the post. Since FFMpeg is cross-platform, it is not necessary to set it as a backend, instead users can use it via torchaudio.io class regardless of the types of operating system.

info function should have an option to demux all packets (without actual audio decoding) and to return latest pts_time. Certain webm+opus files may not have a duration in headers. This can certainly be done with ffprobe/ffmpeg in-process

cc @mthrok is adding pts_time in the info function doable?

[nateanl]

It also would be nice if docs clarified that Python's pysoundfile bindings to libsndfile is used, and not libsndfile directly (right?)

That's right. It is mentioned in the backend doc and has a link to the PySoundFile webpage.

[vadimkantorov]

Well, it's more

thanks for sharing the post. Since FFMpeg is cross-platform, it is not necessary to set it as a backend, instead users can use it via torchaudio.io

How is backend / io-function API related to cross-platformedness? both sox and soundfile are probably also cross-platform. By backend I mean having a single interface of torchaudio.load / torchaudio.save / torchaudio.info being able to work with sox/soundfile/ffmpeg, where the library ("backend") can be chosen by the user.

Regarding fetching duration from pts_time of the last packet - this is more relevant for webm and other general containers. This requires some form of seek or full input stream demuxing (hopefully without decoding). This can be easily done with ffmpeg, but also probably with other "backends" depending on what API they provide. Sometimes even having a full extra decoding pass is practical. This is related to avoiding OOM / denial-of-service by checking input size and capping it. An alternative could be some load functions that support out= argument and let check if overflow was needed. Then probably load function with a start time seek would be useful to be able to read audio from the last ended moment. A small compressed audio can generate a huge decoded audio (e.g. if there is a lot of silence in it).

[mthrok]

The backend was introduced to handle the situation where libsox is not available on Windows. This attempt IMHO was unsuccessful as the pysoundfile and libsox do not provide the same coverage, it made the library stateful, and it's hard to know what project sets what backend. So I am hoping to get rid of the backend somehow (and when doing so perhaps make the current backend implementation public APIs). So I do not plan to add a new backend.

FFmpeg features are all put in torchaudio.io, since FFmpeg is available on all platforms that torchaudio supports (Windows, macOS, Linux) and the current FFmpeg-based features do not fit in simple load/save/info function scheme.

[vadimkantorov]

I agree about being stateful as a bad design decision.

As a backend I mean only having a stable / similar load/save/info API supporting different IO libraries.

Interesting about SoX not available on Windows, from what I understand, there is no official distributed dll on Windows, but people manage to build it on their own...

[mthrok]

How is backend / io-function API related to cross-platformedness? both sox and soundfile are probably also cross-platform. By backend I mean having a single interface of torchaudio.load / torchaudio.save / torchaudio.info being able to work with sox/soundfile/ffmpeg, where the library ("backend") can be chosen by the user.

Interesting about SoX not available on Windows, from what I understand, there is no official distributed dll on Windows, but people manage to build it on their own...

The thing is we need to automate the build process. For this I obtained Windows machine and learned Windows development from scratch (and succeeded to automate FFmpeg-related CI job that was necessary to include FFmpeg features in binary distributions) but Windows support of libsox is not maintained well. It does not compile on recent Visual Studio. I believe the last VS version the official build script supports is something like 2013, I come back to this from time to time, and I try but I never succeed.

As a backend I mean only having a stable / similar load/save/info API supporting different IO libraries.

Sure, but there are other issues on the existing API so I am not sure how much we should mimic the current API, notably, normalize argument is confusing for audio experts as they think its volume/power normalization while in reality it's just type (and value range) conversion. For this reason, I am thinking that FFmpeg-based implementations of such APIs can be put in torchaudio.io with fresh signature, but then I am sure some ppl will find it confusing to have multiple APIs. So I am somewhat in a pickle.