Audio support

[suragch]

There needs to be a standard way to play and record audio in Flutter apps. Not being able to do that is a show stopper for many types of apps.

Background

I've read a number of related issues regarding this:

• Would like an example of playing music with background execution in Flutter Would like an example of playing music with background execution in Flutter website#9250
• Missing Feature: Streaming and Playing Audio Missing Feature: Streaming and Playing Audio #32399
• [video_player ] Ability to play local mp3 audio files. [video_player ] Ability to play local mp3 audio files. #38480

However, these are more specific than what I am requesting here. The following article has good background on the situation of third party plugins:

• How to add music / audio to your Flutter app

Here are a number of current plugins, each with their own limitations:

• https://pub.dev/packages/assets_audio_player
• https://pub.dev/packages/audioplayers
• https://pub.dev/packages/flute_music_player
• https://pub.dev/packages/stereo
• https://pub.dev/packages/music_player
• https://pub.dev/packages/audio_service

Proposal

I'd like to see an official Flutter plugin that includes the following functionality:

• Play local audio from device
• Play local audio from Flutter asset bundle
• Stream remote audio
• Play audio in the foreground or background
• Record audio on device
• Support for Android
• Support for iOS
• Support for Web
• Support for MacOS
• Support for Windows
• Support for Linux

The plugin should not have any specific skin or widget. Rather, the audio functionality could be incorporated into developers' own widget designs or third-party widget plugins.

Native platforms (Android, iOS, Web, Desktop) already have options for handling audio. Flutter can't be the best way to develop for these platforms unless it can also handle audio.

[Sun3]

I completely agree with @suragch audio is a must have plugin. I have tested all of the main audio plugins and each have limitations. The Flutter team already is supporting the video_plugin and unfortunately audio has been left out.

I hope the audio plugin is created it will help the Flutter platform.

Thank you.

[ghost]

I completely agree with @suragch audio is a must have plugin. I have tested all of the main audio plugins and each have limitations. The Flutter team already is supporting the video_plugin and unfortunately audio has been left out.

I hope the audio plugin is created it will help the Flutter platform.

Thank you.

+1

[suragch]

If we approach this using federation, what should the platform interface be?

[suragch]

Does the video_player plugin solve the needs of playing audio since it works for audio as well? Example

[edyzakaria]

I am learning flutter now, and dont have interest of learning either native or other hybrid. I hope this feature will make flutter complete and mature better through this feature.

[ajonno]

End Jan 2020 I’ll be working with a large media organisation in Australia on a new app and would love to use Flutter. In addition to audio support as noted above CarPlay and Android Auto would be needed too

[nickmeinhold]

@suragch the video_player plugin would not (I assume?) allow for playing audio in the background though
EDIT: sorry, I see now from your article that the video_player plugin uses AVPlayer (iOS) and ExoPlayer (Android) and so I guess no reason background would be an issue. I haven't found how to use video_player for background audio yet but I'll keep looking.

[suragch]

@nickmeinhold I think your probably right about it not supporting background playing.

[richardsric]

A much needed and overdue function. I support @suragch. Normally when you work on video the next is audio. I don't know why they seem to have left it out.

[andrewpmoore]

We've ended using a combination of several audio libraries to get what we needed (streaming background audio with lock screen control) and it's been the slowest section of our app to develop. It really needs an official way, especially with the need for desktop /Web support

[mohammadne]

Also this issue in audio_service can help:
ryanheise/audio_service#109

[Sun3]

@suragch Adding play/skip/forward controls on the lock screen is also a must.

[purplecandy]

The background playback is gonna be challenging the Flutter architecture, because right now Dart runs as long as the FlutterActivity is alive if it gets destroyed, no operations will perform. So the app has to restart the activity which brings your back to mainscreen.

It's one of the reasons they haven't been able to implement any state persistence.

[edyzakaria]

The background playback is gonna be challenging the Flutter architecture, because right now Dart runs as long as the FlutterActivity is alive if it gets destroyed, no operations will perform. So the app has to restart the activity which brings your back to mainscreen.

It's one of the reasons they haven't been able to implement any state persistence.

Do you mean like Spotify? Which once we close the app (not simply going to home, but really close the app), the music will stop play.

[mr-mmmmore]

+1
Playing audio is a basic need for many apps and support from the Flutter team seams logical. Including it in the Flutter API would be ideal.

[tantzygames]

[video_player ] Ability to play local mp3 audio files. #38480 is closed, so I'm replying to @timesneath here.

The video_player plugin is not designed for audio file playback, so this is working as intended.

AVPlayer on iOS and ExoPlayer on Android both support audio, as evidenced by the "Supported Formats" links you provide. If video_player does not support audio then please consider changing this section of your README where it says it does.

But since both underlying systems support audio, there's really no reason not to support audio with this plugin by making into a general purpose Media Player.

I have been modifying video_player to support mixing multiple audio tracks with optional video. Merely enabling audio is quite trivial. For iOS removing only 3 lines enables audio (without causing problems for video), and so far on Android is playing M4A audio tracks so it shouldn't be difficult for it to play other formats.

iOS change:

- (void)sendInitialized {
  if (_eventSink && !_isInitialized) {
    CGSize size = [self.player currentItem].presentationSize;
    CGFloat width = size.width;
    CGFloat height = size.height;

    // The player has not yet initialized.
//    if (height == CGSizeZero.height && width == CGSizeZero.width) {
//      return;
//    }
    // The player may be initialized but still needs to determine the duration.
    if ([self duration] == 0) {
      return;
    }

    _isInitialized = true;
    _eventSink(@{
      @"event" : @"initialized",
      @"duration" : @([self duration]),
      @"width" : @(width),
      @"height" : @(height)
    });
  }
}

[snaeji]

In my opinion @ryanheise has done all the ground work with the audio_service plugin. It has solved all of our background playing issues for Android and getting there on iOS. All that is missing is now is better IOS support for that plugin (in the works) and an audio version of the video_player plugin with ExoPlayer maintained by the Flutter team.

Edit: just_audio also by @ryanheise is using ExoPlayer, what a godsend.

[TenzinProgrammer]

a language that does not have audio plugin of its own seems ridiculous. Still waiting for the flutter team to come with a solution.

[xioxin]

Hope to provide something similar to "Web Audio API".
Provides the ability to process audio, not just playback.

labSound may do it, but I won't use FFI

[mr-mmmmore]

I have built a POC that plays HLS audio using audiofileplayer and it works fine on both iOS and Android. Background audio works well, allthought I haven't ran in-depths tests. It uses the OS' native player (AVPlayer, MediaPlayer), it is still basic but covers the playing functionnalities on both iOS and Android. Could be a starting point.

I'd add this to @suragch proposal:

• Handle HTTP headers when streaming remote audio

HTTP headers may be required for playing some remote streams, such as authorization or cookies (signed cookies for AWS CloudFront for example)

[peterIrving]

Does anyone know what it would look like to write an audio plugin with C++? Would a lot of the work for all of the platforms be able to be cross-platform?

I'm masochistically looking for a reason to learn C. Is this a pointless endeavor? It sounds kind of neat in my head, if you can build a hardware plugin that is cross-platform for the most part in C seems to fit the whole Flutter mentality anyways.

[llucax]

I don't think this is the place to start for several reasons:

• Audio subsystems are usually very dependent on the platform
• The parts that are platform-independent can be written in Dart
• just_audio was recently added to the "Flutter Favorite" list, so it might be better to focus on contributing to this project instead of starting a new one.

I really wonder if this issue should be closed as a "won't fix" as this point. The main issue seems to be how fragmented and incomplete existing audio plugins are rather than the necessity to have built-in audio support.

[ryanheise]

There is actually work underway here: ryanheise/just_audio#103

@alexmercerind has done some impressive work in a short amount of time with a C++ audio library based on vlc and has now made it open source so that you can help him to make it better.

And @megamegax has started working on integrating that into just_audio's federated plugin model. I am sure both authors would love help/collaboration to make things go faster and further.

[alexmercerind]

@ryanheise,
thanks for considering my work.
For anyone who has interest in C++ or Flutter plugins for Windows/Linux in general.
My project is here https://github.com/alexmercerind/dart_vlc, feel free to make pull requests for new features etc.

Not just audio playback, but basic video playback is also added.
Thanks! Let me know, if you need me somewhere.

[mr-mmmmore]

Bravo @alexmercerind! I think that using the VLC library is worth the effort as it is quite mature and versatile. It might ease the support of a truly mobile/desktop cross-platform media player.

[ghost]

Just an idea, why not build a wrapper over alexmercerind/dart_vlc and ryanheise/just_audio that way you'd have an audio pkg that supports all platforms with a uniform Interface.

[ryanheise]

A wrapper will work, and I considered such an approach for adding background support to just_audio using audio_service under the hood, but actually the federated plugin model allows for such things in a more transparent way without needing any API changes and it's about the same effort as a wrapper. This is what @megamegax began working on above, although I would consider contributions to either a wrapper or @megamegax 's work on a federated plugin implementation as a win.

[xioxin]

https://github.com/xioxin/lab_sound_flutter
This is a graph-based audio engine similar to WebAudioApi, which has audio analysis, oscillators, and can be used to play or render audio.
He is still in development and currently only supports Android.
I have some difficulties integrating with other platforms, for example ios I don't know how to add CMakeLists.txt to the ios project.

[alexmercerind]

@xioxin,
I don't believe starting from scratch now is a good idea, @ryanheise's just_audio is undoubtedly the most powerful audio library for Flutter yet and has a lot of time spent in it, its the one that should be taken as a base work.

One more thing I wanna say is that, as I maintain dart_vlc many people have told me that it is really large in size (since it contains all plugins for decoding all media types etc. & goes over 115 MB in size), but that is not good for a app which just wants to embed a little audio playback inside.
If anyone is willing to spend time and even has little experience in Dart, then you should really consider adding Windows/Linux support to just_audio while using dart_vlc as dependency. You just need to write wrapper in Dart, all native code is already done on both parts, @ryanheise will be happy to receive your pull request.

Spoilers

So, I've been working on a new project and using native Windows APIs for media playback this time, and will be using FFI. I plan to create a single DLL (1~3 MB) which users can just throw inside project (I can setup the CMake for it or just pack DLL directly without building on user's machine) and get audio playback.
New media playback APIs on Windows 10 are really good & have good codec support. I believe there is not any benefit on depending on third-party solutions like libVLC since it is a very large library and using it is useless when Windows itself provides a lot.
Drawback of this is that it will be only supported on Windows 10+. (I don't believe that's a big deal, why do you use Windows 7!?)
For Linux however, using libVLC is still fine choice because users can just install VLC package from their respective distro repositories & we don't have a already existing solution common to all distros.

Not just media playback but system media transport controls

(I just passed same string to every field, don't worry its alright)

[xioxin]

@alexmercerind

Thank you for your reply.

lab_sound_flutter is the FFI wrapper for LabSound. I just did the wrapping.

I'm developing a project that requires control over the precise mixing of multiple audio clips. There is no other plugin that meets my needs.
LabSound does not support network audio playback. He is more suitable for audio processing, more like an audio editor. It can play back edits in real time, or render and save them as files.

[AlexV525]

Now the video_player should be able to play audio files locally on iOS since flutter/plugins#4639 has been merged. But the plugin is still focused on video, which means the support for audio playback might be separated or removed in the future.

[Gustl22]

@suragch audioplayers 1.0.x now has been released with support for all platforms. Also it doesn't use other third party plugins to satisfy the various platforms, but provides native implementations for each one. It also only focuses on playing audio, to keep it simple (no playlists, no metadata, no recording). You may have a look :)

[merabtenei]

@suragch audioplayers 1.0.x now has been released with support for all platforms. Also it doesn't use other third party plugins to satisfy the various platforms, but provides native implementations for each one. It also only focuses on playing audio, to keep it simple (no playlists, no metadata, no recording). You may have a look :)

Unfortunately on windows it seems that we can't play m4a format (AAC) and many others, the only format that worked on windows is .wav.

[Gustl22]

Unfortunately on windows it seems that we can't play m4a format (AAC) and many others, the only format that worked on windows is .wav.

That's on Windows. You can see the supported formats here:
https://learn.microsoft.com/en-us/windows/win32/medfound/supported-media-formats-in-media-foundation

But your codec should work. Only .ogg is missing support, see bluefireteam/audioplayers#1255

But I'd propose to discuss that in the audioplayers repo.

[merabtenei]

Unfortunately on windows it seems that we can't play m4a format (AAC) and many others, the only format that worked on windows is .wav.

That's on Windows. You can see the supported formats here: https://learn.microsoft.com/en-us/windows/win32/medfound/supported-media-formats-in-media-foundation

But your codec should work. Only .ogg is missing support, see bluefireteam/audioplayers#1255

But I'd propose to discuss that in the audioplayers repo.

@Gustl22 Even though m4a and AAC is listed in that link, it's definitely not working with audioplayers. The error i get is the same as the one mentionned in this issue : bluefireteam/audioplayers#1100 ... which got no solution as far.

[stuartmorgan]

We'll leave this open to continue to track interest, but to set expectations: we currently don't have any plans to take on this significant new plugin work, and encourage people to use the various existing, popular plugins from the community.

Based on the updated guidance in https://github.com/flutter/flutter/wiki/Issue-hygiene#closing-issues, I'm going to close this issue, better reflecting the expectation that this is not something that the Flutter team plans to implement.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.