proposal: x/mobile audio

[rakyll]

In the scope of the Go mobile project, an audio package that supports decoding and playback is a top priority. The current status of audio support under x/mobile is limited to OpenAL bindings and an experimental high-level audio player that is backed by OpenAL.

The current experimental audio package fails to

• provide high level abstractions to represents audio and audio processors,
• implement a memory-efficient playback model,
• implement decoders (e.g. an mp3 decoder),
• support live streaming or other networking audio sources.

In order to address these concerns, I am proposing core abstractions and a minimal set of features based on the proposed abstractions to provide decoding and playback support.

See the proposal document for further details.

[gopherbot]

CL https://golang.org/cl/17262 mentions this issue.

[mattetti]

It looks great!

How about adding AudioFormat string to the Clip interface? For instance a wav audio file can be encoded in a variety of coding formats. It might not belong to the interface since it's a detail that maybe only the decoder cares but I can think of a few examples where it would be useful.

WAV and AIFF both support float PCM values even though the use of float values is unpopular. Should we consider supporting float values?

I think we should, any kind of audio processing would more than likely done in float values, using float right away seems to be a better move IMHO (not a deal breaker tho).

Decoding on desktop. The package will use the system codec libraries provided by Android and iOS on mobile devices. It is not possible to provide feature parity for desktop envs in the scope of decoding.

We can implement the decoders in pure Go code for when codecs are not available. I have partially implemented such decoders and while they might not be as fast as dedicated/optimized codecs, I think they should work well enough.

[mattetti]

L117 small typo, Clips represents PCM audio should read Clips represent PCM audio

[mattetti]

Question, why would we have a generic DecodeWAVBytes decoder function? It seems very specific to wav decoders only, doesn't it? DecodeBytes might be a better choice (?)

[mattetti]

I created a quick and dirty but conform aiff decoder to get a feel for the API: https://github.com/mattetti/exp/blob/master/audio/aiff/decoder.go

I think it might be missing an API to get frames out of a clip. The current proposal goes all the way to get the clip and frame info but we can't get the actual PCM data (which would probably some sort of [][int]float format). Depending on the use case, we might not want to load the entire audio file in memory, so the API might want to take that in consideration.

What do you think?

[rakyll]

Thanks for the feedback!

How about adding AudioFormat string to the Clip interface? For instance a wav audio file can be encoded in a variety of coding formats. It might not belong to the interface since it's a detail that maybe only the decoder cares but I can think of a few examples where it would be useful.

The AudioFormat would represent the coding format of the original source, not the Clip's. The decoders won't work with Clips but arbitrary encoded streams of bytes and convert them into a Clip. The decoder can return the audio format of the original source. Consider the following decoder function:

func Decode(src io.ReadSeeker) (clip Clip, format string, err error)

I think we should, any kind of audio processing would more than likely done in float values, using float right away seems to be a better move IMHO (not a deal breaker tho).

This is unfortunately coming with a performance cost. We need to find a way to support float values but also allow integer-only values for faster processing for those who doesn't care about float-level precision.

We can implement the decoders in pure Go code for when codecs are not available.

We won't have enough human resources probably to reimplement decoders in Go, therefore using the available codecs is going to be our initial step. The proposal is not against of a decoder being implemented in vanilla Go though.

Question, why would we have a generic DecodeWAVBytes decoder function? It seems very specific to wav decoders only, doesn't it? DecodeBytes might be a better choice (?)

It was a sample for decoder implementations, I was not proposing it to be in the audio package. DecodeBytes is a better name in any case, I will update the proposal.

I think it might be missing an API to get frames out of a clip.

We had an earlier debate about providing APIs that work with frames rather than byte slices. The previous API was in the lines of what's below.

type Frame []int64

type Clip interface {
     Read(f []Frame, offset int64) (n int, err error)
}

There are two arguments against the frame-based APIs:

• It is easy to calculate how long a frame is, therefore working with byte slices doesn't require much additional work.
• Read(f []Frame, offset int64) (n int, err error) almost reads like io.ReadSeeker. Not introducing a new interface type allows us to reuse utility functions from the stdlib on Clip instances (such as io.Copy).

[mattetti]

My only concern is that a frame from a wav file can contain data using different encoding and non PCM formats will definitely not have PCM frames. For instance, if one would be to write a mp3 decoder, would the decoder have to provide a clip with an io.ReadSeeker that would return PCM data only?
If that's the case, you might want to make that a little bit clearer.

[rakyll]

For instance, if one would be to write a mp3 decoder, would the decoder have to provide a clip with an io.ReadSeeker that would return PCM data only?

Correct. You can read only and only PCM formatted data from clips. A decoder is responsible to decode any input down to a PCM formatted frames. I am going to update the document to make it clearer.

[mewmew]

provide high level abstractions to represents audio and audio processors,

Some work has already been done in this domain, by the audio package of Azul3d. It is intended to define interfaces analogous to the Image interface of the standard library for audio.

Perhaps some ideas could be stolen from its design. @slimsag is the original developer, and he may have some insight into problems encountered when designing the interface, and how they were solved by Azul3d.

implement decoders (e.g. an mp3 decoder),

There exist at least two FLAC decoders in pure Go.

1. https://github.com/eaburns/flac (GoDoc)
2. https://github.com/mewkiz/flac (GoDoc)

The second FLAC package is conceptually a back-end for decoding FLAC audio files, for which one or more front-end may be implemented. Currently a front-end has been implemented for the Azul3d audio abstraction, and it should be trivial to implement another front-end for the Go mobile audio abstraction. Analogously to the image package of the standard library, to decode a FLAC audio stream, it suffices to have

import _ "azul3d.org/audio/flac.v0"

in a program's main package.

Below follows a short example program for decoding FLAC audio files with Azul3d:

package main

import (
    "flag"
    "fmt"
    "log"
    "os"

    "azul3d.org/audio.v1"
    _ "azul3d.org/audio/flac.v0" // Add FLAC decoding support.
)

func main() {
    flag.Parse()
    for _, path := range flag.Args() {
        err := parseFLAC(path)
        if err != nil {
            log.Fatal(err)
        }
    }
}

func parseFLAC(path string) error {
    // Open file.
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()

    // Create decoder.
    dec, _, err := audio.NewDecoder(f)
    if err != nil {
        return err
    }

    // Decode audio stream.
    for {
        samples := make(audio.PCM32Samples, 1024)
        n, err := dec.Read(samples)
        if err != nil {
            if err == audio.EOS {
                return nil
            }
            return err
        }
        fmt.Println(samples[:n])
    }
}

I hope this may give some ideas, and hope improve the API and design of the Go mobile audio package.

Cheers /u

[rakyll]

Thanks for the input. The proposal is currently on hold and require more work on a variety of topics:

• The interfaces need to consider VBR compression.
• Seeking is often expensive, we may adopt an io.ReadSeeker-like interface to make Seek a standalone operation to put some more emphasis on its cost.
• It is not clear how big a buffer is required for a frame for VBR encoded media. A Frame type may be to introduced.
• We need working prototype decoders for AIFF, FLAC, MP3 and OGG before finalizing the proposal.