Support for Audio AC-4 samples with variable frame sizes

[HESA-GrahamClift]

[REQUIRED] Use case description

In the case where audio and video is delivered over MPEG DASH, it can be beneficial for the audio and video segments to have equal duration (improves channel change time for instance). If the audio is not PCM, for instance AC-4, and when the frame rate of the audio is not an integer multiple of the frame rate of the video (eg 48000Hz, 59.94fps), then the AC-4 sample frame count will not be constant.
In DefaultAudioSink, handleBuffer method, there is an assumption that the framesPerEncodedSample needs to be read just once at the beginning. In variable frames per sample cases this can deliver incorrect result to framesToDuration(getWrittenframes()), and in some cases a drift can be seen in the currentPosition when the erroneous result is less than the current position reported from native calls to the decoder.

Proposed solution

Remove the test in handleBuffer for getting the framesPerEncodedSample for the first sample only. Instead, extract the count for every sample.

if (!configuration.isInputPcm /*&& framesPerEncodedSample == 0*/) {
    // If this is the first encoded sample, calculate the sample size in frames.
    framesPerEncodedSample = getFramesPerEncodedSample(configuration.outputEncoding, buffer);
    if (framesPerEncodedSample == 0) {
         // We still don't know the number of frames per sample, so drop the buffer.
         // For TrueHD this can occur after some seek operations, as not every sample starts with
         // a syncframe header. If we chunked samples together so the extracted samples always
         // started with a syncframe header, the chunks would be too large.
         return true;
     }
 }

Alternatives considered

None

[krocard]

Not all formats support variable frame size, so this detection should only be enable for those that do.

Another issue, is that the current code to parse AC-4 frame sample count (and the other format) allocates multiple times. As ExoPlayer policy is that (non special) frame processing should not need allocation, we would need to benchmark if this causes a performance regression or rewrite the code to avoid allocation.

Finally, in audio offload, each buffer contains multiple encoded frame. Thus frame concatenation, which is now done in the Renderer, would need to be move in DefaultAudioSink, after frame size parsing. Such rework is not trivial.

I will add this request to our backlog.

[HESA-GrahamClift]

Would an averaging of the first say 20 samples be sufficient at least to overcome the drift issue without big performance concerns understanding that most encoders (live broadcast ATSC3 for instance) I have seen tend to track the target average to meet the video frame rate rather than adjust all at once or randomly. So for video frame rate of 59.94fps and a dash segment size of, say 2 secs, a fps of audio of 48000Hz, an audio sample rate of 30samples per second, we might expect to see sample frame sizes like 1602, 1601,1602, 1602, 1601, 1602 etc.

A simple 20 sample average should prevent drift issues that impact play position. Another option is to pick the largest number in the set. This would prevent the Min function from kicking in and effecting the drift.

[krocard]

I think it is preferable to avoid relying on assumptions about the media encoding. Especially when refactoring parseAc4SyncframeAudioSampleCount to avoid allocations is of similar complexity.