[Feature Request] Add support for ReplayGain

[breversa]

Hi !
Dozens of Android audio players, but only a handful of them actually support ReplayGain, and none of them are ExoPlayer-based AFAIK.

So would implementing ReplayGain support in ExoPlayer allow the batch of ExoPlayer-based players (such as OxygenCobalt/Auxio#7 ?) to natively honor ReplayGain tags (especially R128 for FLAC) ?

[marcbaechinger]

I actually don't really know what your request is asking for in detail I'm afraid. :)

natively honor ReplayGain tags

If you let me know what tags this is refering too in which media format, the I can probably tell how you can access these tags so you can implement support for ReplayGain yourself. You probably also want to clarify what you mean by 'natively honor`. If this would be adjusting the volume according to the tags found, then you can probably do this yourself by listening to the tags and then using the API to change the player volume.

So either way, to implement this yourself or to give us a bit more information of what this would mean, you should provide us with some more information.

[breversa]

Thank you @marcbaechinger for your reply, and sorry for not being clear enough, so here's some more info:

First of all, I'm no Android developer, but just a music app user.

I thought I understood that many Android music apps are basically some kind of front-end to ExoPlayer that handle the media loading/playing/etc.

And I thought that tag-reading was part of that too, and that if ExoPlayer doesn't read ReplayGain tags, then the music apps built upon it won't be able to apply the ReplayGain tags (that's why apps like Vanilla Music and Foobar2000 Mobile had to write their own tag-reader).

[marcbaechinger]

According to the Wikipedia article you linked, the tag is an 8-byte field that is delivered with a tag of the given format. Different audio formats have different ways of including these tags in the binary file. The article mentions FLAC, Vorbis and MP3. I think we deliver tags for these formats already via MetadataOutput. So regarding native support I think this should be possible with ExoPlayer.

While I see the point in your request it is very unspecific. I can't remember developers asking for supporting this here on GitHub. This can be because there is no interest in this, or there are not many audio files taged like this. It can also be that it's just already possible to get the tags from the metadata output of ExoPlayer and apply the volume change.

I leave this issue open and mark it as an enhancement. However, it's unlikely that we support this out of the box so that it is just applied by the library. That's more of an app responsibility to do. If we get from developers that the tag can't be read for a certain audio format they are using in their app, we may look into this to provide access to the tag and make it possible for apps to support it.

[OxygenCobalt]

@marcbaechinger Hey, I'm a developer of a music app that would benefit from this support, so I'm trying to implement this feature. What you said about MetadataOutput is interesting. I tried to attach a MetadataRenderer to my player instance and then add an instance of MetadataOutput, like below:

val audioRenderer = RenderersFactory { handler, _, audioListener, _, _ ->
    arrayOf(
        MediaCodecAudioRenderer(this, MediaCodecSelector.DEFAULT, handler, audioListener),
        MetadataRenderer(
            MetadataOutput {
                Log.d("MetadataOutput", "Metadata received")
            },
            handler.looper
        )
    )
}

val extractorsFactory = DefaultExtractorsFactory().setConstantBitrateSeekingEnabled(true)

return ExoPlayer.Builder(this, audioRenderer)
    .setMediaSourceFactory(DefaultMediaSourceFactory(this, extractorsFactory))
    .build()

But this doesn't seem to call the MetadataOutput instance at all, even with media that has metadata. Is there anything that I'm doing wrong? If I'm able to get metadata through this, I'll implement replaygain support on my end and then attach the code to this thread so that others can implement support for this easily.

[marcbaechinger]

Can you add your media to the demo app and then see if the metadata is reported? You would see this in the logs of the demo app or when placing a breakpoint in the EventLogger.

Alternatively you can post a link to the media here, explain what tags you expect so I can have a look. If you're unable to share test content publicly, please send them to dev.exoplayer@gmail.com using a subject in the format "Issue #9796". Please also update this issue to indicate you’ve done this.

[OxygenCobalt]

Metadata is not reported on the demo app either, but any audio/video Renderer instance reports it just fine. I've decided just to override onTracksInfoChanged and just get the metadata from the currently selected track, as that seems to work.

[marcbaechinger]

Can you please send us a sample URI to a stream and let me know what metadata you expect?

I would expect that onMediaMetadataChanged(MediaMetadata mediaMetadata) is called when it arrives. I marked this as a bug so we investigate this more closely as soon as we have your link to the content.

If you're unable to share bug reports or test content publicly, please send them to dev.exoplayer@gmail.com using a subject in the format "Issue #9796".

[icbaker]

Note that MetadataOutput will not receive the same info as Player.ListeneronMediaMetadataChanged. Specifically, MetadataOutput will only received dynamic/timed metadata that's decoded during playback (e.g. MP4 event messages). onMediaMetadataChanged will receive this dynamic metadata, as well as static metadata which stays constant for the length of a file (e.g. ID3 tags read at the start of an MP3 file).

By the description of this ReplayGain metadata, it sounds like static ID3-like metadata, so I suspect it's expected that it doesn't get delivered to MetadataOutput.

If you want to receive a callback when either the static or dynamic metadata changes (either because dynamic metadata arrives, or the track changes to one with different static metadata) then Player.Listener.onMediaMetadataChanged will provide this. More info: https://exoplayer.dev/retrieving-metadata.html

From the info provided I'm not convinced there's a bug here, so I'm going to mark this back as an enhancement.

[OxygenCobalt]

Oh. I assumed MetadataRetriever and MetadataOutput read static metadata due to their names and documentation. However, I don't think I can use Listener.onMediaMetadataChanged since MediaMetadata does not provide raw metadata entries, which is required for ReplayGain as it relies on custom tags such as ID3v2's TXXX. Still, I can just use onTracksInfoChanged and do my static metadata parsing there. Thanks for clarifying anyway.

[marcbaechinger]

Can you provide me with a media file I can try with?

[OxygenCobalt]

As in media with replaygain tags, or media with TXXX tags? The former is pretty hard, as ReplayGain tags are super diverse, but I could give some files with TXXX and others if you want to implement support for that.

[marcbaechinger]

I would be interested to see specific ReplayGain streams.

TXXX ID3v2 tags are delivered to onMetadata(MetaData). I've used the code below for accessing it:

public void onMetadata(Metadata metadata) {
      for (int i = 0; i < metadata.length(); i++) {
        Metadata.Entry entry = metadata.get(i);
        if (entry instanceof TextInformationFrame) {
          TextInformationFrame textFrame = (TextInformationFrame) entry;
          if ("TXXX".equals(textFrame.id)) {
            // do something with TXXX tags.
          }
        } 
      }
    }

Does this work for you?

[OxygenCobalt]

Huh. I'd expect TXXX to be static metadata. I know the most common ReplayGain tags in ID3v2 are based on TXXX, so I'll try to create a file that contains such and post it here, alongside testing if TXXX is indexed by onMetadata.

[marcbaechinger]

You and icbaker are probably right. That's why I wanted to have a media piece for testing. I don't have much experience with static metadata to be honest. The ID3 tags I have worked with have been delivered as in-band metadata that was received as timed metadata in HLS and I mean to remember this was similar for artworks in audio files. However, I'm probably wrong as I said. :) Lets check as soon as you posted some content here.

[OxygenCobalt]

Sounds good. Let me source an actual ReplayGain file from one of my users.

[OxygenCobalt]

Heres some sample files. Sorry for taking so long, I was busy with other projects. I did test these files, and onMetadata is not called. I'm still trying my hand at creating an implementation.

replaygain.zip

[OxygenCobalt]

Okay, I've created a ReplayGain implementation that works with both ID3v2 and FLAC [In the form of R128], largely derived from Vanilla Music. The overall code is below:

import android.util.Log
import androidx.core.math.MathUtils
import com.google.android.exoplayer2.ExoPlayer
import com.google.android.exoplayer2.metadata.Metadata
import com.google.android.exoplayer2.metadata.flac.VorbisComment
import com.google.android.exoplayer2.metadata.id3.TextInformationFrame
import kotlin.math.pow

const val RG_TRACK = "REPLAYGAIN_TRACK_GAIN"
const val RG_ALBUM = "REPLAYGAIN_ALBUM_GAIN"
const val R128_TRACK = "R128_TRACK_GAIN"
const val R128_ALBUM = "R128_ALBUM_GAIN"

val replayGainTags = arrayOf(
    RG_TRACK,
    RG_ALBUM,
    R128_ALBUM,
    R128_TRACK
)

data class Gain(val track: Float, val album: Float)

/**
 * Applies replaygain to an ExoPlayer instance. This is based off Vanilla Music's implementation.
 */
fun applyReplayGain(player: ExoPlayer, metadata: Metadata) {
    val gain = parseReplayGain(metadata)

    // Currently we consider both the album gain first. One might want to add
    // configuration to handle more cases.
    var adjust = 0f

    if (gain != null) {
        adjust = if (gain.album != 0f) {
            gain.album
        } else {
            gain.track
        }
    }

    // Final adjustment along the volume curve. 
    // Ensure this is clamped to 0 or 1 so that it can be used as a volume.
    adjust = MathUtils.clamp((10f.pow((adjust / 20f))), 0f, 1f)

    Log.d("applyReplayGain", "Applying ReplayGain adjustment: $adjust")

    player.volume = adjust
}

private fun parseReplayGain(metadata: Metadata): Gain? {
    data class GainTag(val key: String, val value: Float)

    var trackGain = 0f
    var albumGain = 0f
    var found = false

    val tags = mutableListOf<GainTag>()

    for (i in 0 until metadata.length()) {
        val entry = metadata.get(i)

        // Sometimes the ReplayGain keys will be lowercase, so make them uppercase.
        if (entry is TextInformationFrame && entry.description?.uppercase() in replayGainTags) {
            tags.add(GainTag(entry.description!!.uppercase(), parseReplayGainFloat(entry.value)))
            continue
        }

        if (entry is VorbisComment && entry.key.uppercase() in replayGainTags) {
            tags.add(GainTag(entry.key.uppercase(), parseReplayGainFloat(entry.value)))
        }
    }

    // Case 1: Normal ReplayGain, most commonly found on MPEG files.
    tags.findLast { tag -> tag.key == RG_TRACK }?.let { tag ->
        trackGain = tag.value
        found = true
    }

    tags.findLast { tag -> tag.key == RG_ALBUM }?.let { tag ->
        albumGain = tag.value
        found = true
    }

    // Case 2: R128 ReplayGain, most commonly found on FLAC files.
    // While technically there is the R128 base gain in Opus files, ExoPlayer doesn't
    // have metadata parsing functionality for those, so we just ignore it.
    tags.findLast { tag -> tag.key == R128_TRACK }?.let { tag ->
        trackGain += tag.value / 256f
        found = true
    }

    tags.findLast { tag -> tag.key == R128_ALBUM }?.let { tag ->
        albumGain += tag.value / 256f
        found = true
    }

    return if (found) {
        Gain(trackGain, albumGain)
    } else {
        null
    }
}

private fun parseReplayGainFloat(raw: String): Float {
    return try {
        raw.replace(Regex("[^0-9.-]"), "").toFloat()
    } catch (e: Exception) {
        0f
    }
}

It doesn't handle every case, notably OPUS files have a volume gain adjustment in their header that should also be handled, but ExoPlayer doesn't parse that metadata. Feel free to use my code however you want.

Personally, I think ReplayGain would be best left to the developer to implement simply due to how obscure the use-case is. But then again, that's what they also said about ICY streams, so I guess it's up to the maintainers.

[OxygenCobalt]

Also, a quick request: It would be quite helpful to access metadata from files other than ID3v2 or FLAC. OGG is pretty widely used and contains effectively the same metadata as FLAC, but ExoPlayer doesn't seem to parse it.

Edit: I just enabled OGG support. Turns out ExoPlayer already parses it but doesn't expose it. You can find it in my fork.

[google-oss-bot]

Hey @breversa. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@breversa if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.