Get subtitle with timecode from video and save it into Lyrics metadata of audio only file: --embed-subs --extract-audio

[porg]

Status Quo

• --embed-subs is currently for videos only
• Some audio file formats and in particular the ID3 metadata format supports to embed timecoded lyrics, see: https://id3.org/Lyrics3v2
• And some audio players show you these lyrics nicely, in sync with playback, e.g. Apple Music:

Proposal

• When you use --embed-subs in conjunction with --extract-audio and write to your audio only file with --output the resulting audio file should get the subtitles encoded into its Lyrics metadata.
• Ideally the lyrics remain timecoded, if the destination file format and/or its metadata format supports this.
• Ideally automatic
  • that means choosing the best suited setting for the given audio output file format without user intervention
  • but if necessary accompanied with additional arguments to specify the lyrics formatting options in detail

Use Cases

• Learn/understand the lyrics of your favorite (foreign) music
• Sing to your Karaoke audio files
• Aid in foreign language learning
  • Listen to audio on Smartphone while looking on lyrics
    • in spoken language
    • in original script or transcribed script
    • or translated into a language that you master

[porg]

There seems to be an app which does "Youtube + Lyrics + Album Art → MP3" out of the box: alltomp3-app

• Problem is: Not maintained anymore, last release 2020-11-17
• Uses yt-dl as its download backend
  • And as we know here yt-dl being outdated/broken was the reason why yt-dlp got into existence
• So maybe some of the lyrics functionality from alltomp3-app can be sourced and integrated into yt-dlp ?

[pukkandan]

Where are you suggesting we get the lyrics from? Give example URL

[porg]

@pukkandan

• My proposal was: If a Youtube video features a subtitle track (as WEBVTT, in the context of a music video we refer to this as "lyrics", in a video/sketch/theater-play this may be referred to as "dialog" or "narration", in a fairy tale it is likely called "narration", etc) then embed it into the "lyrics" metadata section of an audio only file (mp3, aac, opus, etc) , see: Lyrics3v2.
• If a Youtube video features no subtitles (neither manual nor AI-auto-generated) then this fails:
  • With --ignore-errors it fails silently and simply just writes the audio file without subtitles embedded as lyrics.
  • Without --ignore-errors it fails totally, ideally early on: First checks whether (any/requested) subtitle tracks are present in the Youtube video, and it they ain't present, doesn't download the audio track at all, and aborts.
• To be clear: I did not propose in any way to search the lyrics in an external database.
  • I only propose to create an embedding feature on the output file for data that is already present at the source.

[pukkandan]

For devs: https://stackoverflow.com/a/61166697/12907559

[october262]

@october262 Thanks for the hint.

• I've been aware of the possibility to add lyrics via the "Lyrics" tab of the file editor in Apple Music (formerly iTunes).
• The downside of this method: It's text only. Not synced text with timecodes, which can be shown in sync with playback.
• I tried pasting text there in all possible subtitle formats: SRT, WEBVTT, LYR. They all get shown as pure text only (with all the timecode and formatting markup). Not synced with playback.
• So I need a solution which writes whatever timecoded text (SRT, WEBVTT, LYR, etc) into the LYRICS metadata of the file (in the proper formatted way). And that's what the proposal is about.

see this here - https://cweiske.de/tagebuch/embedded-lyrics.htm

[porg]

@october262 thanks for your shared link which is a good writeup of all kind of tools which can edit lyrics and synchronized lyrics (SYLT in ID3) to audio files.

@ALL

Conducted experiments & research

• Converted WEBVTT to LYR
  • This an online converter which is not perfect because it does not create pauses, so not suited for "production use"
  • But for quick testing purposes it was good enough.
• With kid3 3.9.2 on macOS 11 I was able to embedd the .LYR file into the SYLT tag of an MP3 file
• As it turns out Apple Music (I tested 1.1.6.37) does not support synchronized lyrics coming from the ID3 SYLT tag.
  • Apple's support article points out that synchronized lyrics only works for tracks coming from their Apple Music subscription service:

    If the Lyrics button is grayed out

    If nothing happens when you tap Lyrics the Lyrics button, then either you’re playing a song that's not from the Apple Music catalog, or lyrics aren’t available for that song.

• Bad news for my personal playback purposes (I use Apple Music also for self ripped MP3s)
• But technically the SYLT are there (inspected in Hex Editor).

Good news for the issue at hand: Almost all functionality found. "Just" a matter of integration

• kid3 is open source, so you could possible extract the functionality for LYR→SYLT embedding.
• or alternatively yt-lp could support --embed-subs to audio only file with the dependency kid3-cli which works with kid3-cli -c "set SYLT='your-song.lrc' ''" your-song.mp3
• All that is remaining is to add a WEBVTT → LRC converter.
  • WEBVTT: FROM timecode + TO timecode + text
  • LRC: FROM timecode + text
  • To achieve pauses the TO simply becomes the next FROM timestamp with a blank line of text
  • Either there are ready made ones, or the yt-dlp team writes it themselves, it's just some timecode conversion, plus inserting pauses.

[pukkandan]

Converted WEBVTT to LYR

All that is remaining is to add a WEBVTT → LRC converter

We already support conversion to lrc --convert-subs lrc

kid3 is open source, so you could possible extract the functionality for LYR→SYLT embedding.

Low level media manipulation is complex, and beyond the scope of yt-dlp to natively implement - which is why we use ffmpeg etc

or alternatively yt-lp could support --embed-subs to audio only file with the dependency kid3-cli which works with kid3-cli -c "set SYLT='your-song.lrc' ''" your-song.mp3

There is no reason to add another dependency when mutagen can do this just fine (see link I posted above)

As it turns out Apple Music (I tested 1.1.6.37) does not support synchronized lyrics coming from the ID3 SYLT tag.
Bad news for my personal playback purposes (I use Apple Music also for self ripped MP3s)

While the feature request is perfectly valid, if it ends up being useless for you, I suggest closing the issue. If someone else needs the feature, they will ask.

[porg]

Knowing feasibility better now, thanks to your research and sharing, I got a balanced proposal now:

• --convert-subs supports a new additional format simply called txt
  • txt is just "plain text" with all timecode stuff stripped.
• --convert-subs [lrc, txt] can be combined with --embed-subs and --extract-audio
  • Choosing txt converts to plain text with all timecode infos stripped and because it's just plain text embeds that into the normal lyrics tag (no playback sync)
  • Choosing lrc converts to timecoded lyrics and embeds into the synced timecode tag of the audio file (SYLT).
    • You say that mutagen already supports that.
    • Glad to read that you have something readily accessible which you could use.
    • Increases the chances for implementation! Fine!
  • Choosing --convert-subs lrc,txt embeds both lyrics types (plain, synced) into the respective tags of the audio file.

Would you do that?

• To have lyrics at all is certainly an improvement!
  • If you recall some of the lyrics but not the title or artist a full text search finds your song.
  • And I can look at the lyrics. At least with manual scrolling
• And those users use a music app with SYLT playback support (I may consider to switch to one) get synced lyrics.
• And as a user you make use of data which is mostly there at the source anyhow, with no extra effort, other than requesting it with the needed arguments. Would be a big win!

[Grub4K]

The issue is open and triaged. While no specific implementation has been discussed, anybody is free to pick this up and make it work; at no point has this issue been invalidated. Nobody has done that yet though, its a lot of work.

Alternatively someone can create a postprocessor plugin for it, using the cli tool mentioned. This should be less work and still work for what you want.

[fjueic]

Is anyone working on it?
If yes, please confirm.
else I would like to work on it.

[pukkandan]

Nobody is working on this as far as we know. Feel free to do so. As for implementation, you will need to make a new code branch for ext=mp3 and sub.ext=lrc in FFmpegEmbedSubtitlePP. This may help on how to do the actual embedding