Add new post processor: MutagenMetadataPP #8918

llistochek · 2024-01-03T10:25:11Z

Description of your pull request and other information

This post processor was introduced in #8917. It's meant to write multi-value fields with proper separation and it does so by using mutagen.
I've tested it on all formats listed in supported_formats. The following script was used for testing:

#!/bin/bash

# Install all required tools on Debian-based systems:
# sudo apt install python3-mutagen flac vorbis-tools opus-tools atomicparsley

OUT_DIR=/tmp/yt-dlp-test

rm -rf $OUT_DIR
mkdir -p $OUT_DIR

while read format; do
    python3 -m yt_dlp \
        --quiet \
        --extract-audio \
        --audio-format $format \
        --embed-metadata \
        --output "$OUT_DIR/%(title)s.%(ext)s" \
        'https://www.youtube.com/watch?v=ZYVQAsQ6QmI'
done <<EOF
    vorbis
    mp3
    m4a
    flac
    opus
EOF

cd $OUT_DIR
echo '---OGG---'
vorbiscomment *.ogg
echo '---MP3---'
mutagen-inspect *.mp3
echo '---FLAC---'
metaflac --list *.flac
echo '---OPUS---'
opusinfo *.opus
echo '---M4A---'
AtomicParsley *.m4a -t

Output

---OGG---
language=eng
encoder=Lavc59.37.100 libvorbis
title=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
date=20150903
purl=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
synopsis=Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.
DESCRIPTION=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
album=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
artist=Ty Dolla $ign
artist=The Weeknd
artist=Wiz Khalifa
artist=DJ Mustard
---MP3---
-- Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix).mp3
- MPEG 1 layer 3, 134399 bps (VBR), 48000 Hz, 2 chn, 243.02 seconds (audio/mp3)
COMM=ID3v1 Comment=eng=https://www.youtube.com/watch
TALB=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
TDRC=20150903
TIT2=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
TPE1=Ty Dolla $ign / The Weeknd / Wiz Khalifa / DJ Mustard
TRCK=63
TSSE=Lavf59.27.100
TXXX=comment=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
TXXX=description=Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.
TXXX=purl=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
TXXX=synopsis=Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.

---FLAC---
METADATA block #0
  type: 0 (STREAMINFO)
  is last: false
  length: 34
  minimum blocksize: 4608 samples
  maximum blocksize: 4608 samples
  minimum framesize: 772 bytes
  maximum framesize: 24480 bytes
  sample_rate: 48000 Hz
  channels: 2
  bits-per-sample: 24
  total samples: 11663205
  MD5 signature: 98f202d23013239d93c193fc3ddeb224
METADATA block #1
  type: 4 (VORBIS_COMMENT)
  is last: false
  length: 1037
  vendor string: Lavf59.27.100
  comments: 11
    comment[0]: album=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
    comment[1]: title=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
    comment[2]: date=20150903
    comment[3]: purl=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
    comment[4]: synopsis=Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.
    comment[5]: DESCRIPTION=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
    comment[6]: encoder=Lavf59.27.100
    comment[7]: artist=Ty Dolla $ign
    comment[8]: artist=The Weeknd
    comment[9]: artist=Wiz Khalifa
    comment[10]: artist=DJ Mustard
METADATA block #2
  type: 1 (PADDING)
  is last: true
  length: 8165
---OPUS---
Processing file "Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix).opus"...

New logical stream (#1, serial: e2fd8683): type opus
Encoded with Lavf59.27.100
User comments section follows...
	language=eng
	encoder=Lavf59.27.100
	title=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
	date=20150903
	purl=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
	synopsis=Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.
	DESCRIPTION=https://www.youtube.com/watch?v=ZYVQAsQ6QmI
	album=Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
	artist=Ty Dolla $ign
	artist=The Weeknd
	artist=Wiz Khalifa
	artist=DJ Mustard
Opus stream 1:
	Pre-skip: 312
	Playback gain: 0 dB
	Channels: 2
	Original sample rate: 48000 Hz
	Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
	Page duration:   1000.0ms (max), 1000.0ms (avg), 1000.0ms (min)
	Total data length: 4208263 bytes (overhead: 0.898%)
	Playback length: 4m:02.993s
	Average bitrate: 138.5 kbit/s, w/o overhead: 137.3 kbit/s
Logical stream 1 ended
---M4A---
Atom "©nam" contains: Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
Atom "©ART" contains: Ty Dolla $ign
Atom "©ART" contains: The Weeknd
Atom "©ART" contains: Wiz Khalifa
Atom "©ART" contains: DJ Mustard
Atom "©alb" contains: Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)
Atom "©day" contains: 20150903
Atom "©too" contains: Lavf59.27.100
Atom "©cmt" contains: https://www.youtube.com/watch?v=ZYVQAsQ6QmI
Atom "desc" contains: Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.
Atom "ldes" contains: Provided to YouTube by Atlantic Records

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix) · Ty Dolla $ign · The Weeknd · Wiz Khalifa · DJ Mustard

Or Nah (feat. The Weeknd, Wiz Khalifa & DJ Mustard) (Remix)

℗ 2013 Atlantic Recording Corporation

Unknown: Adam Catania
Producer: DJ Mustard
Unknown: Jean-Marie Horvat
Producer: Mike Free
Violin: Peter Lee Johnson
Audio  Recording  Engineer: Ty Dolla $ign
Unknown: Ty Dolla $ign
Lead  Vocals: Ty Dolla $ign
Unknown: Vince Watson
Writer: DJ Mustard
Writer: Lemmie Crockem
Writer: Mike Free
Writer: Ty Dolla $ign
Writer: Wiz Khalifa

Auto-generated by YouTube.

Each file format works great in my music player (Lollypop), except for .m4a, which shows only the first artist. It seems like it's an issue with a format itself¹². QuickTime documentation does not specify proper way of setting multiple values for one field, so every piece of software does it's own thing.
I can think of this solutions for this problem:

Ignore MP4 in MutagenMetadataPP, let FFMpegMetadataPP separate fields with commas. Maybe add argument for specifying custom delimiters, such as + or ;.
Warn users about possible problems with .m4a, encourage converting to .flac. Both formats are lossless anyway

Template

Before submitting a pull request make sure you have:

At least skimmed through contributing guidelines including yt-dlp coding conventions
Searched the bugtracker for similar pull requests
Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Fix or improvement to an extractor (Make sure to add/update tests)
New extractor (Piracy websites will not be accepted)
Core bug fix/improvement
New feature (It is strongly recommended to open an issue first)

- Moved fix_deprecated_fields into _fill_common_fields

This reverts commit 2598790.

pukkandan · 2024-02-22T05:32:55Z

Note to self: After #8869, make this obey the compat option

pukkandan · 2024-02-22T05:35:19Z

Do we need to run ffmpeg and then mutagen? Or can we embed all the fields with mutagen in these formats?

llistochek · 2024-02-22T07:21:22Z

Yes, we can embed all of the fields via mutagen. I can implement this as we originally discussed in #3944. For some reason i thought that the solution with MutagenMetadataPP would be better, but it seems like it isn't. Sorry for the confusion.

pukkandan · 2024-02-22T07:25:20Z

If embedding everything with mutagen has no downsides, that's better. Then we don't have to run two pass. Ffmpeg can be used as fallback if (1) mutagen errors (2) unsupported format (3) --compat-option avoid-mutagen is passed. See EmbedThumbnail for similar implementation

llistochek and others added 13 commits January 3, 2024 08:35

[ie] Add new fields with proper support for multiple values

071326c

Apply suggestions from code review

698199b

Apply suggestions from code review

a691696

Update yt_dlp/extractor/common.py

d60ad19

Update yt_dlp/YoutubeDL.py

ac52bf0

Rename new fields

265e0f7

- Moved fix_deprecated_fields into _fill_common_fields

Revert unrelated changes

41c3dab

Revert unrelated change

c3fe956

Update README

c624659

Update README and fix IE documentation typo

dca6384

Revert MutagenMetadataPP

2598790

Bring MutagenMetadataPP back

bbc5ee2

This reverts commit 2598790.

Remove Musepack from supported formats

aa3498b

pukkandan added the enhancement New feature or request label Jan 4, 2024

Merge branch 'master' into mutagen-metadata

8b3127c

pukkandan mentioned this pull request Feb 22, 2024

Embed thumbnail using mutagen for mp3 files, fix lyrics issue #5635 #8869

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new post processor: MutagenMetadataPP #8918

Add new post processor: MutagenMetadataPP #8918

llistochek commented Jan 3, 2024 •

edited

pukkandan commented Feb 22, 2024

pukkandan commented Feb 22, 2024

llistochek commented Feb 22, 2024 •

edited

pukkandan commented Feb 22, 2024 •

edited

Add new post processor: MutagenMetadataPP #8918

Are you sure you want to change the base?

Add new post processor: MutagenMetadataPP #8918

Conversation

llistochek commented Jan 3, 2024 • edited

Description of your pull request and other information

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

What is the purpose of your pull request?

Footnotes

pukkandan commented Feb 22, 2024

pukkandan commented Feb 22, 2024

llistochek commented Feb 22, 2024 • edited

pukkandan commented Feb 22, 2024 • edited

llistochek commented Jan 3, 2024 •

edited

llistochek commented Feb 22, 2024 •

edited

pukkandan commented Feb 22, 2024 •

edited