Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[postprocessor/EmbedThumbnail,postprocessor/FFmpegMetadata] Fix potential error on attaching thumbnails and info json for mkv/mka #6647

Merged
merged 2 commits into from Mar 27, 2023
Merged

[postprocessor/EmbedThumbnail,postprocessor/FFmpegMetadata] Fix potential error on attaching thumbnails and info json for mkv/mka #6647

merged 2 commits into from Mar 27, 2023

Conversation

Lesmiscore
Copy link
Contributor

@Lesmiscore Lesmiscore commented Mar 27, 2023 •
edited by ghost

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

This PR fixes a potential error (therefore a hidden bug) that caused by yt-dlp has failed to pass ffmpeg a file:-prepended path
This bug remained undiscovered because of "insane" filename sanitization

This log explains the situation:

[debug] Command-line config: ['-f', '141', '--embed-metadata', '--embed-thumbnail', '--write-info-json', '--no-clean-infojson', '--throttled-rate', '100k', '--no-playlist', '--header', 'Accept-Language:ja', '--parse-metadata', '%(album,channel,uploader)s:%(meta_album)s', '--fixup', 'force', '--extractor-args', 'youtube:player_client=web_music,android_music,web;preferred_langs=ja;player_skip=webpage', 'vuwvOHt5R-k', '5GPs1Pn94yI', '--remux', 'mka', '-kv', '--force-over']
[debug] User config "/home/lesmi/.config/ytdl-patched/config": ['--cookies=$HOME/ck.txt', '--escape-long-names', '--extractor-args', 'youtube:preferred_langs=ja', '--remux', 'f4v>mkv', '--ffmpeg-loc', '/mnt/extra-storage/ffmpeg/ffmpeg-n5.1-latest-linux64-gpl-5.1/bin/ffmpeg', '-o', '%(title)s [%(id)s].%(ext)s', '--downloader-args', 'ffmpeg:-b:a 150k']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] ytdl-patched version stable@2023.03.17.1679013592 [044e98b18] (homebrew)
[debug]      from commit 044e98b18
[debug]      based on 216bcb66d
[debug] ** The command you are running is not yt-dlp.
[debug] ** Please make bug reports at https://github.com/ytdl-patched/ytdl-patched/issues/new/choose instead.
[debug] Lazy loading extractors is disabled
[debug] Python 3.10.10 (CPython x86_64 64bit) - Linux-5.17.0-051700-generic-x86_64-with-glibc2.35 (OpenSSL 1.1.1t  7 Feb 2023, glibc 2.35)
[debug] exe versions: ffmpeg n5.1.2-9-g807afa59cc-20221229 (setts), ffprobe n5.1.2-9-g807afa59cc-20221229, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, WebSocketsWrapper-None, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1823 extractors
WARNING: [youtube] Preferring "ja" translated fields. Note that some metadata extraction may fail or be incorrect.
[youtube] Extracting URL: vuwvOHt5R-k
[youtube] vuwvOHt5R-k: Downloading web music client config
[youtube] [debug] Fetching webpage from https://music.youtube.com
[youtube] vuwvOHt5R-k: Downloading player ace4d669
[youtube] [debug] Fetching webpage from https://www.youtube.com/s/player/ace4d669/player_ias.vflset/ja_JP/base.js
[debug] [youtube] Extracted SAPISID cookie
[youtube] vuwvOHt5R-k: Downloading web music player API JSON
[youtube] [debug] Fetching webpage from https://music.youtube.com/youtubei/v1/player
[youtube] vuwvOHt5R-k: Downloading android music player API JSON
[youtube] [debug] Fetching webpage from https://www.youtube.com/youtubei/v1/player
[youtube] vuwvOHt5R-k: Downloading web player API JSON
[youtube] [debug] Fetching webpage from https://www.youtube.com/youtubei/v1/player
[debug] [youtube] Extracting signature function js_ace4d669_108
[debug] Loading youtube-sigfuncs.js_ace4d669_108 from cache
[debug] Loading youtube-nsig.ace4d669 from cache
[debug] [youtube] Decrypted nsig Shs_liZNdewchnf3aw => AoKYrZTnl5Shyg
[debug] Loading youtube-nsig.ace4d669 from cache
[debug] [youtube] Decrypted nsig uyi_HphFj1GRdP40sn => -N4zqkk8yh2pEw
[debug] [youtube] Extracting signature function js_ace4d669_104
[debug] Loading youtube-sigfuncs.js_ace4d669_104 from cache
[youtube] vuwvOHt5R-k: Downloading initial data API JSON
[youtube] [debug] Fetching webpage from https://www.youtube.com/youtubei/v1/next
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Searching for '(?P<meta_album>.+)' in '%(album,channel,uploader)s'
[MetadataParser] Parsed meta_album from '%(album,channel,uploader)s': 'A Disney Spectacular'
[info] vuwvOHt5R-k: Downloading 1 format(s): 141
Deleting existing file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].webp
[info] Downloading video thumbnail 47 ...
[info] Writing video thumbnail 47 to: Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].webp
[info] Writing video metadata as JSON to: Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].info.json
Deleting existing file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka
Deleting existing file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].m4a
[debug] Invoking dashsegments downloader on "https://rr4---sn-nvoxu-ioqed.googlevideo.com/videoplayback?expire=1679904567&ei=1_ogZNSsHamwzN0PouOZoAc&ip=92.202.97.50&id=o-AK-Y3l7JskcIhWuZpx6zTzDBQpifpM7_FeiF2KZudPo3&itag=141&source=youtube&requiressl=yes&mh=sc&mm=31%2C29&mn=sn-nvoxu-ioqed%2Csn-oguelnsk&ms=au%2Crdu&mv=m&mvi=4&pcm2cms=yes&pl=20&ctier=A&pfa=5&gcr=jp&initcwndbps=2248750&hightc=yes&spc=99c5CcUL7Eg7as0oNKoi9XJyR6CrmOO6XlLLmqKuRQpH&vprv=1&mime=audio%2Fmp4&ns=H_2qEYJTyOX0Phna-Mixd_EM&gir=yes&clen=2394409&dur=74.293&lmt=1648578900872456&mt=1679882695&fvip=5&keepalive=yes&fexp=24007246&c=WEB_REMIX&txp=2318224&n=-N4zqkk8yh2pEw&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cpfa%2Cgcr%2Chightc%2Cspc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIuecBwyJrgp31GLpByKMxzbWZRD9yvJeekoyZAHUDFXAiEAr5JQQzDDBTMY4M3tJNlzvG1f4Ktv60fuwseZMgqrg24%3D&sig=AOq0QJ8wRAIgZLSv1PvjbeRj06Ta7SzWe3kZjcZsBUS5OIfpGvft6GoCIDZ-SSZiJB2Bk-l4r8PvdskWtaUOLwrKnQJ1fhS0lAId"
[dashsegments] Total fragments: 1
[download] Destination: Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].m4a

[download] 0.0% of ~2.28MiB at 1.66KiB/s ETA 23:31 (0 fragments of 1)
[download] 0.1% of ~2.28MiB at 4.96KiB/s ETA 07:50 (0 fragments of 1)
[download] 0.3% of ~2.28MiB at 11.56KiB/s ETA 03:21 (0 fragments of 1)
[download] 0.6% of ~2.28MiB at 24.73KiB/s ETA 01:33 (0 fragments of 1)
[download] 1.3% of ~2.28MiB at 43.45KiB/s ETA 53s (0 fragments of 1)  
[download] 2.7% of ~2.28MiB at 72.50KiB/s ETA 31s (0 fragments of 1)
[download] 5.4% of ~2.28MiB at 141.56KiB/s ETA 15s (0 fragments of 1)
[download] 10.9% of ~2.28MiB at 260.39KiB/s ETA 8s (0 fragments of 1)
[download] 21.9% of ~2.28MiB at 416.06KiB/s ETA 4s (0 fragments of 1)
[download] 43.7% of ~2.28MiB at 717.93KiB/s ETA 1s (0 fragments of 1)
[download] 87.5% of ~2.28MiB at 1.15MiB/s ETA 0s (0 fragments of 1)  
[download] 100.0% of ~2.28MiB at 1.12MiB/s ETA 0s (0 fragments of 1)
[download] 100.0% of ~2.28MiB at 1.10MiB/s ETA 0s (1 fragments of 1)
[download] 100% of 2.28MiB in 00:02 at 1.10MiB/s                    
[VideoRemuxer] Remuxing video from m4a to mka; Destination: Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka
[debug] ffmpeg command line: /mnt/extra-storage/ffmpeg/ffmpeg-n5.1-latest-linux64-gpl-5.1/bin/ffmpeg -y -loglevel repeat+info -i 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].m4a' -map 0 -dn -ignore_unknown -c copy -movflags +faststart 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka'
[debug] ffprobe command line: /mnt/extra-storage/ffmpeg/ffmpeg-n5.1-latest-linux64-gpl-5.1/bin/ffprobe -hide_banner -show_format -show_streams -print_format json 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka'
[Metadata] Adding metadata to "Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka"
[debug] ffmpeg command line: /mnt/extra-storage/ffmpeg/ffmpeg-n5.1-latest-linux64-gpl-5.1/bin/ffmpeg -y -loglevel repeat+info -i 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka' -map 0 -dn -ignore_unknown -c copy -write_id3v1 1 -metadata 'title=Dodd: Mickey Mouse March (Arr. K. Whitcomb)' -metadata date=20220331 -metadata 'description=Provided to YouTube by Universal Music Group

Dodd: Mickey Mouse March (Arr. K. Whitcomb) · Cincinnati Pops Orchestra · Erich Kunzel · Indiana University Singing Hoosiers · School for Creative Performing Arts Children'"'"'s Chorus

A Disney Spectacular

℗ 1989 Telarc International Corp., Distributed by Concord.

Released on: 1989-05-23

Producer, Recording  Producer: Robert Woods
Studio  Personnel, Recording  Engineer: Jack Renner
Studio  Personnel, Asst.  Recording  Engineer: Michael Bishop
Studio  Personnel, Asst.  Recording  Engineer, Editor: Thomas Knab
Studio  Personnel, Asst.  Recording  Engineer: John Renner
Studio  Personnel, Editor: Rosalind Ilett
Studio  Personnel, Editor: Elaine Martone
Composer, Author: Jimmie Dodd
Arranger, Work  Arranger: Ken Whitcomb

Auto-generated by YouTube.' -metadata 'synopsis=Provided to YouTube by Universal Music Group

Dodd: Mickey Mouse March (Arr. K. Whitcomb) · Cincinnati Pops Orchestra · Erich Kunzel · Indiana University Singing Hoosiers · School for Creative Performing Arts Children'"'"'s Chorus

A Disney Spectacular

℗ 1989 Telarc International Corp., Distributed by Concord.

Released on: 1989-05-23

Producer, Recording  Producer: Robert Woods
Studio  Personnel, Recording  Engineer: Jack Renner
Studio  Personnel, Asst.  Recording  Engineer: Michael Bishop
Studio  Personnel, Asst.  Recording  Engineer, Editor: Thomas Knab
Studio  Personnel, Asst.  Recording  Engineer: John Renner
Studio  Personnel, Editor: Rosalind Ilett
Studio  Personnel, Editor: Elaine Martone
Composer, Author: Jimmie Dodd
Arranger, Work  Arranger: Ken Whitcomb

Auto-generated by YouTube.' -metadata 'purl=https://www.youtube.com/watch?v=vuwvOHt5R-k' -metadata 'comment=https://www.youtube.com/watch?v=vuwvOHt5R-k' -metadata 'artist=Cincinnati Pops Orchestra, Erich Kunzel, Indiana University Singing Hoosiers, School for Creative Performing Arts Children'"'"'s Chorus' -metadata 'album=A Disney Spectacular' -attach 'Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].info.json' -metadata:s:1 mimetype=application/json -metadata:s:1 filename=info.json -movflags +faststart 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].temp.mka'
[debug] ffmpeg version n5.1.2-9-g807afa59cc-20221229 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.2.0 (crosstool-NG 1.25.0.90_cf9beb1)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --disable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --disable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20221229
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'file:Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].mka':
  Metadata:
    COMPATIBLE_BRANDS: iso6mp41
    MAJOR_BRAND     : dash
    MINOR_VERSION   : 0
    ENCODER         : Lavf59.27.100
  Duration: 00:01:14.35, start: 0.000000, bitrate: 258 kb/s
  Stream #0:0: Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : ISO Media file produced by Google Inc.
      VENDOR_ID       : [0][0][0][0]
      DURATION        : 00:01:14.350000000
Could not open attachment file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].info.json.

ERROR: Postprocessing: Could not open attachment file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].info.json.
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3515, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3697, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3679, in run_all_pps
    info = self.run_pp(pp, info)
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3657, in run_pp
    files_to_delete, infodict = pp.run(infodict)
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/postprocessor/common.py", line 29, in run
    ret = func(self, info, *args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/postprocessor/common.py", line 140, in wrapper
    return func(self, info)
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 782, in run
    self.run_ffmpeg_multiple_files(
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 351, in run_ffmpeg_multiple_files
    return self.real_run_ffmpeg(
  File "/home/linuxbrew/.linuxbrew/Cellar/ytdl-patched/2023.03.17.1679013592/libexec/lib/python3.10/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 413, in real_run_ffmpeg
    raise FFmpegPostProcessorError(stderr.strip().splitlines()[-1])
yt_dlp.postprocessor.ffmpeg.FFmpegPostProcessorError: Could not open attachment file Dodd: Mickey Mouse March (Arr. K. Whitcomb) [vuwvOHt5R-k].info.json.
Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

🤖 Generated by Copilot at 4c2fdf7

Summary

🐛🎞️🛠️

Fix filename escaping issues for ffmpeg postprocessors. Use _ffmpeg_filename_argument to escape thumbnail and info.json filenames in embedthumbnail.py and ffmpeg.py.

ffmpeg escapes
filenames with special chars
autumn bugfixing

Walkthrough

  • Escape thumbnail and info.json filenames for ffmpeg using _ffmpeg_filename_argument method (link, link) in embedthumbnail.py and ffmpeg.py

@Lesmiscore
[postprocessor/EmbedThumbnail,postprocessor/FFmpegMetadata] Fix poten…
639a658 
…tial error on attaching thumbnails and info json for mkv/mka
@pukkandan
Copy link
Member

Use self._ffmpeg_filename_argument

@pukkandan pukkandan added the bug Bug that is not site-specific label Mar 27, 2023
@Lesmiscore Lesmiscore merged commit 0f0875e into yt-dlp:master Mar 27, 2023
11 checks passed
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024
@Lesmiscore @aalsuwaidi
[postprocessor/EmbedThumbnail,postprocessor/FFmpegMetadata] Fix error…
bbdd974 
… on attaching thumbnails and info json for mkv/mka (yt-dlp#6647)

Authored by: Lesmiscore

Current yt-dlp code never hit this bug, but would hit once filename sanitization gets better
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Lesmiscore Lesmiscore

Labels
bug Bug that is not site-specific
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Lesmiscore @pukkandan