Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure dir existence for thumbnails and subtitles #7985

Merged
merged 1 commit into from Oct 13, 2023

Conversation

Riteo
Copy link
Contributor

@Riteo Riteo commented Aug 29, 2023

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Fixes output template pl_thumbnail.

Before this PR, trying to download a playlist with an output template of type pl_thumbnail pointing to a missing subdirectory doesn't create it, instead erroring out.

This problem went unnoticed while downloading videos because they're written into an already ensured dir, while playlists called the thumbnail writing method directly.

The subtitle change, while technically not required (playlists don't usually have subs) is done for consistency with all the other metadata writing methods. I can remove it if there's the need but it shouldn't hurt.

Also I'm not really sure of the expected order of conditions, like, whether I should've put the checks right at the beginning or somewhere else. Please tell me if you'd like them somewhere else, I can change them no problem.

Example of a fixed invocation (tested with commit 4b3a6ef):

yt-dlp -vU --no-config --flat-playlist --write-thumbnail -o 'pl_thumbnail:test/%(id)s.%(ext)s' https://youtube.com/playlist?list=PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir

verbose log before fix
[debug] Command-line config: ['-vU', '--no-config', '--flat-playlist', '--write-thumbnail', '-o', 'pl_thumbnail:test/%(id)s.%(ext)s', 'https://youtube.com/playlist?list=PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version stable@2023.07.06 [b532a3481] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 4b3a6ef1b
[debug] Python 3.11.4 (CPython x86_64 64bit) - Linux-6.4.10-x86_64-with-libc (OpenSSL 3.1.2 1 Aug 2023, libc)
[debug] exe versions: ffmpeg git-2023-07-24-5b11ee9 (setts), ffprobe git-2023-07-24-5b11ee9
[debug] Optional libraries: certifi-2022.12.07, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1864 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
yt-dlp is up to date (stable@2023.07.06)
[youtube:tab] Extracting URL: https://youtube.com/playlist?list=PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading webpage
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Redownloading playlist API JSON with unavailable videos
[download] Downloading playlist: ULTRAKILL
[info] Downloading playlist thumbnail 3 ...
[info] Writing playlist thumbnail 3 to: test/PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir.jpg
ERROR: [Errno 2] No such file or directory: 'test/PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir.jpg'
Traceback (most recent call last):
  File "/home/riteo/srg/yt-dlp/yt_dlp/YoutubeDL.py", line 1575, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riteo/srg/yt-dlp/yt_dlp/YoutubeDL.py", line 1731, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riteo/srg/yt-dlp/yt_dlp/YoutubeDL.py", line 1860, in process_ie_result
    return self.__process_playlist(ie_result, download)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/riteo/srg/yt-dlp/yt_dlp/YoutubeDL.py", line 1958, in __process_playlist
    self._write_thumbnails('playlist', ie_result, self.prepare_filename(ie_copy, 'pl_thumbnail'))
  File "/home/riteo/srg/yt-dlp/yt_dlp/YoutubeDL.py", line 4265, in _write_thumbnails
    with open(encodeFilename(thumb_filename), 'wb') as thumbf:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'test/PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir.jpg'
verbose log after fix
[debug] Command-line config: ['-vU', '--no-config', '--flat-playlist', '--write-thumbnail', '-o', 'pl_thumbnail:test/%(id)s.%(ext)s', 'https://youtube.com/playlist?list=PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: a675ef135
[debug] Python 3.11.4 (CPython x86_64 64bit) - Linux-6.4.10-x86_64-with-libc (OpenSSL 3.1.2 1 Aug 2023, libc)
[debug] exe versions: ffmpeg git-2023-07-24-5b11ee9 (setts), ffprobe git-2023-07-24-5b11ee9
[debug] Optional libraries: certifi-2022.12.07, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1839 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.03.04
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/latest/download/_update_spec
ERROR: You cannot update when running from source code; Use git to pull the latest changes
[youtube:tab] Extracting URL: https://youtube.com/playlist?list=PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading webpage
WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (1/3)...
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading webpage
WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (2/3)...
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading webpage
WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (3/3)...
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading webpage
WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [youtube:tab] Incomplete yt initial data received; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U. Giving up after 3 retries
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading API parameters API JSON
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Downloading API JSON
[youtube:tab] PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir: Redownloading playlist API JSON with unavailable videos
[download] Downloading playlist: ULTRAKILL
[info] Downloading playlist thumbnail 3 ...
[info] Writing playlist thumbnail 3 to: test/PLFG57rtsH8Jso3qfd5eO6egyBtW3nyvir.jpg
[youtube:tab] Playlist ULTRAKILL: Downloading 81 items of 81
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 81
[download] Downloading item 2 of 81
[download] Downloading item 3 of 81
[download] Downloading item 4 of 81
[download] Downloading item 5 of 81
[download] Downloading item 6 of 81
[download] Downloading item 7 of 81
[download] Downloading item 8 of 81
[download] Downloading item 9 of 81
[download] Downloading item 10 of 81
[download] Downloading item 11 of 81
[download] Downloading item 12 of 81
[download] Downloading item 13 of 81
[download] Downloading item 14 of 81
[download] Downloading item 15 of 81
[download] Downloading item 16 of 81
[download] Downloading item 17 of 81
[download] Downloading item 18 of 81
[download] Downloading item 19 of 81
[download] Downloading item 20 of 81
[download] Downloading item 21 of 81
[download] Downloading item 22 of 81
[download] Downloading item 23 of 81
[download] Downloading item 24 of 81
[download] Downloading item 25 of 81
[download] Downloading item 26 of 81
[download] Downloading item 27 of 81
[download] Downloading item 28 of 81
[download] Downloading item 29 of 81
[download] Downloading item 30 of 81
[download] Downloading item 31 of 81
[download] Downloading item 32 of 81
[download] Downloading item 33 of 81
[download] Downloading item 34 of 81
[download] Downloading item 35 of 81
[download] Downloading item 36 of 81
[download] Downloading item 37 of 81
[download] Downloading item 38 of 81
[download] Downloading item 39 of 81
[download] Downloading item 40 of 81
[download] Downloading item 41 of 81
[download] Downloading item 42 of 81
[download] Downloading item 43 of 81
[download] Downloading item 44 of 81
[download] Downloading item 45 of 81
[download] Downloading item 46 of 81
[download] Downloading item 47 of 81
[download] Downloading item 48 of 81
[download] Downloading item 49 of 81
[download] Downloading item 50 of 81
[download] Downloading item 51 of 81
[download] Downloading item 52 of 81
[download] Downloading item 53 of 81
[download] Downloading item 54 of 81
[download] Downloading item 55 of 81
[download] Downloading item 56 of 81
[download] Downloading item 57 of 81
[download] Downloading item 58 of 81
[download] Downloading item 59 of 81
[download] Downloading item 60 of 81
[download] Downloading item 61 of 81
[download] Downloading item 62 of 81
[download] Downloading item 63 of 81
[download] Downloading item 64 of 81
[download] Downloading item 65 of 81
[download] Downloading item 66 of 81
[download] Downloading item 67 of 81
[download] Downloading item 68 of 81
[download] Downloading item 69 of 81
[download] Downloading item 70 of 81
[download] Downloading item 71 of 81
[download] Downloading item 72 of 81
[download] Downloading item 73 of 81
[download] Downloading item 74 of 81
[download] Downloading item 75 of 81
[download] Downloading item 76 of 81
[download] Downloading item 77 of 81
[download] Downloading item 78 of 81
[download] Downloading item 79 of 81
[download] Downloading item 80 of 81
[download] Downloading item 81 of 81
[download] Finished downloading playlist: ULTRAKILL

Fixes... Nothing I could find with a couple of searches with various keywords.

Sorry, it's my first time here. Do you allow direct PRs? CONTRIBUTING.MD doesn't say much either. If that's not the case, I can open a simple issue right away.

I hope that making the fix directly is better (merged or not I still need some workaround for this issue in the meantime so I'm not wasting time for myself anyways).

Closes #8203

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Copilot Summary

馃 Generated by Copilot at a675ef1

Summary

馃悰馃摑馃毃

Add checks for output directory existence in _write_subtitles and _write_thumbnails. Update docstring of _write_thumbnails.

When writing subtitles or thumbs
You might face some errors or bums
If the dir does not exist
You'll get an exception twist
So this PR adds some checks and sums

Walkthrough

  • Add checks to ensure output directories exist before writing subtitle and thumbnail files (link, link)
    • Modify the docstring of _write_thumbnails to reflect this possibility (link)

@bashonly bashonly added the enhancement New feature or request label Aug 29, 2023
@bashonly bashonly added the core-triage triage requested from a core dev label Sep 16, 2023
@bashonly bashonly added bug Bug that is not site-specific regression Works in youtube-dl/older yt-dlp labels Sep 25, 2023
yt_dlp/YoutubeDL.py Outdated Show resolved Hide resolved
yt_dlp/YoutubeDL.py Outdated Show resolved Hide resolved
yt_dlp/YoutubeDL.py Outdated Show resolved Hide resolved
@bashonly bashonly added the pending-fixes PR has had changes requested label Sep 26, 2023
Fixes output template `pl_thumbnail`.

This problem went unnoticed while downloading videos because they're
written into an already ensured dir, while playlists called the
thumbnail writing method directly.
@Riteo
Copy link
Contributor Author

Riteo commented Oct 11, 2023

There, I've applied all suggestions. Sorry for the delay!

Copy link
Member

@bashonly bashonly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

In the future, please do not force push; all commits will be squashed upon merge anyways

@bashonly bashonly added pending-review PR needs a review and removed pending-fixes PR has had changes requested labels Oct 11, 2023
Copy link
Member

@coletdjnz coletdjnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine to me

@Grub4K Grub4K removed pending-review PR needs a review core-triage triage requested from a core dev labels Oct 13, 2023
@bashonly bashonly merged commit 2acd1d5 into yt-dlp:master Oct 13, 2023
16 checks passed
@bashonly bashonly removed their assignment Dec 11, 2023
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug that is not site-specific enhancement New feature or request regression Works in youtube-dl/older yt-dlp
Projects
None yet
4 participants