Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--exclusion-archive and --error-archive ( the opposite of --download-archive ) #23296

Open
wayneworkman opened this issue Dec 3, 2019 · 0 comments
Labels

Comments

@wayneworkman
Copy link

@wayneworkman wayneworkman commented Dec 3, 2019

  • I'm reporting a feature request
  • I've verified that I'm running youtube-dl version 2019.11.28
  • I've searched the bugtracker for similar feature requests including closed ones

Description

I use youtube-dl to archive YouTube channels. I run the same commands over and over to get new videos. The command I run looks like this (bash variables left in there):

youtubedl --sleep-interval 60 -i --user-agent '${useragent}' --cookies /data/cookies.txt --download-archive ${directory}/archive.txt --write-info-json --write-description --write-thumbnail --socket-timeout 30 --restrict-filenames -o "${directory}/%(uploader_id)s_%(upload_date)s_%(title)s.%(ext)s" $channelUrl

There's certain videos that seem to never download, I don't know why. Maybe it's user error. Here are a few examples:

[download] Downloading video 92 of 3668
[youtube] zP3IHPlyCqs: Downloading webpage
[youtube] zP3IHPlyCqs: Downloading video info webpage
[info] Writing video description to: /data/youtube/Suspicious0bservers/Suspicious0bservers_20190915_Water_Emergency_Earth_Sun_Galaxy_Electric_Sheets_S0_News_Sep.15.2019.description
[info] Writing video description metadata as JSON to: /data/youtube/Suspicious0bservers/Suspicious0bservers_20190915_Water_Emergency_Earth_Sun_Galaxy_Electric_Sheets_S0_News_Sep.15.2019.info.json
[youtube] zP3IHPlyCqs: Downloading thumbnail ...
[youtube] zP3IHPlyCqs: Writing thumbnail to: /data/youtube/Suspicious0bservers/Suspicious0bservers_20190915_Water_Emergency_Earth_Sun_Galaxy_Electric_Sheets_S0_News_Sep.15.2019.jpg
[download] Sleeping 60 seconds...
[download] Resuming download at byte 161691

In my error log, I see things like this regularly:

ERROR: unable to download video data: HTTP Error 404: Not Found
ERROR: Did not get any data blocks

I would like to have an argument that allows writing the names of error files to a specified file.
--error-archive
Any files that fail to download should be written there. I would like for the format of this file to be the same as the --download-archive file format, and be acceptable by another new argument: --exclusion-archive

The way I see this possibly working is - as CRON runs youtube-dl on schedule, error'd videos get written to the error archive file. Because I would specify the exact same file for --error-archive and --exclusion-archive, the next time CRON runs, the error'd files would be excluded. This would allow me to not get flagged by YouTube for too many requests, and not waste YouTube API time, and I can figure out what's happening with these videos at a later date instead of youtube-dl trying over and over.

@wayneworkman wayneworkman added the request label Dec 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.