Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] How are videos uploaded during a channel playlist download handled? #56

Closed
3 tasks done
undaunt opened this issue Feb 7, 2021 · 6 comments
Closed
3 tasks done
Labels
question Question

Comments

@undaunt
Copy link

undaunt commented Feb 7, 2021

Checklist

  • I'm asking a question
  • I've looked through the README and FAQ for similar questions
  • I've searched the bugtracker for similar questions including closed ones

Question

Just wondering what the behavior is if a channel uploads a video midway through my download process. Eg: I'm downloading all videos from a channel via playlist, then running a second job to grab all remaining non-playlist videos. If a channel uploads a new video to a playlist during the playlist snatching, will it get discovered if that playlist hasn't already been scanned?

What I'm trying to determine is, if my batch file has a lot of content to crawl through, it's possible a video that belongs in a playlist could get missed during a long first run and then inadvertently added to the archive.log during the second, channel based job? I'm going to write a bash script to iterate through both as one single cron job, wondering if to cover my bases I need to trigger the playlist job a second time during the initial load of all backdated content so I don't miss anything?

If a video is added during the initial channel download, I imagine it will just be skipped as I'd be running that job second and the pages of all existing videos on the channel are scanned at the start on /videos based jobs, right?

Thanks!

@undaunt undaunt added the question Question label Feb 7, 2021
@pukkandan
Copy link
Member

pukkandan commented Feb 7, 2021

it's possible a video that belongs in a playlist could get missed during a long first run and then inadvertently added to the archive.log during the second, channel based job

Yes. Here's a trick to handle this:

(My example code is for windows, but you can easily adapt it for unix)

:start
set has_new_dl=0
youtube-dl URL --download-archive archivefile --exec "set has_new_dl=1 && echo.Downloaded: {}"`
if ["%has_new_dl%"]==["1"] goto start

This will loop until youtube-dl runs once without downloading anything

@pukkandan
Copy link
Member

Another trick would be to use youtube-dl URL --get-id --download-archive archivefile to check if there are any un-downloaded videos. But this will return any deleted/blocked videos as well

@undaunt
Copy link
Author

undaunt commented Feb 7, 2021

Thanks, let me work on this today before I close things out but super helpful!

@undaunt
Copy link
Author

undaunt commented Feb 8, 2021

@undaunt undaunt closed this as completed Feb 8, 2021
@pukkandan
Copy link
Member

pukkandan commented Feb 8, 2021

I see you are using a file rather than a variable, but that's fine.

Have you tried running it though? I suspect it wont work. Since the exec command has no {}, youtube-dl tries to add one at the end. So the command is treated as --exec "touch $MEDIADIR/youtube/new.downloads {}". This is done so that something like --exec post.sh will get the filename as the argument.

You should probably use {} > $MEDIADIR/youtube/new.downloads instead of touch

Edit: Upon testing, it seems touch takes multiple files as args. It is touching both new.downloads as well as the downloaded file. So it will work, no problem

@undaunt
Copy link
Author

undaunt commented Feb 8, 2021

I had issues setting the variable between the various bash shells. Touching a file and then checking for its existence cleared that up.

Thanks for testing, I just woke up and was going to re-test to confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Question
Projects
None yet
Development

No branches or pull requests

2 participants