[Question] How are videos uploaded during a channel playlist download handled?

[undaunt]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

Just wondering what the behavior is if a channel uploads a video midway through my download process. Eg: I'm downloading all videos from a channel via playlist, then running a second job to grab all remaining non-playlist videos. If a channel uploads a new video to a playlist during the playlist snatching, will it get discovered if that playlist hasn't already been scanned?

What I'm trying to determine is, if my batch file has a lot of content to crawl through, it's possible a video that belongs in a playlist could get missed during a long first run and then inadvertently added to the archive.log during the second, channel based job? I'm going to write a bash script to iterate through both as one single cron job, wondering if to cover my bases I need to trigger the playlist job a second time during the initial load of all backdated content so I don't miss anything?

If a video is added during the initial channel download, I imagine it will just be skipped as I'd be running that job second and the pages of all existing videos on the channel are scanned at the start on /videos based jobs, right?

Thanks!

[pukkandan]

it's possible a video that belongs in a playlist could get missed during a long first run and then inadvertently added to the archive.log during the second, channel based job

Yes. Here's a trick to handle this:

(My example code is for windows, but you can easily adapt it for unix)

:start
set has_new_dl=0
youtube-dl URL --download-archive archivefile --exec "set has_new_dl=1 && echo.Downloaded: {}"`
if ["%has_new_dl%"]==["1"] goto start

This will loop until youtube-dl runs once without downloading anything

[pukkandan]

Another trick would be to use youtube-dl URL --get-id --download-archive archivefile to check if there are any un-downloaded videos. But this will return any deleted/blocked videos as well

[undaunt]

Thanks, let me work on this today before I close things out but super helpful!

[undaunt]

Check out my main file https://github.com/undaunt/youtube-metadata-parse/blob/master/bin/yt-dlc-execute.sh

[pukkandan]

I see you are using a file rather than a variable, but that's fine.

Have you tried running it though? I suspect it wont work. Since the exec command has no {}, youtube-dl tries to add one at the end. So the command is treated as --exec "touch $MEDIADIR/youtube/new.downloads {}". This is done so that something like --exec post.sh will get the filename as the argument.

You should probably use {} > $MEDIADIR/youtube/new.downloads instead of touch

Edit: Upon testing, it seems touch takes multiple files as args. It is touching both new.downloads as well as the downloaded file. So it will work, no problem

[undaunt]

I had issues setting the variable between the various bash shells. Touching a file and then checking for its existence cleared that up.

Thanks for testing, I just woke up and was going to re-test to confirm.