[Feature Request]: Optionally downloading #Shorts and livestreams, and giving them their own page

[pairofcrocs]

Already implemented?

• I have read through the wiki.
• I understand the scope of this project and am aware of the known limitations and my idea is not already on the roadmap.

Your Feature Request

Adding the ability to select if shorts are downloaded or not. Also, moving the shorts out of the channel page and into their own shorts page.

Your help is needed!

• Yes I can help with this feature request!

[bbilly1]

Thanks for collecting that here. Some additional findings: Shorts can be extracted from the /shorts channel page, e.g. https://www.youtube.com/channel/UC4JX40jDee_tINbkjycV4Sg/shorts will return all shorts, so we can add this to ES and filter from there.

[PhuriousGeorge]

Also need to get #livestreams added, linked as streams.

Can be extracted from /streams channel page, e.g. https://www.youtube.com/channel/UC4JX40jDee_tINbkjycV4Sg/streams will return all Livestreams

Personally I find this a high-level... not issue, but broken feature due to YTs changes.

[bbilly1]

Thanks for adding that. In the meantime, as these videos have the same url, you can also add the video directly to the download queue.

[simpleman19]

I'm interested in helping to do the work to add this feature as a few of the channels that I subscribe to almost exclusively do live content which was broken in the youtube change of moving to a separate page.

I have been playing around a little bit in the codebase and have a more or less working solution but want some guidance on the best way to handle this since I am new to this codebase.

Initial question is should there be a ta_live and ta_shorts to keep lives and shorts separate from all the normal videos or just add a flag for live and short to the ta_video?

Also, at first glance it doesn't look like there are any flags coming from the youtube api to indicate short except for them coming from the "shorts" page. I think the process is to check for a '#short' in the title or description as well as ensure <= 60 seconds in duration.

[pairofcrocs]

Thanks for reaching out!

We're thinking of doing a Discord live stream soon. kind of going over the codebase to help new devs get comfortable.

If you're interested, you can see it there when we go live, otherwise, we'll be recording it too.

@bbilly1 will be able to answer your initial questions before that though.

[simpleman19]

@pairofcrocs Awesome, I will definitely tune in. I usually don't do much with open source projects, mainly stick with my contracting work but I would really like to see this implemented so I don't mind spending a few hours knocking it out.

The codebase is pretty straightforward, I didn't have a whole lot of problems finding everything and getting a POC implemented in a couple hours just to get familiar and play with everything.

[bbilly1]

@simpleman19 thanks for reaching out, I'm very glad for your help.

You might be further along with a solution here, so take my input with a grain of salt, obviously, I haven't thought it all through.

should there be a ta_live and ta_shorts to keep lives and shorts separate from all the normal videos or just add a flag for live and short to the ta_video?

I think it would make sense to have it all in the same index, ta_video but just have a key there indicating where this video belongs, like video_type that could be a keyword of regular, shorts or stream, or any future additions by YouTube. As shorts were downloading, indexing and playing fine before the change on YouTube, there shouldn't be any other changes needed. Then adapt the requests to filter by that keyword to get the correct results.

Also, at first glance it doesn't look like there are any flags coming from the youtube api to indicate short except for them coming from the "shorts" page.

I was hoping by getting the /channel/<id>/streams page we'd be able to identify all streams video IDs, same for /channel/<id>/shorts, so we shouldn't have to do some regex matching (I failed with that before...). I was thinking to add that to the downloads postprocessing tasks, to match new videos to the correct video_type.

Also see #385 with some additional input, particularly the download form parsing needs fixing...

Ideally we should be able to configure this on a per channel basis, maybe you saw, I implemented some channel overwrite logic for a few other things. It has come up a few times before, it would be nice an easy way to skip shorts or live streams.

So that's the outline :-). Feel free to tackle any or all of that, I'd be more than happy to assist, usually on Discord.

[bbilly1]

@simpleman19 were you able to make any progress on this one? Please reach on on Discord so we can coordinate to avoid making duplicate work.

[simpleman19]

Apologies for the delayed response, last week was a super busy week for me. I will grab the discord link and reach out so we can coordinate further, I should have some time this week to burn on this.

[simpleman19]

@bbilly1 Just opened a PR #395

[bbilly1]

This is now merged in v0.3.1. Thanks for your help with that!