[Feature Request]: RSS feed downloader

[pairofcrocs]

Already implemented?

• I have read through the wiki.
• I understand the scope of this project and am aware of the known limitations and my idea is not already on the roadmap.

Your Feature Request

Is your feature request related to a problem? Please describe.

Sometimes channels delete videos shortly after uploading and scheduling a refresh of channels every hour isn’t great from a rate limit standpoint.

Describe the solution you'd like

Using an RSS feed to get notified of new uploads could trigger TA to search for only channels that have new videos. I found this article talking about how to get an RSS feed from YouTube: https://danielmiessler.com/blog/rss-feed-youtube-channel/

Additional context

I’m assuming this would require significant restructuring of the scheduler/download function, however, with this method it should “speed” up TAs downloading and refreshing because it would only look at new videos on channels, without having to go through everything.

Your help is needed!

• Yes I can help with this feature request!

[bbilly1]

So with that, you still need to constantly poll youtube.com, one request per interval for every channel. Instead of using yt-dlp, you'd be making regular requests to the feed. You'd still face the same problems and probably rate limiting issues. The only new videos part yes, but with a reasonable channel page size of 50, that's still one request only.

What you really need to solve that problem is to start reacting to push events, and not constantly polling the same URLs. Something like described here: https://developers.google.com/youtube/v3/guides/push_notifications.

But yeah, to implement that, you'd either need to expose your TA to the internet, publicly, to be able to accept these POST requests, or you'd need your own public endpoint to receive these post requests from youtube, then you'd need some way to store these links, then a way to get them into your TA queue.

Either still by polling that endpoint every few minutes, might not be unrealistic, as that would be only one request per interval and the database layout could be structured and indexed to make these requests as efficiently as possible. Or even better, websockets, so you could subscribe to that and react to events in realtime.

Is this where we make a pro version, and implement that on tubearchivist.com?

[bbilly1]

This idea is now realized with https://members.tubearchivist.com/.