Persistence: Settings Panel for user-prefered youtube-dl parameters (always apply) and archive.txt

[GlassedSilver]

Ideally something beyond a copy and paste text box that let's you dump your settings string in there, because youtube-dl - as mighty as it is - has TONS of parameters and digesting them can take - I kid you not - dozens of minutes.

I think it'd be nice to have both, a text field to add parameters manually and then you append (and in case of conflict overwrite this) with UI elements like check boxes and the like.

Having proper "settings" as user expects to stay in tact during a docker container update or a reinstall would necessitate having an appdata config diretory that needs to be bound to a host path.

Also: we should store an archive.txt there or in a separate folder. When the user moves out downloaded items we want to make sure that the items don't get downloaded again through an eventual feature like playlist/channel subscriptions/manual downloads. Why a separate location for the archive.txt? Well, it would make it easily sharable with other instanced of youtube-dl regardless of frontend and device. (think youtube-dl on your laptop storing the archive in a dropbox.)

[Tzahi12345]

Great ideas. I do think there should be a better system for custom args. It will, like you suggested, require adding a settings panel. I've already been wanting to allow the majority of the config to be changed from the UI, so I can include "global preferred parameters/args" as another config option.

There's a few challenges for this feature in particular, in that figuring out which parameters conflict with which will be pretty much impossible (there are so many). So I think I will leave it up to user control, but at the very least show them the predicted complete args list (default ones + custom args + global custom args) so they can at least see what is happening.

Still, it's a great idea, and I'll put it in the pipeline.

Also: we should store an archive.txt there or in a separate folder. When the user moves out downloaded items we want to make sure that the items don't get downloaded again through an eventual feature like playlist/channel subscriptions/manual downloads.

Makes sense. I was considering not using an archive.txt it because of this one edge case: what if a user is "subscribed" to a channel (meaning the channel's video auto download), and they accidentally delete the channel's video folder. When the server goes to check for new videos, the archive.txt will still have all the now-deleted videos, meaning the deleted videos won't get re-downloaded.

Not sure how likely that scenario is. Hmmm..

[GlassedSilver]

Makes sense. I was considering not using an archive.txt it because of this one edge case: what if a user is "subscribed" to a channel (meaning the channel's video auto download), and they accidentally delete the channel's video folder. When the server goes to check for new videos, the archive.txt will still have all the now-deleted videos, meaning the deleted videos won't get re-downloaded.

Not sure how likely that scenario is. Hmmm..

An archive.txt allows me to put videos where I want them and have them externally processed, e.g. fed into Plex with metadata, but also not be tied to Plex, in case I switch solutions there.

It also allows me to use more than one application for youtube-dl tasks, because let's be honest, when you're out and about and prefer your home server not to be accessible from the internet, you do run into iffy workflows.

I get your train of thought though, so here's how I'd solve it: archive.txt could be optional, so for users who want to have your application run as both downloader and viewer or at least manager, they might want to skip using it. Everyone else will love the ability to use archive.txt.

Alternatively you can offer to delete archive.txt entries for their respective downloads if a user deletes them through the application. :)

[Tzahi12345]

I get your train of thought though, so here's how I'd solve it: archive.txt could be optional, so for users who want to have your application run as both downloader and viewer or at least manager, they might want to skip using it. Everyone else will love the ability to use archive.txt.

I like this solution, especially because it gives users the ability to choose which system they prefer. Maybe a use_youtubedl_archive config option for subscriptions, with an easy export or delete button for the archive when you navigate to that subscription's page (I'm planning for this page to contain info on the subscription (channel or playlist), its videos, and potential settings if I get to it.

Alternatively you can offer to delete archive.txt entries for their respective downloads if a user deletes them through the application. :)

Was thinking about this too this morning. It does introduce another set of complexities: what if someone wants to delete a video and not re-download? (then the archive.txt should still contain the video's ID). Maybe if you have archive.txt enabled, deleting should present two options: delete and re-download (which will remove the line), or delete forever (which will not touch archive.txt).

[GlassedSilver]

I like this solution, especially because it gives users the ability to choose which system they prefer. Maybe a use_youtubedl_archive config option for subscriptions, with an easy export or delete button for the archive when you navigate to that subscription's page (I'm planning for this page to contain info on the subscription (channel or playlist), its videos, and potential settings if I get to it.

Please do not limit this to automatic downloads. I cannot remember all the downloads of videos I start manually.

Was thinking about this too this morning. It does introduce another set of complexities: what if someone wants to delete a video and not re-download? (then the archive.txt should still contain the video's ID). Maybe if you have archive.txt enabled, deleting should present two options: delete and re-download (which will remove the line), or delete forever (which will not touch archive.txt).

Hmm... Maybe we should generate a proxy archive.txt that is being updated from two separate archive.txt's.

One's for actually downloaded videos and the other for "blacklisted" videos. That blacklist could also be fed from other options eventually, maybe even externally. Think of stuff like blacklisted words or something... Idk, it does however open the door to lots of creativity, even plug-in-style.

We could re-use that code to also read from another read-only archive.txt that may be written to from another machine like a laptop on the go and which's archive.txt is synched with Dropbox. (for privacy reasons I don't want Dropbox to know ALL of my downloads...)

Multiple files, because archive.txt stays a good log of things you not only don't want to download anymore, but rather of videos you do HAVE.

So you also know that, say your drive dies with the videos on it, you'll know exactly what you lost and can easily try to recover, at least the ones that are still online and all the ones you yeeted don't get added back. :)

I know, backups, but especially with big media libraries of content that can usually be obtained again you may lower your safeguards down to something like unRAID parity without second copies on other media... Economics of backing up dozens of terabytes are... interesting. :D

Sorry for rambling I guess, but I'm very passionate about giving youtube-dl the proper GUI with a server instance it deserves! <3

[Tzahi12345]

Please do not limit this to automatic downloads. I cannot remember all the downloads of videos I start manually.

Totally understand. I couldn't get this in this latest update, meaning archive functionality is only for subscriptions. I'll leave this issue open until that gets implemented, perhaps in the next update.

Hmm... Maybe we should generate a proxy archive.txt that is being updated from two separate archive.txt's.

Hmm, yeah I like this better than the current implementation (though it does slightly complicate things). I'll see if I can get something like this working for subscriptions v2. As it stands, deleted (aka blacklisted) videos are still kept in the archive.txt.

Sorry for rambling I guess, but I'm very passionate about giving youtube-dl the proper GUI with a server instance it deserves! <3

I do appreciate the thoughts, it helps me both understand what advanced users want, and how to best implement or improve those features.

Anyways, check out the latest update, 3.4! It includes playlist subscriptions, settings panel in the UI, global custom args for youtube-dl, and the basic archive functionality mentioned above. Let me know if you have any more ideas, comments, or questions, I'm always open to them :)

[GlassedSilver]

Will do right away. You have no idea how much I love you for all this work!
Any new path mapping necessary? (ytdl_subscriptions_base_path aside) Where are the configs stored?

[Tzahi12345]

Will do right away. You have no idea how much I love you for all this work!

<3

Any new path mapping necessary?

Nope! Just ytdl_subscriptions_base_path.

Where are the configs stored?

Primary config is the config/default.json file. There is also a database file db.json which stores additional information (settings for individual playlists or subscription).

[GlassedSilver]

Where are the configs stored?

Primary config is the config/default.json file. There is also a database file db.json which stores additional information (settings for individual playlists or subscription).

Doesn't that mean that removing and then re-adding the container will get rid of these appdata settings? That is unexpected behavior afaik. Anything a user can configure/produce should be able to survive a wipe of the docker image.

I'd HEAVILY suggest we instate a persistent appdata folder that gets mapped to a host directory. Mind you, I'm coming from an unRAID background, maybe this is different in other implementations of docker, but I doubt that. Either way, I will keep this in mind and include the config folder in the unRAID CA container's template as a host-mapped directory.

At this point I'm just waiting for archive.txt for manual downloads and maybe a few more youtubedl-args as user-friendly options in the settings panel. Like the file name and subfolder logic. :)

Then I think it's ready for a beta release to the CA's of unRAID.

[Tzahi12345]

Doesn't that mean that removing and then re-adding the container will get rid of these appdata settings? That is unexpected behavior afaik. Anything a user can configure/produce should be able to survive a wipe of the docker image.

Good point. It won't be too hard to include an extra folder with the docker-compose.yml, but it will require users to do an extra step before they can run docker-compose up, which is to unzip the docker compose zip file (with the appdata folder included) instead of just the yml. I think it's worth the tradeoff.

Another issue is the config and database folders are in different directories. Gotta think about how to link or copy those files from the single appdata folder into their proper places. Probably a job for the backend lol

At this point I'm just waiting for archive.txt for manual downloads and maybe a few more youtubedl-args as user-friendly options in the settings panel. Like the file name and subfolder logic. :)

It is possible to output your files into specific directories from the main downloader (just enable advanced download mode). It persists over page reloads as well, and supports youtube-dl's output template, and subfolders will get automatically created if you use a string like %(channel)s/%(title)s.

Let me know if you have any ideas on this front! It isn't the most intuitive system as it stands.

Then I think it's ready for a beta release to the CA's of unRAID.

Cool!

[GlassedSilver]

Good point. It won't be too hard to include an extra folder with the docker-compose.yml, but it will require users to do an extra step before they can run docker-compose up, which is to unzip the docker compose zip file (with the appdata folder included) instead of just the yml. I think it's worth the tradeoff.
Hmm.... The docker-compose.yml could write the template and the application itself when it sees no saved config is found will fill a saved config from the template.

That way preferenced passed by docker-compose should be honored on first setup, but any further configuration by the user in the application itself won't be overwritten by container updates and complete re-installations.

Another issue is the config and database folders are in different directories. Gotta think about how to link or copy those files from the single appdata folder into their proper places. Probably a job for the backend lol

I haven't checked your codebase yet so I'll take the backseat here for now, however I am sure you can work something out. I'll keep thinking about it and if I come up with something I'll let you know.

Is your main worry that changing them to one united directory would break existing installs? That's kinda the kind of the reason why I'm not publishing to CA's yet, the rough groundwork should happen now, before even a beta submission, although I definitely think your application looks the most promising of all the youtube-dl wrappers I know and there are a few. :D

Mostly it's that I like your DIRECTION and vision. I think we think alike in many ways and that's somewhat of a nice change in my youtube-dl needs.

It is possible to output your files into specific directories from the main downloader (just enable advanced download mode). It persists over page reloads as well, and supports youtube-dl's output template, and subfolders will get automatically created if you use a string like %(channel)s/%(title)s.

Let me know if you have any ideas on this front! It isn't the most intuitive system as it stands.

Kinda thinking of having this as a separate setting, maybe with movable tokens displayed as graphical elements. This would make this probably very commonly needed function a lot more approachable. Having full youtube-dl args support is pure gold and VERY good. It cannot be underrated and basically already gives the user almost all the control they need, but to have custom settings for the most common args would help make this GUI a lot more helpful.

In an ideal world we could gradually pass all youtube-dl args as checkboxed/list-viewed/radio-boxed/... settings from the UI, until then however having covered at least the most needed features would go a LONG way. (before anyone slaps me: I KNOW how many args there... navigating that section of the youtube-dl docs is however jarring to say the least. With proper categorization and nesting of the settings and guidance we could make those args more approachable to power users who don't like reading for hours. I kid you not, I read the docs for hours to find my settings and later on found out I still missed a few that I liked to know about.)

We should probably create an issue to track which args are on the radar for those that would be most urgent?

I could offer help with lay-outing such a settings panel. (PSD-file-ish)
I think for now I'd be more of your UI/support/distribution guy. :D

Not necessarily excluding interest in diving into the code some day though. I love what you're doing!

[Tzahi12345]

That way preferenced passed by docker-compose should be honored on first setup, but any further configuration by the user in the application itself won't be overwritten by container updates and complete re-installations.

I was actually thinking of removing the docker-compose environment variables and just relying on the appdata config to define the settings, with a volume already setup in the docker-compose to link the appdata folder with the config folder. I think it'll simplify things a bit.

Is your main worry that changing them to one united directory would break existing installs?

That's basically the crux of my hesitation. After giving it more thought, a unified data/settings directory is probably the best way to go forward. I'll do some tinkering around, but I think the easiest thing to do is move the db.json into the config folder (currently sits at the root) and set up that appdata volume to map it to the config folder.

Kinda thinking of having this as a separate setting, maybe with movable tokens displayed as graphical elements.

I could offer help with lay-outing such a settings panel. (PSD-file-ish)

I had the same idea, it'll make it more visual and interactive. Let me know if you make a mockup, it'll be helpful in figuring out how to organize the various "tokens" layout-wise and how to actually add those tokens from some menu. For the latter part (adding the tokens), there already exists nested menu support in the Material Design library I'm using (there's a demo at the link), but it may be too cluttered/dense for our purposes.

Anyways, let me know if you come up with some kind of layout for this. Should end up being quite a helpful feature, scrolling through youtube-dl's README I'm also always finding new stuff.

I think for now I'd be more of your UI/support/distribution guy. :D

Not necessarily excluding interest in diving into the code some day though. I love what you're doing!

Feel free to peek around the codebase, for the long-term I should have contribution guidelines where I can explain the codebase a bit more. Nonetheless, I'm happy with whatever help is available :)

[Tzahi12345]

I've released version 3.5 which includes system-wide archive.txt functionality and a first-take at an args modifier dialog. For now, it includes all available args, searchable either by text or category (I categorized them based on how youtube-dl did it in their repo), with their descriptions included.

There's several things I want to change in the future. A more user-friendly UI, along with the ability to move around or delete args as right now only adding is available in the dialog. Some args should also be removed as they aren't really relevant (like --help).

Let me know your thoughts on the dialog and what can be improved. Lots more included in the update so be sure to check it out!