Add ability to use gallery-dl features on any site youtube-dl supports? #1680
Comments
|
At least the basics for this are done in 36ac219. You can tell gallery-dl to download anything with youtube-dl by prefixing it with The subcategory gets set to the Todo:
and there are definitely bugs ... edit: #878 is more or less the same as this issue |
The key would be used as the subcategory name so it would make sense for it to be lowercase. Although when people add the keys they might just copy exactly what's in the list of keys which wouldn't be lowercase. I can't get "raw-options" to work. Before changing the config at all I tried downloading a youtube video with "ytdl:" and it used all the options I had in "downloader.ytdl.raw-options". The options in "extractor.ytdl.raw-options" don't do anything at all even after removing "downloader.ytdl" to test it. I also tried moving the raw-options to "extractor.ytdl.Youtube" but that didn't work either. All of the directory and filename options I put in "extractor.ytdl.Youtube" worked fine. |
|
Adding "module" to a subcategory doesn't change the module used. Sometimes features work on one version of youtube-dl but not the other. |
|
Are you talking about the Like this? "extractor":
{
"ytdl":
{
"module": "youtube_dl"
}
} |
|
This makes the module "youtube_dl": {
"extractor": {
"ytdl": {
"module": "youtube_dl"
}
}
}This doesn't make the module for "Youtube" "yt_dlp": {
"extractor": {
"ytdl": {
"module": "youtube_dl",
"Youtube": {
"module": "yt_dlp"
}
}
}
} |
|
Huh, that kind of nested object doesn't make sense. Using dot property notation, this would be something like this: I think So what if you just try this? {
"extractor": {
"ytdl": {
"module": "yt_dlp"
}
}
}
|
|
I already know that works. I was trying to make it use a different module for Youtube specifically (as an example). youtube-dl isn't always the best for every site and yt-dlp isn't always the best for every site either. Edit: I thought it should have worked because other options like "filename", "base-directory" and "directory" can be set to different things in each subcategory to override the option specified in the base category. |
Yeah, but the Okay, so you want to switch dynamically between youtube-dl and yt-dlp, depending on the input? I see.. Don't know how to properly tackle this, honestly. I mean, how to know - in advance - which is more suitable of these two downloader modules for a given site? But you can do something like this as a workaround: Use gallery-dl with input files gallery-dl/gallery_dl/option.py Lines 99 to 104 in 36ac219 Like this: Because gallery-dl can parse those input files, and you can define options there: See what I mean? |
|
I could just use -o. I just brought it up as an idea because this extractor is still in development. I found a big bug but I want to test it a lot more before going into detail. |
requests' CookieJar class is not quite compatible with the standard http.cookiejar.CookieJar used by youtube_dl
|
Using a different youtube_dl module per subcategory should work with 9a849cd. It now
So it needs to imports up to 2 youtube_dl modules, the main one to find a subcategory value and the one to actually use. That means the main module must support the input URL, otherwise it'll just fall back to the generic extractor, ignore the URL, or even raise an error. Another issue is the fact that the downloader uses its own youtube_dl module/object independently from the one used by the extractor. I'll most likely just transfer the extractor |
The idea is good, but it wouldn't have worked with how I implemented it. |
|
Wait a sec... |
This got fixed with something you changed. |
|
Using yt-dlp as the module:
Using youtube-dl as the module:
"writeinfojson": true,
"writedescription": true,
"writethumbnail": true,
"merge_output_format": "mkv"only "writeinfojson" worked when placed in "extractor.ytdl.raw-options" ("writeinfojson" is redundant anyways)
No "raw-options" work in subcategories. config for "extractor": "ytdl": {
"enabled": false,
"format": "bestvideo+bestaudio/best",
"logging": true,
"generic": true,
"module": "youtube_dl",
"directory": "",
"raw-options": {
"writecomments": true,
"writeinfojson": true,
"writedescription": true,
"writethumbnail": true,
"merge_output_format": "mkv"
},
"Youtube": {
"base-directory": "/example/path/youtube.com/",
"directory": [
"{channel}_{channel_id}"
]
}
}config for "downloader": "downloader": {
"filesize-min": null,
"filesize-max": null,
"part": true,
"part-directory": null,
"mtime": true,
"rate": null,
"retries": 4,
"timeout": 30.0,
"verify": true,
"http": {
"adjust-extensions": true,
"headers": null
},
"ytdl": {
"format": null,
"forward-cookies": false,
"logging": true,
"module": "youtube_dl",
"outtmpl": null,
"raw-options": {
"writecomments": true,
"writeinfojson": true,
"writedescription": true,
"writethumbnail": true,
"merge_output_format": "mkv"
}
}
}Each time I tested something I made sure that the value for "module" in "downloader.ytdl" and "extractor.ytdl" were the same.
After writing this comment out I can't get this to work anymore either It's really important that "raw-options" works in subcategories because that is often different for each site. |
Apply 'extractor.ytdl.module' for every URL, not just the first.
yt_dlp supports multiple outtmpl settings for different file types and uses its 'outtmpl_dict' for that.
I somehow managed to remove the options parameter for the YoutubeDL constructor in 9a849cd without noticing ...
allows specifying downloader-specific options per subcategory but overwrites all downloader.ytdl settings
|
Apply 'extractor.ytdl.module' for every URL, not just the first.
yt_dlp supports multiple outtmpl settings for different file types and uses its 'outtmpl_dict' for that.
I somehow managed to remove the options parameter for the YoutubeDL constructor in 9a849cd without noticing ...
allows specifying downloader-specific options per subcategory but overwrites all downloader.ytdl settings
Which sites is yt-dlp not the best for? Isn't it a direct fork that only improves upon the main repo? |
|
Command line options would definitely be good. These options can't do everything that I'd want to use (many things are missing actually). Maybe instead of putting keys (like "writethumbnail") from that page I just linked into the config file you could use the argument names (like "write-thumbnail" or "--write-thumbnail") instead (you might know a better way to do this)? |
|
@mikf Are you okay with donations for priority? I could donate $100. Command line options would be really important for me. |
|
I must have somehow missed your previous comment, my apologies.
That's not really the purpose of the "Sponsor" button, and $100 would be far too much for me to accept in good conscience. Thank you, but I have to decline.
What exactly? I'd need to know what command-line options specifically to be able to take a look at how youtube-dl handles such an option internally, so I can maybe add a workaround.
That can be done, but it would just be via internal translation from By "youtube-dl command-line options" at the top I meant adding some extra command-line options to gallery-dl to (quickly) be able to override some often-used youtube-dl options, for example |
|
Just looked at the youtube-dl source code again to figure out how and where it processes command-line arguments and found quite a few that are cannot easily be done with They can be used, though, but it's a bit more complicated. To use "postprocessors": [
{"key": "FFmpegMetadata"},
{"key": "FFmpegEmbedSubtitle"}
],It is also possible to reuse youtube-dl's command-line option parser and load options from a youtube-dl config file it seems, although that would mean a lot of duplicated lines of code. And then there's the issue of making this also compatible with yt-dlp. I'll see what I can do. |
- adds 'config-file' and 'cmdline-args' options for both ytdl downloader and extractor - create 'ytdl' helper module, which combines YoutubeDL creation and option parsing. - most likely a buggy mess due to incompatibilities between the original youtube-dl and yt-dlp.
|
Thank you so much :) "ytdl": {
"cmdline-args": {
"write-info-json": true,
"write-description": true,
"write-thumbnail": true,
"merge-output-format": "mkv"
}
}and this: {
"ytdl": {
"Youtube": {
"cmdline-args": {
"write-info-json": true,
"write-description": true,
"write-thumbnail": true,
"merge-output-format": "mkv"
}
}
}
}If I try one of those I get this error: If I remove the "cmdline-args" section of the config it runs fine but without the options. |
|
Sorry for not giving a proper example or adding any docs on how to use these options.
"cmdline-args": "--write-info-json --write-description --write-thumbnail --merge-output-format mkv",
"cmdline-args": [
"--write-info-json",
"--write-description",
"--write-thumbnail",
"--merge-output-format", "mkv"
]
"config-file": "~/.config/youtube-dl/config"
The options in the |
|
Something I should probably point out is that if Without setting I'm just mentioning it so it's not a problem for other people. It would be nice to be able to have the progress bar on other extractors and not have to disable it completely. |
'speed' is not guaranteed to be defined or convertible to 'int'
|
It's crashing on private videos and copyright claimed videos (and possibly other types of unavailable videos) The errors are skipped automatically when not using gallery-dl. |
|
It seems postprocessors don't work with this extractor. I'm going to start using an exec postprocessor on all of the files I download with gallery-dl. Would this be possible? Thanks :) |
|
Huh? |
|
The extractor that this issue is about (ytdl). Any site I've tested with that extractor doesn't do the postprocessors set up in the config. The postprocessesors work on every other extractor just fine. |
|
Yeah, that is not a "normal" extractor, it just handles importing yt-dlp so that it can be called from within gallery-dl.. I don't even know what you are trying to do, to be honest.. The output options for yt-dlp are here: https://github.com/yt-dlp/yt-dlp#output-template |
|
Are you think of https://github.com/mikf/gallery-dl/blob/master/gallery_dl/downloader/ytdl.py rather than https://github.com/mikf/gallery-dl/blob/master/gallery_dl/extractor/ytdl.py ?
The output options are all handled by gallery-dl. Example (wip) config for single YouTube videos: "Youtube": {
"module": "yt_dlp",
"#format": "bestvideo+bestaudio/best",
"#rate": "100k",
"archive": "$ROOT/downloads/gallery-dl/archives/youtube.sqlite3",
"archive-prefix": "youtube.com, ",
"archive-format": "{channel_id}, {id}, {live_status}",
"cmdline-args": [
"#--write-info-json",
"--write-description",
"--write-annotations",
"--write-thumbnail",
"--embed-thumbnail",
"--add-metadata",
"--no-continue",
"--no-overwrites",
"--all-subs",
"--embed-subs",
"--verbose",
"--progress",
"#--no-progress",
"--ignore-errors",
"--check-formats",
"--sleep-requests",
"1",
"--sleep-interval",
"5",
"#--max-sleep-interval",
"#30",
"--merge-output-format",
"mkv",
"--external-downloader",
"aria2c",
"--external-downloader-args",
"-x 4 -k 1M",
"#--list-formats"
],
"base-directory": REDACTED,
"directory": [
"{channel} [{channel_id}]"
],
"filename": {
"is_live == True": "{channel} - {upload_date} - {title} [{id}] [live].{extension!l}",
"": "{channel} - {upload_date} - {title} [{id}].{extension!l}"
}
}Being able to run "exec" postprocessors on the files downloaded would be extremely useful. |
|
No, I didn't miss that, I was thinking of https://github.com/mikf/gallery-dl/blob/master/gallery_dl/extractor/ytdl.py But if you've read that even cursorily, you might have noticed that it's basically a wrapper around https://github.com/mikf/gallery-dl/blob/master/gallery_dl/ytdl.py (oversimplification, I know)
You could have said that in the first comment..
You know, if you want to download something from YouTube, simply using yt-dlp would probably be a lot easier.. |
I don't know how to read code
I very clearly did. That was the whole point of the comment. This conversation is pointless. Most of the comments I see you post on issues just distract from the issue. If you reply to this, I'm not responding since that would lead nowhere. |
|
@Twi-Hard what exactly are you trying to do? I just ran a simple test with "postprocessors": [
{
"name": "exec",
"command": ["echo", "{channel}"]
}
]and that, as expected, echoed the channel name after the download. |
|
I'm so dumb.. The postprocessors I tested are whitelisted to specific sites.. that's why it wasn't outputting anything. |
Could you make it possible to use gallery-dl's features on any youtube-dl download? There's been many times that the way gallery-dl handles naming from metadata would have been extremely useful.
You could maybe make youtube-dl a category in the config with each site being a subcategory so you can configure the directories and filenames for each site separately.
I don't know if this would be possible or not but it's something I've been wanting for a long time now.
The text was updated successfully, but these errors were encountered: