REQ: Allow custom Board server / URL

[Quackdoc]

Since there are a lot of image boards around but most of them share the same board software as others, it would be really nice if we could do something like imageboard_downloader -s moebooru -i https://yande.re ... this would help a lot instead of needing to hard code many sites that may not be popular.

as for custom syntax, maybe regex?

Instead of hoping it works out well, perhaps a config file might be better and more flexible?

#based on konachan
[yandere]
server = "moebooru"
useragent = ""
extractor_user_agent = ""
base_url = "https://yande.re"
post_url = ""
post_list_url = "https://yande.re/post.json"
pool_idx_url = ""
max_post_limit = 100
auth_url = ""

[FerrahWolfeh]

I've been thinking about adding some sort of custom server functionality and I found your idea quite interesting and I liked the concept of doing it in a config file-like structure.
Yet, by doing this I'll need to rewrite and/or overhaul a good part of the extractors logic (might not be that difficult), so, this might take a while for now... But I will do what I can

[Quackdoc]

I would be glad to see it happen for sure, this tool has already become my favourite grabber, So cheers for the great app.

[FerrahWolfeh]

Things are progressing on the custom_server branch. You may test the new operating mode there ;P (beware, the code is still very messy)

[FerrahWolfeh]

I finished something that works. Now it only needs some testing and a lot of cleanup. The config file for the custom servers is located at wherever the var IBDL_SERVER_CFG points at or in your config directory. It should be rather simple to add new servers according to the examples in there.

[Quackdoc]

I've tested it and a couple things,

• Yandere works fine
• rule34 and realbooru is now broken it wants "{}&tags={}", instead of "{}?tags={}", this doesn't seem to break gelbooru. This will fix rule34, but realbooru still seems broken

The application panicked (crashed).
Message:  called `Option::unwrap()` on a `None` value
Location: ibdl-extractors\src\imageboards\gelbooru\mod.rs:413

I also tried xbooru on the off chance that it would work but predictably it didn't since it uses Gelbooru Beta 0.2.0 and gelbooru is allergic to a semi-stable API, and it want's

format!(
    "https://img.xbooru.com//images//{}/{}",
    post["directory"].as_str().unwrap(),
    post["image"].as_str().unwrap().to_string()
)

I wonder if something like below would be suitable, it would at least work for the stuff that is exposed by json hopefully

[servers.xbooru]
pretty_name = "xbooru"
server = "gelbooru"
base_url = "https://xbooru.com/"
post_url = "https://xbooru.com/index.php?page=dapi&s=post&q=index&json=1"
post_list_url = "https://xbooru.com/index.php?page=dapi&s=post&q=index&json=1"
max_post_limit = 100
image_url = "https://img.xbooru.com//images//{}/{}"
image_args = [ "directory", "image" ]

EDIT: this is more or less the same syntax as what you have for realbooru, but a bit more flexible since you don't need to specify md5 and ext seperately

[FerrahWolfeh]

I managed to fix the rule34 error. It was easy enough to just place it along with the other query parameters (idk why I formatted that with text before passing to reqwest).

As for realbooru and your test for xbooru, yeah, those two use the same very old and archaic API, and realbooru originally had a small hack to work.

Realbooru might still not work (bc I forgot to change the server type inside the code 😅), but on the custom server, you may try using the server "gelbooru_020" and see if it works.

[Quackdoc]

I wasn't able to take a look at the the latest changes but im getting this error Image source returned status 404. Skipping download. is realbooru still hardcoded I wonder if it's trying to pull from realbooru?

[FerrahWolfeh]

Sort of is. It is pulling from realbooru, but since it's running without a beta 0.2 quirk (by using server = "gelbooru" instead of server = "gelbooru_020"), it will always fail or crash or whatever else. Using a custom beta 0.2 server with the correct config should work

[Quackdoc]

ah yeah, I get what you mean now, I made a realbooru custom server and it worked, I had tried the xbooru before.

[FerrahWolfeh]

Great! Now that the main stuff is working, I will tackle the full cleanup of the code and patching some other bugs.

Whatever happens, you may run the app with RUST_LOG=debug as it is quite verbose and can help diagnose some stuff.