Using in conjunction with curl to download .htm files, failing to do so for Deviantart deviations with / in title?

[Jeddostotle7]

Ok, so I'm continuing on with a new problem I'd faced and didn't get an answer to while getting some help weeks ago for my prior questions, and haven't been able to ask about again before now for various reasons. Anyway: so currently, for Deviantart I have an exec postprocessor that uses curl to grab .htm files of the deviation pages, in addition to the image itself, the metadata .json, and description .txt the program itself and the metadata postprocessors grab. This worked fine for me in my tests, EXCEPT for one deviation: https://www.deviantart.com/korblborp/art/354079667. It'll download the image, .json, and .txt just fine as usual, but something breaks down with the exec postprocessor using curl and doesn't grab the .htm. I'm assuming the reason is because there's a / in the title of the deviation. When I run the command gallery-dl -c C:\Users\tfaho\gallery-dl\gallery-dl-example.conf --postprocessor metadata "https://www.deviantart.com/korblborp/art/354079667" to test the site on its own, it throws up this:

deviantart: Requesting public access token
C:\Users\tfaho\gallery-dl\deviantart\korblborp\deviantart_354079667_MJOLNIR Gen 2⧸ARC.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file
Warning: \\?\C:\Users\tfaho\gallery-dl\deviantart\korblborp\MJOLNIR Gen
Warning: 2/ARC-354079667.htm: Invalid argument
  1  549k    1  7685    0     0   9758      0  0:00:57 --:--:--  0:00:57  9764
curl: (23) Failure writing output to destination
postprocessor.exec: '['C:\\Users\\tfaho\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe', '-o', '\\\\?\\C:\\Users\\tfaho\\gallery-dl\\deviantart\\korblborp\\MJOLNIR Gen 2/ARC-354079667.htm', 'https://www.deviantart.com/korblborp/art/354079667']' returned with non-zero exit status (23)

I was trying to figure out where the problem might lie, and then in writing this out, I think I realized the problem must specifically be with the filename-writing part of the exec postprocessor using curl, since it doesn't have any provision for changing a normal / slash into something else like a Big Solidus ⧸ slash the way gallery-dl normally does:

     "postprocessors": [
         {
             "name": "metadata",
             "mode": "custom",
             "directory"       : "Descriptions",
             "content-format"  : "{description}\n",
             "extension-format": "descr.txt"
         },
         {
             "name": "exec",
             "command": [
                 "C:\\Users\\tfaho\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe",
                 "-o", "{_directory}{title}-{index}.htm",
                 "https://www.deviantart.com/{author[username]}/art/{index}"
             ]
         }
     ]

Is there anything I can put in there to have it modify problematic characters like slashes when writing the filename so it doesn't fail on works with them in the name?

Also, once this is figured out, I would like help figuring out what I'd need to modify on these postprocessors to get basically the same setup of not just image and metadata .json, but also a description.txt postprocessor and exec curl .htm postprocessor, to other extractors like pixiv, furaffinity, and boorus (I think those are all I'd need them anytime in the near future).

[ClosedPort22]

Normally I'd manually replace the offending characters like this (reference):

{title:R:/_/}  # works
{title:R//_/}  # does not work

But for some reason it doesn't like slashes and throws an error. It would be nice to have a conversion specifier to sanitize the variable like yt-dlp does, but for the time being, I think you'd have to use a Python expression in place of the argument for "-o":

"\fE '{}{}-{}'.format(_directory, title.replace('/', '_'), index)"

The exec postprocessor does not support directory. In order to tell curl to download your HTML document to a separate directory, you need to modify the argument for "-o" and use "--create-dirs":

{_directory}mydir\\{title}-{index}.htm
"\fE '{}{}{}\\-{}'.format(_directory, title.replace('/', '_'), 'mydir', index)"

Also, once this is figured out, I would like help figuring out what I'd need to modify on these postprocessors to get basically the same setup of not just image and metadata .json, but also a description.txt postprocessor and exec curl .htm postprocessor, to other extractors like pixiv, furaffinity, and boorus (I think those are all I'd need them anytime in the near future).

I'd recommend checking the metadata JSON files before deciding whether to download the web pages. Much if not all of the metadata available on the web page is already included in the JSON file. Similarly, a JSON postprocessor would dump everything available to gallery-dl, so a description PP would only make it slightly more readable. Anyway, take furaffinity for example, here is the complete configuration:

"furaffinity": {
    "#": "remove this line if you prefer plain text instead",
    "descriptions": "html",
    "postprocessors": [
        {
            "#": "this is already included in the json file!",
            "name": "metadata",
            "mode": "custom",
            "directory": "Descriptions",
            "content-format": "{description}\n",
            "extension-format": "descr.htm"
        },
        {
            "name": "metadata",
            "mode": "json",
            "filename": "{id}_info.json",
            "event": "file"
        },
        {
            "name": "exec",
            "command": [
                "C:\\Users\\tfaho\\curl\\curl-7.86.0_2-win64-mingw\\bin\\curl.exe",
                "--create-dirs", "-o", "{_directory}mydir\\{title}-{id}.htm",
                "https://www.furaffinity.net/view/{id}/"
            ]
        }
    ]
}

You can run gallery-dl -K or check the JSON files directly to get a list of available fields for a particular extractor.

[ClosedPort22]

Hm, this seems to be an encoding issue. I tried same configuration for exec and got the same error (23), except that the filename part of the line below the "Warning: Failed to open the file" line was a jumbled mess. I'm guessing it has something to do with the system's code page. Personally I'd rather use underscores than deal with this mess...

[rautamiekka]

Ohhh, actually using the Unicode code for it makes sense, and that is indeed very elegant.

Extremely non-elegant+painful when you don't have a GUI/TUI and ?* ("must have 1 character at minimum") won't do due to ambiguity, I found that out the hard way.

[Jeddostotle7]

Oh, I've made progress! Turns out the reason the .replace().replace() etc. version wasn't working wasn't because of the quotation mark failing to be escaped by its one backslash, but because of the backslashes by themselves not having sufficient escaping. Just changed it from .replace('\\', 'test1') to .replace('\\\\', 'test1') and it seems to actually be working now! Now I just need to figure out how to get the alternate forms of the characters in the replacer spots in a way it'll actually accept.

EDIT: ok, so my first couple different attempts at using a chr() didn't function, because as I know now I misunderstood how they function (aka I left them within the ' ' quotes); so then I tried inserting the Big Solidus slash in for the replacer for normal slash, and the program didn't throw up any errors. It even output the .htm file, although for some reason..... the / ended up replaced with an underscore??? Even though there's nothing in this expression about an underscore as a replacer???? And then I tried the chr(10744) without the ' ' quotes, and it worked the same way, which meant that I figured out how to use the chr() function in this situation, but that doesn't solve the roadblock of seemingly the replacement character itself getting replaced by an underscore.

Just for reference, the expression currently looks like this:

"-o", "\fE '{}deviantart_{}_{}_{}.htm'.format(_directory, author['username'], index, title.replace('\\\\', 'test1').replace('/', x).replace('|', 'test3').replace(':', 'test4').replace('*', 'test5').replace('?', 'test6').replace('\"', 'test7').replace('<', 'test8').replace('>', 'test9'))",

except in place of x was alternately 'chr(10744)', '(chr(10744))', 'print(chr(10744))' ,'⧸', and chr(10744).

[Jeddostotle7]

Alright, after some further testing, I think this really is just some kind of explicit character-replacement thing on curl's end. Specifically I tried one or two \ in front of the Big Solidus, and with one it told me it was invalidly escaped, but with two it output the .htm with a _ in place of the original normal slash in the deviation's title. Why curl has a functioning character-replacement mechanism to handle nonstandard characters in this specific function but can't handle the / itself I have no idea. Regardless, it seems like I would need some kind of additional command to add to them to get it to avoid changing them, or have to make a config for curl itself???

[Jeddostotle7]

Ok, long time since last update but I finally have a solution, even though I have no idea why it works.

In the time since my last message, I had also asked a question on Stack Overflow about why curl was replacing characters with underscores, and the one person who in that time responded mentioned that on his Linux version, it didn't do that. A while after waiting for more responses on that question and getting none, I decided to just bite the bullet and have curl put underscores in the filenames since it wouldn't be that big a deal since the .htms are right next to the images, metadata jsons, etc. anyway because of how i structured the filenames with having the artist name and then site upload id before the actual filename.

However, when I tested it on Pixiv, I found out a much MUCH larger problem with curl's replacing of "odd" characters with underscores: it apparently does it with Japanese characters as well, and with the filenaming command including different folders by uploader, when there's something uploaded by a user with Japanese characters in their username (as, of course, there are many of on Pixiv), it replaces those characters with underscores and then, failing to find a folder with the name ______, it errors out and doesn't even download the .htm. So, I was much more urgent to find a solution.

So, I asked my friend for help, and he also tested to see what versions of curl he experienced the problem with; he didn't experience it on Linux, but did on his Windows version of curl. So, he decided to download the current source code for curl from their github repo and compile it himself and run through it with a debugger to see what the problem was. Oddly, though, for reasons still unknown to us, his compiled version.... wasn't changing any characters to underscores?? Like, it was just working fine like his and the Stack Overflow guy's Linux versions.

So, TLDR: my friend gave me a Windows version of curl that actually works and doesn't have the "changing characters to underscores" problem simply by grabbing the source code and compiling it himself rather than downloading the most recent Windows release or using the built-into-Windows version. We have no idea why it works but it does (my best guess is to do with slight differences from different compilers), and so I can FINALLY move on with my downloading.