Trying to download from deviantart

[swaprin]

Hi so I’m kind of a noob
I’m trying to download a gallery from deviantart and I’m getting huge delays.
So I managed to get a refresh token but have no idea where or how to put it.
Could anyone help me?
Thanks in advance

[swaprin]

Okay so reading up, I did use oauth:deviantart and it said token is in cache, but when I try downloading I still get the delays

[biznizz]

Could you post what your extractor settings for deviantart in your config file looks like?

[swaprin]

Where would I be able to find that?

[biznizz]

If you don't have a configuration set up, read this:

https://github.com/mikf/gallery-dl#configuration

What's your OS? And do you have a program that allows you to edit .conf or .json files?

[swaprin]

Im using Windows 10. What programs can I use for editing?

[swaprin]

So I read what you sent me and I downloaded gallery-dl.conf. So now I must rename the file and put it into the gallery-dl folder in the user/myusername folder (the same place where downloads save). I have to edit the gallery-do.conf file so how do I go about this?

[biznizz]

This program, Notepad++, is a versatile program that is an improvement over the stock MS Notepad. When installed, it get linked to the context menu when you right click, so you can open and edit .json or .conf files easily. https://notepad-plus-plus.org/downloads/v7.8.9/

If you downloaded the example gallery-dl.conf, you do not have to rename it, simply keep it in your %USERPROFILE% folder (aka C:/Users/USER folder).

[biznizz]

When you are able to open the .conf file for editing, respond and I'll advise you what to change.

As is, this should be the stock extractor settings for deviantart, so confirm if you have this.

 "deviantart":
        {
            "extra": false,
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": false,
            "original": true,
            "quality": 100,
            "wait-min": 0
},

[swaprin]

Hi

I downloaded note pad ++ and copied gallery-dl.conf to the user/username folder.

I can confirm i have the stock extractor

[biznizz]

Okay, in addition to changes to the extractor settings that will come later, the best results for using deviantart extraction involve creating/using a client id and your browser cookies.

Regarding the cookies, there are extensions for Chrome and Firefox that allow the exportation of the cookies stored in your browser. For example: I use cookies.txt to export the cookies into, well, a txt file. You can find it here or here

While you are logged onto DA, export your cookies to create a txt file. Put it in a place where it won't be accidentally deleted, and when the time comes, we'll be able to link your cookies.txt in your configuration file.

Next is to create an app on DA to get a client-id and a client-secret that allows for better fetching for the extractor. Follow the steps listed here: https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractordeviantartclient-id--client-secret

When you have your app set up and your cookies ready, we'll move on to the next steps.

[swaprin]

Okay I have my cookies, client Id and client-secret

[biznizz]

Alright, open your gallery-dl.conf with Notepad++ and go to your extractor settings for Deviantart. Make the following changes.

"deviantart":
		{
			"include": "gallery,scraps",
			"refresh-token": "cache",
                        "client-id": "####",
			"client-secret": "######",
			"flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": true,
            "original": true,
		    "quality": 100,
		    "extra": true,
            "wait-min": 0,
			"cookies": "C:\\PATH\\TO\\cookies.txt",
			"cookies-update": true
            
		},

You can make adjustments here and there, but this is what they mean.

include means that the extractor will only grab what values you have listed; in this case it's only going to grab gallery and scraps, not favorites the profile user has faved.

refresh-token is the Oauth value. Setting it to cache means that it will read your cache for the value as opposed to having to manually input the value into the config.

Use the numbers your got from your app and put it into client-id and client-secret.

extra is set to true so the extractor can get sta.sh links that might be in the description of any submissions.

original being set to true means that the extractor will try to get the original submission file if it is available, where as quality is set to 100 so if the original file isn't available, the extractor will download a JPEG at the best quality it can.

wait-min is set to 0 by default, but adding numbers to the value will increase the number of second between API calls, which should reduce timeout errors. Only change if you encounter errors.

cookies should be linked directly to where your cookies.txt should be, such as C:\\Users\\USER\\Desktop\\cookies.txt (you have to double slash each escape to prevent parsing errors), and cookies-update set to true will update your cookies.txt file if it detects any new information encountered in the cookies from interacting with the API.

My suggestion regarding cookies is that, since the exported txt file contains cookies from all websites used on the browser, to copy and paste all deviantart.com cookies into a separate txt file and have that called your cookies.txt.

Then, at the bottom of your config file, add this below the "output" setting:

"cache": {
        "file": "%USERPROFILE%/.gallery-dl.cache"
    },

This will have your cache file generate in your %USERPROFILE%, where you can identify it easily.

Once this is all done, save the changes made to your configuration file, and run another Oauth request for Deviantart, since you will need to generate a new cache file with a new refresh-token.

[swaprin]

` "output":
{"cache": {
"file": "%USERPROFILE%/.gallery-dl.cache"
},
"mode": "auto",
"progress": true,
"shorten": true,
"log": "[{name}][{levelname}] {message}",
"logfile": null,
"unsupportedfile": null
},

"netrc": false

}`

So for the last part I copied it exactly this, is it correct? I left %USERPROFILE% as is. I ask this because I dont see a cache file generated.

Also I tried downloading again and Im still experiencing delays but they seem shorter.

[biznizz]

"output":
    {
        "mode": "terminal",
        "log": {
            "level": "info",
            "format": {
                "debug"  : "\u001b[0;37m{name}: {message}\u001b[0m",
                "info"   : "\u001b[1;37m{name}: {message}\u001b[0m",
                "warning": "\u001b[1;33m{name}: {message}\u001b[0m",
                "error"  : "\u001b[1;31m{name}: {message}\u001b[0m"
            }
        },
        "logfile": {
            "path": "~/gallery-dl/log.txt",
            "mode": "w",
            "level": "debug"
        },
        "unsupportedfile": {
            "path": "~/gallery-dl/unsupported.txt",
            "mode": "a",
            "format": "{asctime} {message}",
            "format-date": "%Y-%m-%d-%H-%M-%S"
        }
    },

    "cache": {
        "file": "%USERPROFILE%/.gallery-dl.cache"
    },

    "netrc": false
}

Use this at the end. Keeping %USERPROFILE% in the config file is acceptable to use since Windows knows that points to your User folder.

EDIT: If you are still experiencing delays, try increasing the wait-min to 2 or 4. Also, could you post the error you get with the delays, just for curiosity's sake?

Is it something like "API responded with 429 Too Many Requests"?

[swaprin]

The error I was getting was [deviantart][warning] API responded with 429 Too Many Requests. Using 64s delay.
And occasionally say �[1;37mdeviantart: Refreshing private access token�[0m

It ssems to be working fine now though. Thank you so much for your help and assistance 👍
I'm very grateful, you're awesome :)

[biznizz]

Just keep in mind that, infrequently, the cookies-update for your browser might update incorrectly. You'll be able to tell when the userinfo cookie no longer contains your DA user name in the string.

All you have to do is just export the cookies again. It's one of the reasons I have a separate txt file for my DA cookies only that my extractor points to; it's easier to see if it updates incorrectly.

Also, keep in mind that the refresh-token only has a life of 3 months (or until the erasure of the cache file), so you'll have to update it at least once every quarter.

Some things with this program are set it and forget it, but others aren't, so you'll have to keep an eye out. Thankfully, you can keep files open in NP++ and it'll remember them (and tell if you they were changed such as the cookies.txt) even when you close the program, so I'd keep your cookies.txt and your config file open in it at all times.

[swaprin]

I see, so I’ll be sure to keep an eye out for that and sort it out when it happens.
Thanks for telling me that, huge help!

[biznizz]

So I read what you sent me and I downloaded gallery-dl.conf. So now I must rename the file and put it into the gallery-dl folder in the user/myusername folder (the same place where downloads save). I have to edit the gallery-do.conf file so how do I go about this?

One last thing, since I noticed you closed this: you are able to change the location of your downloaded files from when you use gallery-dl.

At the very top of your extractor settings in your configuration file, there is a setting for base-directory. which is by default set to ./gallery-dl/ aka the gallery-dl folder in your %USERPROFILE%

You can change the settings for your base directory to a different location. E.g. if you have a separate hard drive, D:, and don't want to have all your saves on your C: drive due to space, you can set your download location to

"base-directory": "D:/downloaded galleries/",

which will create a new folder called downloaded galleries in your D: drive where in all your downloaded files are sorted by website/user (such as D:/downloaded galleries/deviantart/USER). Useful for boot drives that have limited storage space or for user preference.

[swaprin]

Thank you so much,

So I read what you sent me and I downloaded gallery-dl.conf. So now I must rename the file and put it into the gallery-dl folder in the user/myusername folder (the same place where downloads save). I have to edit the gallery-do.conf file so how do I go about this?

One last thing, since I noticed you closed this: you are able to change the location of your downloaded files from when you use gallery-dl.

At the very top of your extractor settings in your configuration file, there is a setting for base-directory. which is by default set to ./gallery-dl/ aka the gallery-dl folder in your %USERPROFILE%

You can change the settings for your base directory to a different location. E.g. if you have a separate hard drive, D:, and don't want to have all your saves on your C: drive due to space, you can set your download location to

"base-directory": "D:/downloaded galleries/",

which will create a new folder called downloaded galleries in your D: drive where in all your downloaded files are sorted by website/user (such as D:/downloaded galleries/deviantart/USER). Useful for boot drives that have limited storage space or for user preference.

Thank you so much, I just installed a new hard drive and this was exactly what I was looking for :) Thank you for the tremendous help