DeviantArt download fails after few images

[UnforeseenOcean]

First 20 or so images are downloaded properly it seems, but after that, it keeps returning json error. The log with -v switch is attached below.

[1/2] https://www.deviantart.com/nahelus
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/nahelus'
[deviantart][debug] Using DeviantartUserExtractor for 'https://www.deviantart.com/nahelus'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/nahelus/gallery'
[deviantart][debug] Using custom API credentials (client-id 14376)
[deviantart][debug] Sleeping 1.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/nahelus HTTP/1.1" 403 919
[deviantart][error] Unable to download data:  JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[deviantart][debug]
Traceback (most recent call last):
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\job.py", line 127, in run
    for msg in extractor:
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\deviantart.py", line 103, in items
    profile = self.api.user_profile(self.user)
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\cache.py", line 34, in __call__
    value = self.cache[key] = self.func(*args, **kwargs)
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\deviantart.py", line 1281, in user_profile
    return self._call(endpoint, fatal=False)
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\deviantart.py", line 1360, in _call
    data = response.json()
  File "C:\Users\Torbjorn\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[defaultd3f4lt]

Same here too, here's my log:

[deviantart][debug]
Traceback (most recent call last):
  File "requests\models.pyc", line 971, in json
  File "json\__init__.pyc", line 357, in loads
  File "json\decoder.pyc", line 337, in decode
  File "json\decoder.pyc", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 127, in run
  File "gallery_dl\extractor\deviantart.pyc", line 103, in items
  File "gallery_dl\cache.pyc", line 34, in __call__
  File "gallery_dl\extractor\deviantart.pyc", line 1281, in user_profile
  File "gallery_dl\extractor\deviantart.pyc", line 1360, in _call
  File "requests\models.pyc", line 975, in json
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)```

[TheMerricat]

Having the same issue as well.

[mikf]

Seems like one now gets blocked for doing too many requests in too short a time:

ERROR: The request could not be satisfied

Request blocked.

We can't connect to the server for this app or website at this time.
There might be too much traffic or a configuration error.
Try again later, or contact the app or website owner.

If you provide content to customers through CloudFront,
you can find steps to troubleshoot and help prevent this error
by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront)

--sleep-request or using a somewhat high wait-min might help (3 seconds?) once your IP / ClientID is no longer blocked which seems to happen after only a few minutes.

[Twi-Hard]

I'm getting 2 at a time and each error skips to the next extractor which is really bad. In this you can see if goes from "gallery" to "journal" to "scraps" in about a second so none of the accounts I'm trying to download from are getting through the whole account since the extractor stops when it gets this error. Is it possible to make gallery-dl retry what it failed to download after the rate limit ends?

[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/55883A45-DA56-7A60-A761-AE7BA49960D0?mature_content=true HTTP/1.1" 403 919
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/55883A45-DA56-7A60-A761-AE7BA49960D0?mature_content=true HTTP/1.1" 403 919
[deviantart][error] Unable to download data:  JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[deviantart][debug]
Traceback (most recent call last):
  File "/home/mane/.local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 127, in run
    for msg in extractor:
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 144, in items
    content = self._extract_content(deviation)
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 347, in _extract_content
    self._update_content(deviation, content)
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 397, in _update_content_default
    data = self.api.deviation_download(deviation["deviationid"], public)
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 1299, in deviation_download
    return self._call(endpoint, params=params, public=False)
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 1418, in _call
    data = response.json()
  File "/home/mane/.local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[deviantart][debug] Using DeviantartJournalExtractor for 'https://www.deviantart.com/8bitlola/posts'
[deviantart][debug] Using custom API credentials (client-id 11034)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/browse/user/journals?username=8BitLola&offset=0&limit=50&mature_content=true&featured=false HTTP/1.1" 403 919
[deviantart][error] Unable to download data:  JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[deviantart][debug]
Traceback (most recent call last):
  File "/home/mane/.local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 127, in run
    for msg in extractor:
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 115, in items
    for deviation in self.deviations():
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 1477, in _pagination
    data = self._call(endpoint, params=params, public=public)
  File "/home/mane/.local/lib/python3.10/site-packages/gallery_dl/extractor/deviantart.py", line 1418, in _call
    data = response.json()
  File "/home/mane/.local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[deviantart][debug] Using DeviantartScrapsExtractor for 'https://www.deviantart.com/8bitlola/gallery/scraps'
[deviantart][debug] Using custom API credentials (client-id 11034)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET / HTTP/1.1" 403 919
[deviantart][info] Waiting until 12:50:21 for rate limit reset.

[mikf]

These errors now get properly handled and downloading gets resumed after 5 minutes (waiting for 3 minutes is not long enough). (925123e)

CloudFront blocking requests to DeviantArt happened before quite some time ago (#655), but then it was only for non-API requests. Seems like it got extended to all HTTP requests including OAuth API ones.

This fix is included in the latest v1.26.9 release, by the way.

[Twi-Hard]

Do you think if somebody brought up the fact that cloudfront is doing this with the api now too that they might change it? It might have been unintentional.

[TheMerricat]

I can tell you right now that they're going to have to do something because it also broke their own official mobile app. I can scroll my watch feed for about two screens before I get knocked offline.

…

On Sun, Mar 24, 2024, 5:33 AM Twi-Hard ***@***.***> wrote: Do you think if somebody brought up the fact that cloudfront is doing this with the api now too that they might change it? — Reply to this email directly, view it on GitHub <#5363 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2MFSYF6HV3K67DGVQPDKTYZ2TYVAVCNFSM6AAAAABFEMTGUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWG43DINZUGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mikf]

I'd think this was a deliberate decision by DeviantArt, given that CloudFront's services cost money. They wouldn't accidentally extend CF's protection without doing some sort of cost-benefit analysis beforehand, or would they?

You might as well give it a try. Maybe it works, especially if this whole thing has negative effects on their mobile users and otherwise.