Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[deviantart] jwt token validation failed #4548

Closed
mdashlw opened this issue Sep 19, 2023 · 24 comments
Closed

[deviantart] jwt token validation failed #4548

mdashlw opened this issue Sep 19, 2023 · 24 comments

Comments

@mdashlw
Copy link

mdashlw commented Sep 19, 2023

DeviantArt extractor works by faking a JWT token by using alg=none, it appears to be no longer working.

Unsigned alg=none: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0. (returns 401 token validation failed)
SIgned alg=HS256 by deviantart: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 (works)

JWT payload for both of these is exactly the same:

{
  "sub": "urn:app:7e0d188982264373a5f0d415ea0d26e0",
  "iss": "urn:app:7e0d188982264373a5f0d415ea0d26e0",
  "obj": [
    [
      {
        "path": "/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png"
      }
    ]
  ],
  "aud": [
    "urn:service:file.download"
  ]
}
@ClosedPort22
Copy link
Contributor

That sucks...

@Polygon1177
Copy link

Polygon1177 commented Sep 19, 2023

I got those errors with Deviantart as well, since yesterday. Wasn't an issue till the day before.

@Mirrorman95
Copy link

I recently reported a similar set of errors befalling another downloader.

@Twi-Hard
Copy link

I wrote this in a different issue as an edit to my existing post so I don't know if anybody even saw it. I just wanted to mention downloading the full res of this paid art still works. I don't know if that's significant or might help or not.

https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824

@mdashlw
Copy link
Author

mdashlw commented Sep 20, 2023

I wrote this in a different issue as an edit to my existing post so I don't know if anybody even saw it. I just wanted to mention downloading the full res of this paid art still works. I don't know if that's significant or might help or not.

https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824

Cannot reproduce.
[downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'

@Twi-Hard
Copy link

I tested it again right before posting my comment. It must be because I downloaded it right before the issue began so it still lets me I guess.

@ClosedPort22
Copy link
Contributor

I tested it again right before posting my comment. It must be because I downloaded it right before the issue began so it still lets me I guess.

That sounds very unlikely. Can you access the URL in a browser?

@Twi-Hard
Copy link

Twi-Hard commented Sep 20, 2023

Yes.

❯ file 'deviantart_603249824_The Moon Rises.png'
deviantart_603249824_The Moon Rises.png: PNG image data, 7680 x 4320, 8-bit/color RGBA, non-interlaced

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.
The dot is a part of the url. It was downloaded without an archive and with --no-skip

@mdashlw
Copy link
Author

mdashlw commented Sep 20, 2023

Yes.

❯ file 'deviantart_603249824_The Moon Rises.png'
deviantart_603249824_The Moon Rises.png: PNG image data, 7680 x 4320, 8-bit/color RGBA, non-interlaced

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ. The dot is a part of the url. It was downloaded without an archive and with --no-skip

Can you access the url in incognito? Could you try reproducing with curl?

@Twi-Hard
Copy link

It works in incognito. It doesn't work with curl:

❯ cat image.png
token validation failed

@ClosedPort22
Copy link
Contributor

It could be a regional cache sort of thing. I just checked and Cloudfront caches the images for a long period of time:

cache-control: public, max-age=2592000, immutable
age: 74973

@mdashlw
Copy link
Author

mdashlw commented Sep 20, 2023

It could be a regional cache sort of thing. I just checked and Cloudfront caches the images for a long period of time:

cache-control: public, max-age=2592000, immutable
age: 74973

Likely. you can bypass Cloudfront cache by adding random query params like ?token=...&1234

@zakman4466
Copy link

zakman4466 commented Sep 22, 2023

To confirm, once the commit goes live, should the issue with these errors (and the associated 401 errors) be resolved?

@kattjevfel
Copy link
Contributor

@zakman4466 it "resolves" it by just always going with the lowres images since you can't get the highres pictures anyway.

mikf added a commit that referenced this issue Sep 24, 2023
And again, a huge thank you to @Ironchest337
for discovering this.
@mikf
Copy link
Owner

mikf commented Sep 24, 2023

Ironchest337 did it again! 20d1683

@mikf mikf closed this as completed Sep 24, 2023
@ClosedPort22
Copy link
Contributor

Ironchest337 did it again! 20d1683

Weirdly enough, that trick does not work for the example given by @mdashlw (403 Forbidden), and that's the only exception I've noticed so far. At first I thought it was because it was a relatively new submission, but later I was able to download even newer submissions.

@mikf
Copy link
Owner

mikf commented Sep 24, 2023

It works when using the full DA URL, but this post is also downloadable without any JWT shenanigans. It doesn't actually get touched by any JWT logic to begin with, even when original is disabled.

@Ironchest337
Copy link

Weirdly enough, that trick does not work for the example given by @mdashlw (403 Forbidden), and that's the only exception I've noticed so far. At first I thought it was because it was a relatively new submission, but later I was able to download even newer submissions.

I went ahead and checked and I believe it's an issue of the original token they are using. Removing everything after urn:app: for sub and iss allows it to work

@mdashlw
Copy link
Author

mdashlw commented Sep 25, 2023

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

@Mirrorman95
Copy link

Mirrorman95 commented Sep 25, 2023

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

@mdashlw
Copy link
Author

mdashlw commented Sep 26, 2023

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

Same as before, i.e. downloading all deviantart images, including paid, in original resolution

@Mirrorman95
Copy link

Mirrorman95 commented Sep 28, 2023

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

Same as before, i.e. downloading all deviantart images, including paid, in original resolution

I just tried that The Moon Rises example (using https://github.com/mhogomchungu/media-downloader as a GUI), but it gave me this error:

/gallery-dl" "https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824"
[downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'
[download][info] Trying fallback URL #1

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

@mdashlw
Copy link
Author

mdashlw commented Sep 28, 2023

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

the fix isn't released yet. install latest gallery-dl directly from github:

python3 -m pip install -U -I --no-deps --no-cache-dir https://github.com/mikf/gallery-dl/archive/master.tar.gz

@Mirrorman95
Copy link

Mirrorman95 commented Sep 28, 2023

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

the fix isn't released yet. install latest gallery-dl directly from github:

python3 -m pip install -U -I --no-deps --no-cache-dir https://github.com/mikf/gallery-dl/archive/master.tar.gz

That did it. I have to run it from my Python3 folder on the command line, but it works. Thank you.

EDIT: Some of the deviations in three specific galleries I tried still give me errors that include " (no refresh-token)". Other people's galleries give me no such trouble at all.

EDIT2: Here are links to a few of those images. They're not even blurred and don't even have pay buttons, but they are NSFW:
https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134
https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-2-nude-Photography-910500549

The errors they give me are:
gallery-dl https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134
[deviantart][info] Requesting public access token
[deviantart][warning] Unable to access premium content (no refresh-token)
[deviantart][info] No results for https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants