youtube-dl is unable to download sensitive content from Tumblr

[amg1127]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like this: [x])
• Use the Preview tab to see what your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.07.30.1. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.07.30.1

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ /usr/bin/youtube-dl -v -u "${TUMBLR_USERNAME}" -p "${TUMBLR_PASSWORD}" 'http://sweetliciousnet.tumblr.com/post/163673652169'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', 'http://sweetliciousnet.tumblr.com/post/163673652169']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.30.1
[debug] Python version 3.6.2 - Linux-4.12.3-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.3.2, ffprobe 3.3.2, rtmpdump 2.4
[debug] Proxy map: {}
[Tumblr] 163673652169: Downloading webpage
[generic] 163673652169: Requesting header
WARNING: Could not send HEAD request to https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/: HTTP Error 404: Not Found
[generic] 163673652169: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] 163673652169: Extracting information
ERROR: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 433, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/generic.py", line 2944, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/

---

Description of your issue, suggested solution and other information

A recent Tumblr policy change now requires a Tumblr user to log in and turn off safe mode in order to access sensitive content. According to their FAQ regarding sensitive content:

I marked my blog as explicit. What happens now?

Now you can be sure that only suitable audiences are seeing your content. Blogs that are marked explicit are kept out of search results for people in Safe Mode. Anyone viewing your blog on the web will have to be logged in (with safe mode off) to see it.

However, I am unable to download sensitive content from Tumblr even though I supply valid account credentials on youtube-dl command line.

In order to reproduce the issue, one can obtain a Tumblr account here. An adult content that youtube-dl is unable to get can be accessed here.

[amg1127]

@dstftw ,

In order to reproduce the issue, please use the following Tumblr account:

Username: 13829youtubedlbug@thisisnotmyrealemail.com
Password: youtube-dl_2017.07.30.1

The account is linked to a disposable mail service located at www.mailinator.com .

$ export TUMBLR_USERNAME='13829youtubedlbug@thisisnotmyrealemail.com'
$ export TUMBLR_PASSWORD='youtube-dl_2017.07.30.1'
$ /usr/bin/youtube-dl -v -u "${TUMBLR_USERNAME}" -p "${TUMBLR_PASSWORD}" 'http://sweetliciousnet.tumblr.com/post/163673652169'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', 'http://sweetliciousnet.tumblr.com/post/163673652169']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.08.06
[debug] Python version 3.6.2 - Linux-4.12.3-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.3.2, ffprobe 3.3.2, rtmpdump 2.4
[debug] Proxy map: {}
[Tumblr] 163673652169: Downloading webpage
[generic] 163673652169: Requesting header
WARNING: Could not send HEAD request to https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/: HTTP Error 404: Not Found
[generic] 163673652169: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] 163673652169: Extracting information
ERROR: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 433, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/generic.py", line 2944, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/

[Hrxn]

PS F:\> youtube-dl --ignore-config -v -u <tumblr_test_account> "http://sweetliciousnet.tumblr.com/post/163673652169"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-v', '-u', 'PRIVATE', 'http://sweetliciousnet.tumblr.com/post/163673652169']
Type account password and press [Return]:
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2017.08.27.1
[debug] Python version 3.6.1 - Windows-10-10.0.15063-SP0
[debug] exe versions: ffmpeg N-87043-gf0f4888, ffprobe N-87043-gf0f4888
[debug] Proxy map: {}
[Tumblr] 163673652169: Downloading webpage
[generic] 163673652169: Requesting header
WARNING: Could not send HEAD request to https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/: HTTP Error 404: Not Found
[generic] 163673652169: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] 163673652169: Extracting information
ERROR: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/
Traceback (most recent call last):
  File "c:\users\hrxn\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "c:\users\hrxn\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\extractor\common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "c:\users\hrxn\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\extractor\generic.py", line 2989, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.tumblr.com/safe-mode?url=http://sweetliciousnet.tumblr.com/post/163673652169/

PS F:\>

So, yeah. I used a wrong password on purpose. Authentication does not work at all. Has it ever for Tumblr? Not sure actually, can't remember that it was used at all..

OP, what you could alternatively do is try to supply cookies to youtube-dl.

@dstftw could you please explain what you meant here with account required? Best way forward to add proper auth to tumblr extractor? With support for .netrc, for example..

[raindog469]

I can confirm that the workaround suggested by @Hrxn (extracting and supplying cookies) works as intended for tumblr URLs that would normally redirect to the safe-mode page in youtube-dl version 2017.09.15.

I did this by extracting the cookies in wget format using the "cliget" firefox extension, then converting those cookies to Netscape cookie jar format using the tool at http://crdx.org/misc/cookies/, saving in a local file and passing that file to youtube-dl's --cookies option. I expect this will only last as long as tumblr thinks I'm logged in, so it's not a particularly convenient workaround, but it is faster than visiting every page and using a Firefox download helper to get the videos.

Please note that I also had to use the --user-agent option to match the Firefox version with which I signed into tumblr, also provided with the cliget extension's output.