[Nico Nico Douga] "Unable to find video URL" error when downloading videos above 100MB

[Dioxaz]

• I've verified and I assure that I'm running youtube-dl 2016.08.24.1
• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

---

edit: error still present in 2016.09.19 update (another crashing example: http://www.nicovideo.jp/watch/sm29698977)

You might be aware that Nico Nico Douga raised its upload limit to 1.5GB instead of what formerly was 100MB after August 18th 2016. Some users already began uploading content above 100MB, and this apparently causes some trouble to our well known "video downloaders".

Now when trying to attempt to download such content with youtube-dl, I'm rewarded by a strange and irrelevant "Unable to find video URL" when it actually tries to retrieve data and gather video info. And yes, I'm using the latest version to date and am actually using my login credentials to download the video. For the moment, only VideoDownloadHelper (Firefox add-on) manages to download this particular video (featured in the log) but it unfortunately fails before the end.

• Here's one of the incriminating videos: http://www.nicovideo.jp/watch/sm29513760
• Here's my log (user and password replaced by ***):

F:\>youtube-dl -v -u *** -p *** http://www.nicovideo.jp/watch/sm29513760
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', 'http://www.nicovideo.jp/watch/sm29513760']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2016.08.24.1
[debug] Python version 3.4.4 - Windows-XP-5.1.2600-SP3
[debug] exe versions: ffmpeg 2.8.4, ffprobe N-67999-gd0879a9
[debug] Proxy map: {}
ERROR: Unable to find video URL; please report this issue on https://yt-dl.org/
ug . Make sure you are using the latest version; type  youtube-dl -U  to update
 Be sure to call youtube-dl with the --verbose flag and include its complete ou
put.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvarrvgfb\bild\youtube_dl\YoutubeDL.py", line 691, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvarrvgfb\bild\youtube_dl\extractor\common.py", line 347, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvarrvgfb\bild\youtube_dl\extractor\niconico.py", line 161, in _real_extract
youtube_dl.utils.ExtractorError: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

• Another log (from a different environment, Ubuntu 16.04):

$ youtube-dl -v -u *** -p *** http://www.nicovideo.jp/watch/sm29513760
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'http://www.nicovideo.jp/watch/sm29513760']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.24.1
[debug] Python version 2.7.12 - Linux-4.4.0-34-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 2.8.6-1ubuntu2, avprobe 2.8.6-1ubuntu2, ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2, rtmpdump 2.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm29513760: Downloading webpage
[niconico] sm29513760: Downloading video info page
[niconico] sm29513760: Downloading flv info
ERROR: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 691, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 347, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/niconico.py", line 161, in _real_extract
    raise ExtractorError('Unable to find video URL')
ExtractorError: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[vxbinaca]

@Dioxaz Does using a NND account allow for larger resolution rips?

[Dioxaz]

Since August 18th, it does, but when the source file is above 100MB. I've just checked without using my credentials. It successfully downloads, but it's a lower quality version.
For this particular video: it's turned into 78MB in size, downscaled to 960x540 and re-encoded to average bitrate. In fact, this is the file that nicofox downloads, too, but on the other hand this add-on requires logging in for download.
Whereas the original file is 155MB in size, 1920x720 in resolution and a higher peak for the bitrate (can't tell if i's VBR or ABR). I was able to download it after multiples retries with VideoDownloadHelper (it would always fail before reaching the end).

edit: error still present in 2016.09.19 update (other crashing example: http://www.nicovideo.jp/watch/sm29698977)

[vxbinaca]

@Dioxaz I ask because I archive NicoNico videos, and that's not many I'd have to re-rip. 10 days I did a handful of them.

Thanks

[badp]

I suppose this is the reason why I can't download this (content warning: Japan) but I can download this (content warning: Italy). The thumb_watch response looks completely different from the javascript returned by the working video, although both are javascript fragments that return some HTML.

Considered that my main reason for turning to youtube-dl for nicovideo urls is circumventing their registration screen, this is a little disappointing, but I suppose it cannot be helped :/

[Dioxaz]

@badp That's interesting. Even though it yields the same error message when connected, that video you first linked won't download when not connected either ("Unable to extract thumbPlayKey" error). It seems that both your video and mine return different responses than standard uploads. However, your video was uploaded in March 2016 and has nothing particular (it downloads with nicofox) so that's weird (edit: the video can be neither embedded nor watched outside nicovideo.jp, that would explain why youtube-dl has trouble downloading it).

[yan12125]

Just checked http://www.nicovideo.jp/watch/sm28462799 and http://www.nicovideo.jp/watch/sm29698977; both says:

Currently playing in low quality (economy) mode due to heavy user traffic.

Seems a premium account is necessary to download the original version? Or anyone know whether there's a "non-heavy" time span?

[Dioxaz]

You should be able to download these videos normally outside peak hours (the time slot that Nico Nico Douga allocates as peak hours). Otherwise an economy version is served instead. And these particular two videos don't have a high-resolution variant, only the regular 100MB one.