[niconico] Extracting Videos Formats Accessible only through the HTML5 player

[CompConf]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.06.25. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.06.25

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

• Video with new formats particular to the HTML5 player: http://www.nicovideo.jp/watch/sm31464864
• Another video, but with less available formats due to smaller resolution of uploaded video: http://www.nicovideo.jp/watch/sm30700278
• Older video without new formats, even when viewed in HTML5 player: http://www.nicovideo.jp/watch/sm500873

---

Description of your issue, suggested solution and other information

Niconico account credentials are required to access these videos, as well as all other videos on Niconico.

The HTML5 Player can be accessed when logged in by either manually enabling it (a blue bar above the flash player), or by using a browser without a flash plugin

Last year, Niconico rolled out a new encoding process for videos, along with a new HTML5 video player. Unlike the previous process, Niconico now re-encodes uploads into a variety of formats, based around the resolution size of the uploaded video and the time of the video. The new upload limit is 1.5GB, and the re-encoded videos now exceed 100MB, in contrast to former upload limits. Videos uploaded before the change will retain their original quality in the HTML5 player, with the economy encoding serving as the low quality.

While the old API and flash player still function as normally if a newly uploaded video is under 100MB, uploaded videos which are over 100MB will be re-encoded for the old API and flash player to be under 100MB (as far as I have seen). This will usually result in a video with significantly lower quality than the highest quality format available in the HTML5 player. Therefore, in order to download the highest quality format of newer uploaded videos, being able to download these new formats is essential, especially as people increasingly ignore the former upload limits.

[CompConf]

After further research, it seems that the new encodings in html5 originate from the new "DMC" server, while the traditional "smile" server retains the traditional encoding.

The Saccubus client has support for extracting information and downloading videos from both servers. The core of this process occurs in NicoClient.java.

In particular, it seems the key information on the HTML5 page lies in data-api-data

Hope that helps.

[yan12125]

I've added support for additional HTML5 formats and the feature will be available in the next version. For me (regular account), only 360p formats are available:

So youtube-dl reports those formats only:

$ youtube-dl -vF "http://www.nicovideo.jp/watch/sm31464864" --cookie ~/tmp/cookies.txt
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vF', 'http://www.nicovideo.jp/watch/sm31464864', '--cookie', '/home/yen/tmp/cookies.txt']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.08.06
[debug] Git HEAD: ee6a61166
[debug] Python version 3.6.2 - Linux-4.12.4-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 3.3.2, ffprobe 3.3.2
[debug] Proxy map: {}
[niconico] sm31464864: Downloading webpage
[niconico] sm31464864: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm31464864: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[info] Available formats for sm31464864:
format code                   extension  resolution note
h264_300kbps_360p-aac_64kbps  mp4        640x360     364k video@ 300k, audio@ 64k
h264_600kbps_360p-aac_64kbps  mp4        640x360     664k video@ 600k, audio@ 64k (best)

I believe you can get all formats if providing a premium account. If this is not the case in the next version, feel free to leave comments and I'll reopen this.

[CompConf]

Thanks!

I've already tested it from the repository, and for the new formats, it works completely as expecting for several videos which I have tested using a premium account.

However, there is one important aspect which is missing from the original issue, which I indirectly addressed in the original issue; my apologies for not directly stating it:

While the old API and flash player still function as normally if a newly uploaded video is under 100MB, uploaded videos which are over 100MB will be re-encoded for the old API and flash player to be under 100MB (as far as I have seen). This will usually result in a video with significantly lower quality than the highest quality format available in the HTML5 player.

The inverse also holds. For example, a new video of relatively short duration, which is uploaded with a filesize of 99MB by a premium account, will still have the original video available, which will have a higher quality than any of new formats. Mind you, this will increasingly become the minority of cases, but it is still present, especially if the uploader's video is below 720p (and subjected to a lower bitrate during re-encoding).

Therefore, it is important to not only provide the support for the new formats, but also maintain the ability to access the old API to optionally download the video through the old method.

If I am reading your commit correctly, perhaps this could accomplished by appending the format information of _format_id_from_url(video_real_url) to the condition at line 316-324. I think it would also be safe to assume that this format could be, by default, considered the lowest quality, since this is a very particular edge case, if still extant.

Once again, my apologies for not properly specifying this earlier.

[yan12125]

The inverse also holds. For example, a new video of relatively short duration, which is uploaded with a filesize of 99MB by a premium account, will still have the original video available, which will have a higher quality than any of new formats.

Hmm didn't think of that. Any example?

[CompConf]

Some particularly noticeable examples:
http://www.nicovideo.jp/watch/sm31513457
http://www.nicovideo.jp/watch/sm31109293
http://www.nicovideo.jp/watch/sm31123477
http://www.nicovideo.jp/watch/sm31138691
http://www.nicovideo.jp/watch/sm31563317

There are also many videos where the visual difference is not as great, but the format obtained through the old API is the best quality.

You also have cases like http://www.nicovideo.jp/watch/sm30988109, where the original video was encoded with a quality below the standards of the new formats. So while it might seem like it was re-encoded for both the flash player and the HTML5 player, the flash version is still the original video, and the new formats will only degrade the quality of the video through re-encoding, even if the overall bitrate and filesize are larger.