[ie/niconico] Add support for DMS server; use robust info extraction logic; allow empty danmaku #9282
Conversation
This commit cannot be finished without @xpadev-net's work. Co-authored-by: XPA <96982836+xpadev-net@users.noreply.github.com>
We are not able to check if the video is behind a paywall, login required or geo-restricted.
For comments' download, we don't need to care about if the video is members-only. No comments means no comments.
test_download.py:239: self.assertTrue(tc_filename in finished_hook_called) How to solve it?
|
I cannot make those tests complete (522d9d1): |
|
For thumbnails, have both "ext" and "preference" been removed? yt-dlp/yt_dlp/extractor/niconico.py Lines 451 to 452 in 998dffb |
pukkandan
added
site-enhancement
|
Small correction about sm43110779 (34') - 【高画質版】ヒジリーリ・リーリリEx【東方MMD】 - ニコニコ. It looks like resolutions above 480p are available to guests as well, as I've just checked. Either I made a mistake when trying to review it the first time (apologizes) or they updated something that made that video available to unregistered and non-premium users as well. This is a video longer than 30 minutes. Please someone confirm for other recent videos over 30 minutes in duration and being available in formats above 480p. |
yt_dlp/extractor/niconico.py
Outdated
| 'audios': ('media', 'domand', 'audios', ..., { | ||
| lambda item: (item['id'], { | ||
| 'format_id': str(parse_bitrate(item['id'])), | ||
| 'abr': float_or_none(item['bitRate'], scale=1000), | ||
| 'asr': item['samplingRate'], | ||
| 'acodec': 'aac', | ||
| 'ext': 'm4a', | ||
| }) if item['isAvailable'] else None |
There was a problem hiding this comment.
I think once you have to use a transformation to construct an entire format_dict then it's time to break this up and use separate variables/traversals.
Can you show an example of what the POST data that we send with the API request should look like? Maybe I'm misreading the code, but it looks like we are sending a tuple of the id and the format dict, and that seems wrong
Also, you may have found that the transformation is always called regardless of whether or not the path succeeds up to that point, and that's why you included if item['isAvailable'] else None in the lambda function instead of just filtering. You handled it correctly here, but IMO we should avoid such complexity in transformations to prevent unintended side effects
There was a problem hiding this comment.
Can you show an example of what the POST data that we send with the API request should look like?
Of course. The POST request is:
curl -fsSL 'https://nvapi.nicovideo.jp/v1/watch/sm.../access-rights/hls?actionTrackId=...' \
-H 'x-access-right-key: ...' \
-H 'x-frontend-id: 6' \
-H 'x-frontend-version: 0' \
-H 'x-request-with: https://www.nicovideo.jp' \
--data '{"outputs": [
["video-h264-1080p","audio-aac-128kbps"],
["video-h264-720p","audio-aac-128kbps"],
["video-h264-480p","audio-aac-128kbps"],
["video-h264-360p","audio-aac-128kbps"],
["video-h264-144p","audio-aac-128kbps"]
]}' | jq .The response is:
{
"meta": {
"status": 201
},
"data": {
"contentUrl": "https://delivery.domand.nicovideo.jp/hlsbid/.../playlists/variants/....m3u8?...",
"createTime": "...",
"expireTime": "..."
}
}
Maybe I'm misreading the code, but it looks like we are sending a tuple of the
idand the format dict, and that seems wrong
Well, I'm only sending all IDs, not the format dicts.
I've split the extraction, please re-review it if you have free time.
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
|
Hi, @Dioxaz !
I opened sm43110779 in Firefox and Chrome but found both 480p and 720p are Premium only: 
Did you try opening the video in the private mode or incognito mode of your browser? |
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
|
That's interesting. Yes, I did open the video in a private window. I've just tried a few minutes ago and this is what I am getting on my side: |
Labels are "1080p", "720p", "480p", "360p" and "144p", so the "extract_video_quality" function will not work. Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
|
@Dioxaz ,
Possible! At [info] Available formats for sm43110779:
ID EXT RESOLUTION FPS │ FILESIZE TBR PROTO │ VCODEC VBR ACODEC ABR ASR MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
64 m4a audio only │ ~ 17.12MiB 69k m3u8 │ audio only aac 69k 48k Main Audio
+ 128 m4a audio only │ ~ 28.74MiB 115k m3u8 │ audio only aac 115k 48k Main Audio
h264_360p_low-aac_64kbps-hls mp4 640x360 │ ~ 90.64MiB 364k dmc │ h264 300k aac 64k 48k DMC Low-Quality HLS
h264_360p_low-aac_64kbps-http mp4 640x360 │ ~ 90.64MiB 364k dmc │ h264 300k aac 64k 48k DMC Low-Quality HTTP
+ h264_360p_low-aac_128kbps-hls mp4 640x360 │ ~106.58MiB 428k dmc │ h264 300k aac 128k 48k DMC Low-Quality HLS
+ h264_360p_low-aac_128kbps-http mp4 640x360 │ ~106.58MiB 428k dmc │ h264 300k aac 128k 48k DMC Low-Quality HTTP
153 mp4 256x144 30 │ ~ 38.15MiB 153k m3u8 │ avc1.4d401e 153k video only
h264_360p-aac_64kbps-hls mp4 640x360 │ ~165.35MiB 664k dmc │ h264 600k aac 64k 48k DMC 360p HLS
h264_360p-aac_64kbps-http mp4 640x360 │ ~165.35MiB 664k dmc │ h264 600k aac 64k 48k DMC 360p HTTP
+ h264_360p-aac_128kbps-hls mp4 640x360 │ ~181.29MiB 728k dmc │ h264 600k aac 128k 48k DMC 360p HLS
+ h264_360p-aac_128kbps-http mp4 640x360 │ ~181.29MiB 728k dmc │ h264 600k aac 128k 48k DMC 360p HTTP
614 mp4 640x360 30 │ ~152.95MiB 614k m3u8 │ avc1.4d401e 614k video only
+ 2072 mp4 854x480 60 │ ~515.90MiB 2072k m3u8 │ avc1.4d4020 2072k video only
+ 2118 mp4 1280x720 60 │ ~527.51MiB 2118k m3u8 │ avc1.4d4020 2118k video only
The DMS code also works with sm43318461 (42' 37") - ◆高機動幻想ガンパレード・マーチ 実況プレイ◆がみ千翼長のガンパレ講座② - ニコニコ. The "economy hours" rule of this video is the same as sm43110779. And I also found that not all videos are affected by that rule. For example, I'm able to get high quality streams (720p & 1080p) from sm43211773 (33' 27") - 【東方手書き劇場】クベラさん家のナズーリン - ニコニコ at ANY time. Googling "niconico economy mode" will give us more about "Niconico's economy mode hours". |
|
@bashonly , thanks for your great refactoring work! The new code does work. Today I learned:
Today I also learned:
. |
|
Yesterday I discovered: I wrote code to fix "vbr" and "tbr" and those lines have been modified during refactor. Before: # Then, correct the bitrate of all videos by minusing the minimal audio bitrate.
min_abr = traverse_obj(min(dms_audio_fmts, key=lambda fmt: fmt['abr']), ('abr'), default=0)
for i, fmt in enumerate(dms_video_fmts):
vbr = fmt['tbr'] - min_abr
dms_video_fmts[i].update({
'format_id': str(round(vbr)),
'vbr': vbr, # <---------------
'tbr': None, # <---------------
}) Now: # Calculate the true vbr/tbr by subtracting the lowest abr
min_abr = min(traverse_obj(audios, (..., 'bitRate', {float_or_none})), default=0) / 1000
for video_fmt in video_fmts:
video_fmt['tbr'] -= min_abr # <---------------------------
video_fmt['format_id'] = f'video-{video_fmt["tbr"]:.0f}'
yield video_fmtLeaving "vbr" empty and filling only "tbr" is OK. Our code will calculate bitrates: Lines 5392 to 5402 in aa13a8e . |
|
I'm Premium user. yt-dlp_master(2024/03/01) = NG yt-dlp_master(2024/02/23) + #8685 patch = OK |
|
Hi, @betsu0 !
Thanks for your detailed feedback. I'll go find a Niconico Premium account to perform test. |
|
yt-dlp/yt_dlp/extractor/niconico.py Lines 481 to 489 in 6ad11fe This part is wrong. You don't need to join/follow the "club" to watch these videos. (You don't even need premium account, either; this video can be viewed as a guest as soon as you have legit Japanese IP.) |
Thanks. IIRC this is the test for members-only videos in niconico fanclub. Do you happen to know the correct way to handle both Premium videos and fanclub's ones? |
|
Can't we just check if api_data["media"] exists or not? I cannot find any Premium-only content, but this one https://www.nicovideo.jp/watch/so43461409 is gated for the members. And it returns "media": {
"domand": null,
"delivery": null,
"deliveryLegacy": null
}, |
|
Edit: we can also try reverse-engineering https://resource.video.nimg.jp/web/scripts/pages/watch/modern/watch_app_5fd9b82ecc8c93f40019.js if we want to match the reasons given by Nico's official frontend. |
Closes yt-dlp#8389, Closes yt-dlp#8758, Closes yt-dlp#9254 Authored by: pzhlkj6612, xpadev-net
IMPORTANT: PRs without the template will be CLOSED
Description of your pull request and other information
This PR cannot be finished without @xpadev-net's work. See #8685 for more details.
Fixes #8389, resolves #8758, fixes #9254
Because of the download method, using DMS may help us avoid the speed throttle of DMC (#7365).
The DMS service is still in the test stage. Test videos:
Call for help
How to make sure the output
dictfromtraverse_obj()must contain all specified keys? Then I can get values from thedictwithout testing each key.Template
Before submitting a pull request make sure you have:
In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:
What is the purpose of your pull request?