[extractor,downloader/niconico] support new delivery server called Dowango Media Service(DMS) #8685
+104
−26
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xpadev-net
changed the title
pukkandan
reviewed
Nov 30, 2023
pukkandan
added
enhancement
|
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
11 tasks
Reduce confusion and make the usage consistent throughout the file.
pzhlkj6612
suggested changes
Feb 22, 2024
pzhlkj6612
suggested changes
Feb 22, 2024
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
[extractor/niconico] refactor: call float_or_none() with kwargs
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
|
A new question: Do we really need the niconico DMS downloader? For DMS, the current implementation gives us the following formats: If we directly parse the m3u8 playlist from Advantages:
Doubtful points:
Test video: sm43273809 (56") - 【完結編】映画『ラブライブ！虹ヶ咲学園スクールアイドル同好会 完結編 第1章』【特報】 - ニコニコ My implementaion: from ..utils import (
parse_bitrate,
remove_end,
)
class NiconicoIE(InfoExtractor):
def _real_extract(self, url):
# api_data obtained
formats = []
# DMC stuff
dms_data = traverse_obj(api_data, ({
'videos': ('media', 'domand', 'videos', ..., {
lambda item: item['id'] if item['isAvailable'] else None
}),
'audios': ('media', 'domand', 'audios', ..., {
lambda item: (item['id'], {
'format_id': str(parse_bitrate(item['id'])),
'abr': float_or_none(item['bitRate'], scale=1000),
'asr': item['samplingRate'],
'acodec': 'aac',
'ext': 'm4a',
}) if item['isAvailable'] else None
}),
'accessRightKey': ('media', 'domand', 'accessRightKey'),
'track_id': ('client', 'watchTrackId'),
}, {
'videos': 'videos',
'audios': ('audios', {lambda items: dict(items)}),
'accessRightKey': 'accessRightKey',
'track_id': 'track_id',
}))
dms_m3u8_url = traverse_obj(self._download_json(
f'https://nvapi.nicovideo.jp/v1/watch/{video_id}/access-rights/hls', video_id,
data=json.dumps({
'outputs': list(itertools.product(dms_data['videos'], dms_data['audios']))
}).encode(), query={'actionTrackId': dms_data['track_id']}, headers={
'x-access-right-key': dms_data['accessRightKey'],
'x-frontend-id': 6,
'x-frontend-version': 0,
'x-request-with': 'https://www.nicovideo.jp',
}), ('data', 'contentUrl'))
dms_fmts = self._extract_m3u8_formats(dms_m3u8_url, video_id)
# update audio formats
dms_audio_fmts = [fmt for fmt in dms_fmts if fmt['vcodec'] == 'none']
for i, fmt in enumerate(dms_audio_fmts):
format_id = remove_end(fmt['format_id'], '-%s' % fmt['format_note'])
dms_audio_fmts[i].update(**(traverse_obj(dms_data, ('audios', format_id, {dict})) or {}))
# remove duplicate video formats
dms_video_fmts = [
list(fmts)[0] for _, fmts in itertools.groupby(sorted([
fmt for fmt in dms_fmts if fmt['vcodec'] != 'none'
], key=lambda fmt: fmt['tbr']), lambda fmt: fmt['url'])
]
# correct video bitrate
min_abr = traverse_obj(min(dms_audio_fmts, key=lambda fmt: fmt['abr']), ('abr'), default=0)
for i, fmt in enumerate(dms_video_fmts):
vbr = fmt['tbr'] - min_abr
dms_video_fmts[i].update({
'format_id': str(round(vbr)),
'vbr': vbr,
'tbr': None,
})
formats.extend(dms_video_fmts + dms_audio_fmts)
# Start extracting information. |
|
@pzhlkj6612 Could you make a new PR with it? Even if it is inferior to DMS, we can get a extractor-only solution reviewed and merged much easier than the DMS one (which edits downloader). This PR can still be discussed, but that could act as an immediate solution |
9 tasks
|
@pukkandan Done, see #9282. |
pzhlkj6612
suggested changes
Feb 27, 2024
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
|
@xpadev-net Thank you for your work on this, you have received/will receive credit in the commit and changelog |
pzhlkj6612
reviewed
Mar 6, 2024
There was a problem hiding this comment.
Although this PR has been closed, I still found something valuable. 💎
To @xpadev-net : please do not apply any suggestions, go open new PR instead.
Comment on lines
+450
to
+452
| api_data = self._download_json( | ||
| 'https://www.nicovideo.jp/api/watch/v3_guest/%s?_frontendId=6&_frontendVersion=0&actionTrackId=AAAAAAAAAA_%d' % (video_id, round(time.time() * 1000)), video_id, | ||
| note='Downloading API JSON', errnote='Unable to fetch data')['data'] |
There was a problem hiding this comment.
Suggestion
Suggested change
| api_data = self._download_json( | |
| 'https://www.nicovideo.jp/api/watch/v3_guest/%s?_frontendId=6&_frontendVersion=0&actionTrackId=AAAAAAAAAA_%d' % (video_id, round(time.time() * 1000)), video_id, | |
| note='Downloading API JSON', errnote='Unable to fetch data')['data'] | |
| api_data = self._download_json( | |
| 'https://www.nicovideo.jp/api/watch/v3_guest/%s?_frontendId=6&_frontendVersion=0&actionTrackId=AAAAAAAAAA_%d' % (video_id, round(time.time() * 1000)), video_id, | |
| note='Downloading API JSON', errnote='Unable to fetch data', expected_status=(400))['data'] |
This API will return HTTP 400 but some info of the video.
Why am I here
In #9338 (comment) , I couldn't get the "api_data" JSON from some videos. No json means no metadata. The "v3_guest" API solved my problem.
As a guest, I tested anime episode so43309069 - 戦国妖狐 世直し姉弟編 #3「永禄七年」 アニメ/動画 - ニコニコ動画 without valid Cookies:
VIDEO_ID='so43309069'
curl --silent --get \
--data '_frontendId=6' \
--data '_frontendVersion=0' \
--data "actionTrackId=$(mktemp -u XXXXXXXXXX)_$(date '+%s')000" \
"https://www.nicovideo.jp/api/watch/v3_guest/${VIDEO_ID}" | \
jq '
{
"meta": .meta,
"data": {
"reason": .data.reasonCode,
"duration": .data.data.video.duration,
"comment_count": .data.data.video.count.comment
}
}
'
With Japan IP:
{
"meta": {
"status": 400,
"errorCode": "FORBIDDEN"
},
"data": {
"reason": "PPV_VIDEO",
"duration": null,
"comment_count": null
}
}
Without Japan IP:
{
"meta": {
"status": 400,
"errorCode": "FORBIDDEN"
},
"data": {
"reason": "DOMESTIC_VIDEO",
"duration": 1419,
"comment_count": 3830
}
}
"DOMESTIC_VIDEO" means geo-restriction. "PPV_VIDEO" means "premium_only".
For members-only videos in fanclubs:
{
"meta": {
"status": 400,
"errorCode": "FORBIDDEN"
},
"data": {
"reason": "PPV_OR_CHANNEL_MEMBER_VIDEO",
"duration": null,
"comment_count": null
}
}
{
"meta": {
"status": 400,
"errorCode": "FORBIDDEN"
},
"data": {
"reason": "CHANNEL_MEMBER_ONLY",
"duration": null,
"comment_count": null
}
}
In #9338 (comment) , I wrote :
The fanclub of so43204345 might have mis-configured its payment method. I didn't see a PPV option, but only the "join" button.
Hence, both "PPV_OR_CHANNEL_MEMBER_VIDEO" and "CHANNEL_MEMBER_ONLY" mean "subscriber_only".
Conclusion
With the "v3_guest" API, we are able to extract more information. I think @xpadev-net should add this to the niconico IE by opening a new PR.
Comment on lines
+498
to
+499
| actual_video_id = traverse_obj(api_data, ('video', 'id')) | ||
|
|
There was a problem hiding this comment.
Is this necessary? Did you encounter some problems with the video id?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #9254
Support for downloading videos from the Dowango Media Service(DMS), which has been in use since November.
Dwango has announced that the old server (DMC) will be migrated to the new server (DMS) in stages.
ref: https://blog.nicovideo.jp/niconews/205042.html
For more information about this server, please refer to the following information
tor4kichi/Hohoema#961
Nico Nico Douga's new delivery server "domand" memo for 2023(ニコニコ動画の2023年新配信サーバー「domand」メモ) | scrapbox.io
Template
Before submitting a pull request make sure you have:
In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:
What is the purpose of your pull request?