Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nicovideo.jp issue #29225

Closed
5 tasks done
johninaba opened this issue Jun 6, 2021 · 9 comments · Fixed by #30329
Closed
5 tasks done

nicovideo.jp issue #29225

johninaba opened this issue Jun 6, 2021 · 9 comments · Fixed by #30329

Comments

@johninaba
Copy link

johninaba commented Jun 6, 2021

Checklist

  • I'm reporting a broken site support
  • I've verified that I'm running youtube-dl version 2021.06.06
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar issues including closed ones

Verbose log

C:\Users\XXXXX>"C:\Users\XXXXX\Desktop\Video Tools\Youtube DL\youtube-dl.exe" --verbose  https://www.nicovideo.jp/watch/sm12134023
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.nicovideo.jp/watch/sm12134023']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev, ffprobe 4.4-full_build-www.gyan.dev
[debug] Proxy map: {}
[niconico] sm12134023: Downloading webpage
[niconico] sm12134023: Downloading flv info
ERROR: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 291, in _real_extract
KeyError: 'smileInfo'
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 291, in _real_extract
KeyError: 'smileInfo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 310, in _real_extract
youtube_dl.utils.ExtractorError: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Unable to download from the above website. Related bugs suggest problems with downloading speeds or requiring a login. Neither of those issues appear to be related to the problem I am having.

@marinarasub
Copy link

same issue here

@CoffeeMuffin
Copy link

CoffeeMuffin commented Jun 7, 2021

  • I've verified that I'm running youtube-dl version 2021.06.06
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped

same issue. different log.

a@c:~$ python3 youtube-dl -u "mylogin" -p "mypassword" "http://www.nicovideo.jp/watch/sm37944844" --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', 'http://www.nicovideo.jp/watch/sm37944844', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.8.5 (CPython) - Linux-5.10.0-1029-oem-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm37944844: Downloading webpage
[niconico] sm37944844: Downloading flv info
ERROR: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "youtube-dl/youtube_dl/extractor/niconico.py", line 291, in _real_extract
    video_real_url = api_data['video']['smileInfo']['url']
KeyError: 'smileInfo'
Traceback (most recent call last):
  File "youtube-dl/youtube_dl/extractor/niconico.py", line 291, in _real_extract
    video_real_url = api_data['video']['smileInfo']['url']
KeyError: 'smileInfo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "youtube-dl/youtube_dl/extractor/niconico.py", line 310, in _real_extract
    raise ExtractorError('Unable to find video URL')
youtube_dl.utils.ExtractorError: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

@obskyr
Copy link

obskyr commented Jun 14, 2021

It seems that the API has changed – the smileInfo key no longer exists, and in fact, no flat-out URL appears at all in the API response… Could it be that the video URL is nowadays being derived from this part somehow?

'urls': [{'isSsl': True,
        'isWellKnownPort': True,
        'url': 'https://api.dmc.nico/api/sessions'}],
'videos': ['archive_h264_360p',
            'archive_h264_360p_low'],

That's in api_data['media']['delivery']['movie']['session'].

@obskyr
Copy link

obskyr commented Jun 14, 2021

I've figured the issue out – apparently, Niconico have been recombobulating their API quite a bit lately, including switching from snake case to camel case. Here's a diff patch that makes videos of this type work:

@@ -195,7 +195,7 @@
         def yesno(boolean):
             return 'yes' if boolean else 'no'

-        session_api_data = api_data['video']['dmcInfo']['session_api']
+        session_api_data = api_data['media']['delivery']['movie']['session']
         session_api_endpoint = session_api_data['urls'][0]

         format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
@@ -208,15 +208,15 @@
             data=json.dumps({
                 'session': {
                     'client_info': {
-                        'player_id': session_api_data['player_id'],
+                        'player_id': session_api_data['playerId'],
                     },
                     'content_auth': {
-                        'auth_type': session_api_data['auth_types'][session_api_data['protocols'][0]],
-                        'content_key_timeout': session_api_data['content_key_timeout'],
+                        'auth_type': session_api_data['authTypes'][session_api_data['protocols'][0]],
+                        'content_key_timeout': session_api_data['contentKeyTimeout'],
                         'service_id': 'nicovideo',
-                        'service_user_id': session_api_data['service_user_id']
+                        'service_user_id': session_api_data['serviceUserId']
                     },
-                    'content_id': session_api_data['content_id'],
+                    'content_id': session_api_data['contentId'],
                     'content_src_id_sets': [{
                         'content_src_ids': [{
                             'src_id_to_mux': {
@@ -229,7 +229,7 @@
                     'content_uri': '',
                     'keep_method': {
                         'heartbeat': {
-                            'lifetime': session_api_data['heartbeat_lifetime']
+                            'lifetime': session_api_data['heartbeatLifetime']
                         }
                     },
                     'priority': session_api_data['priority'],
@@ -239,14 +239,14 @@
                             'http_parameters': {
                                 'parameters': {
                                     'http_output_download_parameters': {
-                                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
-                                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                                        'use_ssl': yesno(session_api_endpoint['isSsl']),
+                                        'use_well_known_port': yesno(session_api_endpoint['isWellKnownPort']),
                                     }
                                 }
                             }
                         }
                     },
-                    'recipe_id': session_api_data['recipe_id'],
+                    'recipe_id': session_api_data['recipeId'],
                     'session_operation_auth': {
                         'session_operation_auth_by_signature': {
                             'signature': session_api_data['signature'],
@@ -288,7 +288,8 @@
             return 'economy' if video_real_url.endswith('low') else 'normal'

         try:
-            video_real_url = api_data['video']['smileInfo']['url']
+            # video_real_url = api_data['video']['smileInfo']['url']
+            video_real_url = None
         except KeyError:  # Flash videos
             # Get flv info
             flv_info_webpage = self._download_webpage(
@@ -335,12 +336,11 @@
         else:
             formats = []

-            dmc_info = api_data['video'].get('dmcInfo')
+            dmc_info = api_data['media']['delivery']['movie']
             if dmc_info:  # "New" HTML5 videos
-                quality_info = dmc_info['quality']
-                for audio_quality in quality_info['audios']:
-                    for video_quality in quality_info['videos']:
-                        if not audio_quality['available'] or not video_quality['available']:
+                for audio_quality in dmc_info['audios']:
+                    for video_quality in dmc_info['videos']:
+                        if not audio_quality['isAvailable'] or not video_quality['isAvailable']:
                             continue
                         formats.append(self._extract_format_for_quality(
                             api_data, video_id, audio_quality, video_quality))

I'm hesitant to make a pull request – as it stands, this almost certainly ruins the ability to download other types of videos (like FLV videos, for example). I'll leave the formal fix to someone who's more familiar with Niconico's inner workings. (Does the FLV video endpoint even still work? Does the “old HTML5 video” method still exist? From what I can tell in the API, it might be the case that the DMC approach is the only one at this point, although I may very well be wrong about that.) Here's the fix, either way!

@Polsy
Copy link

Polsy commented Jun 18, 2021

I'm 99.9% sure that FLV videos at least are entirely dead. 10+ year old videos are served as video/MP2T now when they were certainly flv at the time (this also goes for swf videos, which used to be interactive and now are not). I'd take most videos working over none, in any case, so I'll use this patch for now - thanks!

@no1d
Copy link

no1d commented Jun 21, 2021

Thanks for the patch! Required a bit more fixing though.
See here no1d@e493896

@Jorge9811
Copy link

Also getting this issue.

C:\Users\Jorge>ytdl -f best "https://www.nicovideo.jp/watch/sm36773221" -o "C:\Users\Jorge\Desktop\%(title)s.%(ext)s" --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-f', 'best', 'https://www.nicovideo.jp/watch/sm36773221', '-o', 'C:\\Users\\Jorge\\Desktop\\%(title)s.%(ext)s', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.2
[debug] Proxy map: {}
[niconico] sm36773221: Downloading webpage
[niconico] sm36773221: Downloading flv info
ERROR: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 291, in _real_extract
KeyError: 'smileInfo'
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 291, in _real_extract
KeyError: 'smileInfo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\niconico.py", line 310, in _real_extract
youtube_dl.utils.ExtractorError: Unable to find video URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.```

@VarLad
Copy link

VarLad commented Jun 25, 2021

Same error here

@CXwudi
Copy link

CXwudi commented Jul 25, 2021

For downloading nicovideo.jp, I recommend using one of the following alternatives (this list is copied from my another abandoned project readme file):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants