New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[niconico] Downloading long videos requires heartbeat signals #14582

Open
Ristellise opened this Issue Oct 25, 2017 · 9 comments

Comments

Projects
None yet
5 participants
@Ristellise

Ristellise commented Oct 25, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.10.20. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.10.20

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

Python Console:
youtube_dl.YoutubeDL({'username': "username",'password': 'password','verbose':True}).extract_info(url="http://www.nicovideo.jp/watch/sm29409391")

[debug] Encodings: locale cp1252, fs utf-8, out UTF-8, pref cp1252
[debug] youtube-dl version 2017.10.20
[debug] Python version 3.6.2 - Windows-10-10.0.15063-SP0
[debug] exe versions: ffmpeg 3.3.3, ffprobe 3.3.3
[debug] Proxy map: {}
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 795, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 849, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1619, in process_video_result
    self.process_info(new_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\http.py", line 297, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 163, in report_error
    self.ydl.report_error(*args, **kargs)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 612, in report_error
    self.trouble(error_message, tb)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 582, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: giving up after 0 retries

Description of your issue, suggested solution and other information

As of today, Nicovideo's seem to implemented a new session system. After every few minutes, the url of the video changes. This causes a 403 Forbidden error on the old url, Causing the Downloader to fail.
A screenshot of part of a video session:
chrome 2017-10-25 10 43 35 396

Resuming the download works just for a few minutes before the session is dropped.
Note: I have only tested on a non-premium account. Im not sure if it applies to the permium accounts or not.

@Ristellise

This comment has been minimized.

Show comment
Hide comment
@Ristellise

Ristellise Oct 26, 2017

If you want to reporduce, download any video from niconico. (You do need a username and password from a registed account.)

Let it download. it will work for a while(approximately 3minutes ore more before the link changes... then it errors out.)

Ristellise commented Oct 26, 2017

If you want to reporduce, download any video from niconico. (You do need a username and password from a registed account.)

Let it download. it will work for a while(approximately 3minutes ore more before the link changes... then it errors out.)

@f11894

This comment has been minimized.

Show comment
Hide comment
@f11894

f11894 Oct 28, 2017

Nico Nico Douga seems that access will be disconnected unless a heartbeat signal is sent regularly.
ニコニコ動画は定期的にハートビート信号を送らないとアクセスが切断されてしまうようです。

https://qiita.com/tor4kichi/items/91550a71119f3878bfba

f11894 commented Oct 28, 2017

Nico Nico Douga seems that access will be disconnected unless a heartbeat signal is sent regularly.
ニコニコ動画は定期的にハートビート信号を送らないとアクセスが切断されてしまうようです。

https://qiita.com/tor4kichi/items/91550a71119f3878bfba

@yan12125

This comment has been minimized.

Show comment
Hide comment
@yan12125

yan12125 Oct 28, 2017

Collaborator

I added that flag as I can still download full videos and the downloaded file appears fine. Tried both two days ago and just now:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm29409391', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm29409391: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://smile-fnl21.nicovideo.jp/smile?m=29409391.67401'
[download] Destination: テイルズオブベルセリア OP FULL 「BURN」/FLOW-sm29409391.mp4
[download] 100% of 13.56MiB in 00:02

I guess 2 seconds is shorter than the period of heartbeat signals. Do you have longer examples?

Collaborator

yan12125 commented Oct 28, 2017

I added that flag as I can still download full videos and the downloaded file appears fine. Tried both two days ago and just now:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm29409391', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm29409391: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://smile-fnl21.nicovideo.jp/smile?m=29409391.67401'
[download] Destination: テイルズオブベルセリア OP FULL 「BURN」/FLOW-sm29409391.mp4
[download] 100% of 13.56MiB in 00:02

I guess 2 seconds is shorter than the period of heartbeat signals. Do you have longer examples?

@f11894

This comment has been minimized.

Show comment
Hide comment
@f11894

f11894 Oct 28, 2017

Please try with this URL.
このURLで試してください。

http://www.nicovideo.jp/watch/sm31807439

f11894 commented Oct 28, 2017

Please try with this URL.
このURLで試してください。

http://www.nicovideo.jp/watch/sm31807439

@yan12125

This comment has been minimized.

Show comment
Hide comment
@yan12125

yan12125 Oct 28, 2017

Collaborator

Thanks! I can reproduce the error now

$ python -m youtube_dl -v "http://www.nicovideo.jp/watch/sm31807439" -u yan12125@gmail.com
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm31807439', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm31807439: Downloading webpage
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://pb02.dmc.nico:2807/vod/ht2_nicovideo/nicovideo-sm31807439_8e75a000d8b99f43ffb7b843d9a9a72fd4711638418e2af8b574cf48f8fad073?ht2_nicovideo=48110477.823ysw_oyjxl8_jlzjnlmfy8gt'
[download] Destination: 【厳選】奇妙ゲーム集FINAL【PS1】-sm31807439.mp4
[download]  22.8% of 425.49MiB at 844.20KiB/s ETA 06:38[download] Got server HTTP error: Downloaded 101711872 bytes, expected 446155552 bytes. Retrying (attempt 1 of 10)...
[download] Resuming download at byte 101711872
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 286, in real_download
    establish_connection()
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 74, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 2195, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
Collaborator

yan12125 commented Oct 28, 2017

Thanks! I can reproduce the error now

$ python -m youtube_dl -v "http://www.nicovideo.jp/watch/sm31807439" -u yan12125@gmail.com
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm31807439', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm31807439: Downloading webpage
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://pb02.dmc.nico:2807/vod/ht2_nicovideo/nicovideo-sm31807439_8e75a000d8b99f43ffb7b843d9a9a72fd4711638418e2af8b574cf48f8fad073?ht2_nicovideo=48110477.823ysw_oyjxl8_jlzjnlmfy8gt'
[download] Destination: 【厳選】奇妙ゲーム集FINAL【PS1】-sm31807439.mp4
[download]  22.8% of 425.49MiB at 844.20KiB/s ETA 06:38[download] Got server HTTP error: Downloaded 101711872 bytes, expected 446155552 bytes. Retrying (attempt 1 of 10)...
[download] Resuming download at byte 101711872
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 286, in real_download
    establish_connection()
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 74, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 2195, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

@yan12125 yan12125 changed the title from NicoVideos Dont work anymore to [niconico] Downloading long videos requires heartbeat signals Oct 28, 2017

@yan12125 yan12125 added the bug label Oct 28, 2017

@yayugu yayugu referenced this issue Nov 8, 2017

Open

ニコ動対応 #49

@Ristellise

This comment has been minimized.

Show comment
Hide comment
@Ristellise

Ristellise Nov 16, 2017

any update on this?

Ristellise commented Nov 16, 2017

any update on this?

@awei78

This comment has been minimized.

Show comment
Hide comment
@awei78

awei78 Jan 26, 2018

try this way:
in youtube_dl\extractor\niconico.py, get heartbeat datas and execute heart-beat:

def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
    ...
    session_response = self._download_json(
        session_api_endpoint['url'], video_id,
        query={'_format': 'json'},
        headers={'Content-Type': 'application/json'},
        note='Downloading JSON metadata for %s' % format_id,
        data=...)

        # get heart-beat datas
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        hb_data = json.dumps(session_response['data'])

        # send heart-beat command
        def heart_beat():
            import urllib2
            try:
                urllib2.urlopen(url=api_url, data=hb_data)
                print '........................................heart beat!'
            except Exception as ex:
                print '........................................heart beat fail: ' + ex.message

            # need a condition to end timer
            if !downloaded:
                timer = threading.Timer(25, heart_beat)
                timer.start()

        heart_beat()

        resolution = video_quality.get('resolution', {})
        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
        }

enjoy it!

awei78 commented Jan 26, 2018

try this way:
in youtube_dl\extractor\niconico.py, get heartbeat datas and execute heart-beat:

def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
    ...
    session_response = self._download_json(
        session_api_endpoint['url'], video_id,
        query={'_format': 'json'},
        headers={'Content-Type': 'application/json'},
        note='Downloading JSON metadata for %s' % format_id,
        data=...)

        # get heart-beat datas
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        hb_data = json.dumps(session_response['data'])

        # send heart-beat command
        def heart_beat():
            import urllib2
            try:
                urllib2.urlopen(url=api_url, data=hb_data)
                print '........................................heart beat!'
            except Exception as ex:
                print '........................................heart beat fail: ' + ex.message

            # need a condition to end timer
            if !downloaded:
                timer = threading.Timer(25, heart_beat)
                timer.start()

        heart_beat()

        resolution = video_quality.get('resolution', {})
        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
        }

enjoy it!

@prince0203

This comment has been minimized.

Show comment
Hide comment
@prince0203

prince0203 Jan 30, 2018

@awei78 Could you create pull request for this?

prince0203 commented Jan 30, 2018

@awei78 Could you create pull request for this?

@awei78

This comment has been minimized.

Show comment
Hide comment
@awei78

awei78 Feb 3, 2018

@prince0203 I am sorry for my poor English. I think that i can not create it, but I can provide a solution:

  1. extend NiconicoIE:
from ..extractor.niconico import NiconicoIE as InconicoBase
...
class NiconicoIE(InconicoBase):
    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
        ...
                # get heart-beat data
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        data = json.dumps(session_response['data'])
        resolution = video_quality.get('resolution', {})

        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
            'heartbeat_url': api_url,  # pay attention here
            'heartbeat_data': data,
        }
  1. extend HttpFD:
from http import HttpFD

class HttpHB(HttpFD):
    def real_download(self, filename, info_dict):
        result = False
        if 'heartbeat_url'in info_dict:
            def heart_beat():
                try:
                    urllib2.urlopen(url=info_dict['heartbeat_url'], data=info_dict['heartbeat_data'])
                    print '........................................heart beat!'
                except Exception as ex:
                    print '........................................heart beat fail: ' + ex.message
                    pass

                if not result:
                    timer = threading.Timer(25, heart_beat)
                    timer.start()

            heart_beat()

        result = super(HttpHB, self).real_download(filename, info_dict)
        return result
  1. check heartbeat_url sign and use HttpHB to download:
            if 'heartbeat_url' in info:
                fd = HttpHB(self._ydl, params)
            ...

awei78 commented Feb 3, 2018

@prince0203 I am sorry for my poor English. I think that i can not create it, but I can provide a solution:

  1. extend NiconicoIE:
from ..extractor.niconico import NiconicoIE as InconicoBase
...
class NiconicoIE(InconicoBase):
    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
        ...
                # get heart-beat data
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        data = json.dumps(session_response['data'])
        resolution = video_quality.get('resolution', {})

        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
            'heartbeat_url': api_url,  # pay attention here
            'heartbeat_data': data,
        }
  1. extend HttpFD:
from http import HttpFD

class HttpHB(HttpFD):
    def real_download(self, filename, info_dict):
        result = False
        if 'heartbeat_url'in info_dict:
            def heart_beat():
                try:
                    urllib2.urlopen(url=info_dict['heartbeat_url'], data=info_dict['heartbeat_data'])
                    print '........................................heart beat!'
                except Exception as ex:
                    print '........................................heart beat fail: ' + ex.message
                    pass

                if not result:
                    timer = threading.Timer(25, heart_beat)
                    timer.start()

            heart_beat()

        result = super(HttpHB, self).real_download(filename, info_dict)
        return result
  1. check heartbeat_url sign and use HttpHB to download:
            if 'heartbeat_url' in info:
                fd = HttpHB(self._ydl, params)
            ...

@nekojun nekojun referenced this issue May 19, 2018

Open

[niconico] use HeartBeat #16495

5 of 9 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment