[niconico] Downloading long videos requires heartbeat signals

[Ristellise]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like this: [x])
• Use the Preview tab to see what your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.10.20. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.10.20

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

Python Console:
youtube_dl.YoutubeDL({'username': "username",'password': 'password','verbose':True}).extract_info(url="http://www.nicovideo.jp/watch/sm29409391")

[debug] Encodings: locale cp1252, fs utf-8, out UTF-8, pref cp1252
[debug] youtube-dl version 2017.10.20
[debug] Python version 3.6.2 - Windows-10-10.0.15063-SP0
[debug] exe versions: ffmpeg 3.3.3, ffprobe 3.3.3
[debug] Proxy map: {}
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 795, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 849, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1619, in process_video_result
    self.process_info(new_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\http.py", line 297, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 163, in report_error
    self.ydl.report_error(*args, **kargs)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 612, in report_error
    self.trouble(error_message, tb)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 582, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: giving up after 0 retries

Description of your issue, suggested solution and other information

As of today, Nicovideo's seem to implemented a new session system. After every few minutes, the url of the video changes. This causes a 403 Forbidden error on the old url, Causing the Downloader to fail.
A screenshot of part of a video session:

Resuming the download works just for a few minutes before the session is dropped.
Note: I have only tested on a non-premium account. Im not sure if it applies to the permium accounts or not.

[Ristellise]

If you want to reporduce, download any video from niconico. (You do need a username and password from a registed account.)

Let it download. it will work for a while(approximately 3minutes ore more before the link changes... then it errors out.)

[f11894]

Nico Nico Douga seems that access will be disconnected unless a heartbeat signal is sent regularly.
ニコニコ動画は定期的にハートビート信号を送らないとアクセスが切断されてしまうようです。

https://qiita.com/tor4kichi/items/91550a71119f3878bfba

[yan12125]

I added that flag as I can still download full videos and the downloaded file appears fine. Tried both two days ago and just now:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm29409391', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm29409391: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://smile-fnl21.nicovideo.jp/smile?m=29409391.67401'
[download] Destination: テイルズオブベルセリア OP FULL 「BURN」／FLOW-sm29409391.mp4
[download] 100% of 13.56MiB in 00:02

I guess 2 seconds is shorter than the period of heartbeat signals. Do you have longer examples?

[f11894]

Please try with this URL.
このURLで試してください。

http://www.nicovideo.jp/watch/sm31807439

[yan12125]

Thanks! I can reproduce the error now

$ python -m youtube_dl -v "http://www.nicovideo.jp/watch/sm31807439" -u yan12125@gmail.com
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm31807439', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm31807439: Downloading webpage
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://pb02.dmc.nico:2807/vod/ht2_nicovideo/nicovideo-sm31807439_8e75a000d8b99f43ffb7b843d9a9a72fd4711638418e2af8b574cf48f8fad073?ht2_nicovideo=48110477.823ysw_oyjxl8_jlzjnlmfy8gt'
[download] Destination: 【厳選】奇妙ゲーム集FINAL【PS1】-sm31807439.mp4
[download]  22.8% of 425.49MiB at 844.20KiB/s ETA 06:38[download] Got server HTTP error: Downloaded 101711872 bytes, expected 446155552 bytes. Retrying (attempt 1 of 10)...
[download] Resuming download at byte 101711872
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 286, in real_download
    establish_connection()
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 74, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 2195, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

[Ristellise]

any update on this?

[awei78]

try this way:
in youtube_dl\extractor\niconico.py, get heartbeat datas and execute heart-beat:

def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
    ...
    session_response = self._download_json(
        session_api_endpoint['url'], video_id,
        query={'_format': 'json'},
        headers={'Content-Type': 'application/json'},
        note='Downloading JSON metadata for %s' % format_id,
        data=...)

        # get heart-beat datas
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        hb_data = json.dumps(session_response['data'])

        # send heart-beat command
        def heart_beat():
            import urllib2
            try:
                urllib2.urlopen(url=api_url, data=hb_data)
                print '........................................heart beat!'
            except Exception as ex:
                print '........................................heart beat fail: ' + ex.message

            # need a condition to end timer
            if !downloaded:
                timer = threading.Timer(25, heart_beat)
                timer.start()

        heart_beat()

        resolution = video_quality.get('resolution', {})
        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
        }

enjoy it!

[ciffelia]

@awei78 Could you create pull request for this?

[awei78]

@prince0203 I am sorry for my poor English. I think that i can not create it, but I can provide a solution:

1. extend NiconicoIE:

from ..extractor.niconico import NiconicoIE as InconicoBase
...
class NiconicoIE(InconicoBase):
    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
        ...
                # get heart-beat data
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        data = json.dumps(session_response['data'])
        resolution = video_quality.get('resolution', {})

        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
            'heartbeat_url': api_url,  # pay attention here
            'heartbeat_data': data,
        }

2. extend HttpFD:

from http import HttpFD

class HttpHB(HttpFD):
    def real_download(self, filename, info_dict):
        result = False
        if 'heartbeat_url'in info_dict:
            def heart_beat():
                try:
                    urllib2.urlopen(url=info_dict['heartbeat_url'], data=info_dict['heartbeat_data'])
                    print '........................................heart beat!'
                except Exception as ex:
                    print '........................................heart beat fail: ' + ex.message
                    pass

                if not result:
                    timer = threading.Timer(25, heart_beat)
                    timer.start()

            heart_beat()

        result = super(HttpHB, self).real_download(filename, info_dict)
        return result

3. check heartbeat_url sign and use HttpHB to download:

            if 'heartbeat_url' in info:
                fd = HttpHB(self._ydl, params)
            ...

[na0x2c6]

@yan12125
How is this issue status?
It seems that panhartstuff/youtube-dl@ff19685 works well for me.
I have been able to download videos which take over an hour video with this patch.

Thanks.

[yan12125]

Sorry, but I'm no longer working on youtube-dl. Current developers will determine whether any of pending pull requests (#18230, #16495) are good enough to merge or not. Or, if neither works, you'll need to open a new pull request.

[na0x2c6]

@yan12125
Thank you for your reply !!

@dstftw @remitamine

Current developers will determine whether any of pending pull requests (#18230, #16495) are good enough to merge or not. Or, if neither works, you'll need to open a new pull request.

Are "Current developers" you two?
Thank you for developing the awesome tool.
How are these pull request ?

I'm sorry if my poor english were rude expression...

[na0x2c6]

@dstftw @remitamine
Hi,
How is this issue status ?

[jxu]

Workaround is to keep running command using loop #19261 (comment)

[TheCardinalSystem]

Issue status? Is there a fix for users using the windows executable?