Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[niconico] Downloading long videos requires heartbeat signals #14582

Open
Ristellise opened this issue Oct 25, 2017 · 14 comments · May be fixed by #18230
Open

[niconico] Downloading long videos requires heartbeat signals #14582

Ristellise opened this issue Oct 25, 2017 · 14 comments · May be fixed by #18230
Labels
bug

Comments

@Ristellise
Copy link

@Ristellise Ristellise commented Oct 25, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.10.20. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.10.20

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

Python Console:
youtube_dl.YoutubeDL({'username': "username",'password': 'password','verbose':True}).extract_info(url="http://www.nicovideo.jp/watch/sm29409391")

[debug] Encodings: locale cp1252, fs utf-8, out UTF-8, pref cp1252
[debug] youtube-dl version 2017.10.20
[debug] Python version 3.6.2 - Windows-10-10.0.15063-SP0
[debug] exe versions: ffmpeg 3.3.3, ffprobe 3.3.3
[debug] Proxy map: {}
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 795, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 849, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1619, in process_video_result
    self.process_info(new_info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\http.py", line 297, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\downloader\common.py", line 163, in report_error
    self.ydl.report_error(*args, **kargs)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 612, in report_error
    self.trouble(error_message, tb)
  File "C:\Users\joshw\AppData\Local\Programs\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 582, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: giving up after 0 retries

Description of your issue, suggested solution and other information

As of today, Nicovideo's seem to implemented a new session system. After every few minutes, the url of the video changes. This causes a 403 Forbidden error on the old url, Causing the Downloader to fail.
A screenshot of part of a video session:
chrome 2017-10-25 10 43 35 396

Resuming the download works just for a few minutes before the session is dropped.
Note: I have only tested on a non-premium account. Im not sure if it applies to the permium accounts or not.

@Ristellise
Copy link
Author

@Ristellise Ristellise commented Oct 26, 2017

If you want to reporduce, download any video from niconico. (You do need a username and password from a registed account.)

Let it download. it will work for a while(approximately 3minutes ore more before the link changes... then it errors out.)

@f11894
Copy link

@f11894 f11894 commented Oct 28, 2017

Nico Nico Douga seems that access will be disconnected unless a heartbeat signal is sent regularly.
ニコニコ動画は定期的にハートビート信号を送らないとアクセスが切断されてしまうようです。

https://qiita.com/tor4kichi/items/91550a71119f3878bfba

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 28, 2017

I added that flag as I can still download full videos and the downloaded file appears fine. Tried both two days ago and just now:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm29409391', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm29409391: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://smile-fnl21.nicovideo.jp/smile?m=29409391.67401'
[download] Destination: テイルズオブベルセリア OP FULL 「BURN」/FLOW-sm29409391.mp4
[download] 100% of 13.56MiB in 00:02

I guess 2 seconds is shorter than the period of heartbeat signals. Do you have longer examples?

@f11894
Copy link

@f11894 f11894 commented Oct 28, 2017

Please try with this URL.
このURLで試してください。

http://www.nicovideo.jp/watch/sm31807439

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 28, 2017

Thanks! I can reproduce the error now

$ python -m youtube_dl -v "http://www.nicovideo.jp/watch/sm31807439" -u yan12125@gmail.com
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.nicovideo.jp/watch/sm31807439', '-u', 'PRIVATE']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Git HEAD: 8e01f3ca8
[debug] Python version 3.6.3 - Linux-4.13.9-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] sm31807439: Downloading webpage
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_128kbps
[niconico] sm31807439: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_1000kbps_540p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm31807439: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://pb02.dmc.nico:2807/vod/ht2_nicovideo/nicovideo-sm31807439_8e75a000d8b99f43ffb7b843d9a9a72fd4711638418e2af8b574cf48f8fad073?ht2_nicovideo=48110477.823ysw_oyjxl8_jlzjnlmfy8gt'
[download] Destination: 【厳選】奇妙ゲーム集FINAL【PS1】-sm31807439.mp4
[download]  22.8% of 425.49MiB at 844.20KiB/s ETA 06:38[download] Got server HTTP error: Downloaded 101711872 bytes, expected 446155552 bytes. Retrying (attempt 1 of 10)...
[download] Resuming download at byte 101711872
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1892, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1831, in dl
    return fd.download(name, info)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 286, in real_download
    establish_connection()
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/http.py", line 74, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 2195, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
@yan12125 yan12125 changed the title NicoVideos Dont work anymore [niconico] Downloading long videos requires heartbeat signals Oct 28, 2017
@yan12125 yan12125 added the bug label Oct 28, 2017
@Ristellise
Copy link
Author

@Ristellise Ristellise commented Nov 16, 2017

any update on this?

@awei78
Copy link

@awei78 awei78 commented Jan 26, 2018

try this way:
in youtube_dl\extractor\niconico.py, get heartbeat datas and execute heart-beat:

def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
    ...
    session_response = self._download_json(
        session_api_endpoint['url'], video_id,
        query={'_format': 'json'},
        headers={'Content-Type': 'application/json'},
        note='Downloading JSON metadata for %s' % format_id,
        data=...)

        # get heart-beat datas
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        hb_data = json.dumps(session_response['data'])

        # send heart-beat command
        def heart_beat():
            import urllib2
            try:
                urllib2.urlopen(url=api_url, data=hb_data)
                print '........................................heart beat!'
            except Exception as ex:
                print '........................................heart beat fail: ' + ex.message

            # need a condition to end timer
            if !downloaded:
                timer = threading.Timer(25, heart_beat)
                timer.start()

        heart_beat()

        resolution = video_quality.get('resolution', {})
        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
        }

enjoy it!

@ciffelia
Copy link

@ciffelia ciffelia commented Jan 30, 2018

@awei78 Could you create pull request for this?

@awei78
Copy link

@awei78 awei78 commented Feb 3, 2018

@prince0203 I am sorry for my poor English. I think that i can not create it, but I can provide a solution:

  1. extend NiconicoIE:
from ..extractor.niconico import NiconicoIE as InconicoBase
...
class NiconicoIE(InconicoBase):
    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
        ...
                # get heart-beat data
        api_url = session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT'
        data = json.dumps(session_response['data'])
        resolution = video_quality.get('resolution', {})

        return {
            'url': session_response['data']['session']['content_uri'],
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
            'height': resolution.get('height'),
            'width': resolution.get('width'),
            'heartbeat_url': api_url,  # pay attention here
            'heartbeat_data': data,
        }
  1. extend HttpFD:
from http import HttpFD

class HttpHB(HttpFD):
    def real_download(self, filename, info_dict):
        result = False
        if 'heartbeat_url'in info_dict:
            def heart_beat():
                try:
                    urllib2.urlopen(url=info_dict['heartbeat_url'], data=info_dict['heartbeat_data'])
                    print '........................................heart beat!'
                except Exception as ex:
                    print '........................................heart beat fail: ' + ex.message
                    pass

                if not result:
                    timer = threading.Timer(25, heart_beat)
                    timer.start()

            heart_beat()

        result = super(HttpHB, self).real_download(filename, info_dict)
        return result
  1. check heartbeat_url sign and use HttpHB to download:
            if 'heartbeat_url' in info:
                fd = HttpHB(self._ydl, params)
            ...
@nekojun nekojun mentioned this issue May 19, 2018
5 of 9 tasks complete
@Gorfiend Gorfiend linked a pull request that will close this issue Nov 18, 2018
5 of 9 tasks complete
@alexvong1995 alexvong1995 mentioned this issue Feb 22, 2019
5 of 9 tasks complete
@nao0x2c6
Copy link

@nao0x2c6 nao0x2c6 commented Nov 16, 2019

@yan12125
How is this issue status?
It seems that panhartstuff/youtube-dl@ff19685 works well for me.
I have been able to download videos which take over an hour video with this patch.

Thanks.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Nov 16, 2019

Sorry, but I'm no longer working on youtube-dl. Current developers will determine whether any of pending pull requests (#18230, #16495) are good enough to merge or not. Or, if neither works, you'll need to open a new pull request.

@nao0x2c6
Copy link

@nao0x2c6 nao0x2c6 commented Nov 16, 2019

@yan12125
Thank you for your reply !!

@dstftw @remitamine

Current developers will determine whether any of pending pull requests (#18230, #16495) are good enough to merge or not. Or, if neither works, you'll need to open a new pull request.

Are "Current developers" you two?
Thank you for developing the awesome tool.
How are these pull request ?

I'm sorry if my poor english were rude expression...

@nao0x2c6
Copy link

@nao0x2c6 nao0x2c6 commented Jan 23, 2020

@dstftw @remitamine
Hi,
How is this issue status ?

@jxu
Copy link
Contributor

@jxu jxu commented Jun 17, 2020

Workaround is to keep running command using loop #19261 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

7 participants
You can’t perform that action at this time.