Unable to download from nicovideo.jp #19261

kateabr · 2019-02-18T08:28:07Z

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2019.02.18. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

I've verified and I assure that I'm running youtube-dl 2019.02.18

Before submitting an issue make sure you have:

At least skimmed through the README, most notably the FAQ and BUGS sections
Searched the bugtracker for similar issues including closed ones
Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

Bug report (encountered problems with youtube-dl)
Site support request (request for adding support for a new site)
Feature request (request for a new functionality)
Question
Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

PS D:\Music> youtube-dl --verbose http://www.nicovideo.jp/watch/sm34642257
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'http://www.nicovideo.jp/watch/sm34642257']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2019.02.18
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2
[debug] Proxy map: {}
[niconico] sm34642257: Downloading webpage
[niconico] sm34642257: Downloading JSON metadata for h264_1080p-aac_192kbps
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\extractor\common.py", line 605, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\YoutubeDL.py", line 2225, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Description of your issue, suggested solution and other information

For the past few days youtue-dl is unable to download most niconico videos.
URL example (not working): http://www.nicovideo.jp/watch/sm34642257
URL example (working): http://www.nicovideo.jp/watch/sm34425299

alexvong1995 · 2019-02-22T20:32:12Z

Background

I'm not sure, but the underlying problem could be #14582. As mentioned in
#14582, niconico now requires the downloader to send heartbeat signals to keep
the connection alive. If you have a slow connection (which I can simulate by
downloading through tor), you can get a 403. Unfortunately, PR #16495 which
should solve this problem has not been merged yet.

Recommendation

As a workaround, I recommend you to tried re-downloading the video for multiple
times until it succeeds. Also, it's probably best to download around 3:00 a.m.
(UTC+9) during the mid-night of japan since the traffic is the lowest at that
time.

Test Results

sm34642257 requires 0 retry:

alexvong1995@debian:~$ until torsocks youtube-dl --verbose https://www.nicovideo.jp/watch/sm34642257; do printf 'Retrying...\n\n'; done
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.nicovideo.jp/watch/sm34642257']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.02.18
1550865024 WARNING torsocks[11226]: [syscall] Unsupported syscall number 217. Denying the call (in tsocks_syscall() at syscall.c:568)
[debug] Python version 3.7.0 (CPython) - Linux-4.19.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[niconico] sm34642257: Downloading webpage
[niconico] sm34642257: Downloading JSON metadata for h264_1080p-aac_192kbps
[niconico] sm34642257: Downloading JSON metadata for h264_720p-aac_192kbps
[niconico] sm34642257: Downloading JSON metadata for h264_480p-aac_192kbps
[niconico] sm34642257: Downloading JSON metadata for h264_360p-aac_192kbps
[niconico] sm34642257: Downloading JSON metadata for h264_360p_low-aac_192kbps
[niconico] sm34642257: Downloading JSON metadata for h264_1080p-aac_64kbps
[niconico] sm34642257: Downloading JSON metadata for h264_720p-aac_64kbps
[niconico] sm34642257: Downloading JSON metadata for h264_480p-aac_64kbps
[niconico] sm34642257: Downloading JSON metadata for h264_360p-aac_64kbps
[niconico] sm34642257: Downloading JSON metadata for h264_360p_low-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://pb038c42f0d.dmc.nico/vod/ht2_nicovideo/nicovideo-sm34642257_798a1c8453b97175c32f8cd2d7a11f68c7f49fb6789cba150f9b83b666ef7b69?ht2_nicovideo=6-Ka3xBnW2W6_1550865029516.4jd2nm_pncf49_2zz4l315pkxhw'
[download] Destination: 【MATCHA】user_TerakomuroP【Original】-sm34642257.mp4
[download] 100% of 24.24MiB in 02:04

sm34425299 requires 2 retrys:

alexvong1995@debian:~$ until torsocks youtube-dl --verbose https://www.nicovideo.jp/watch/sm34425299; do printf 'Retrying...\n\n'; done
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.nicovideo.jp/watch/sm34425299']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.02.18
1550865312 WARNING torsocks[11287]: [syscall] Unsupported syscall number 217. Denying the call (in tsocks_syscall() at syscall.c:568)
[debug] Python version 3.7.0 (CPython) - Linux-4.19.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[niconico] sm34425299: Downloading webpage
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://pa0515be243.dmc.nico/vod/ht2_nicovideo/nicovideo-sm34425299_333ff0ef2febbf9cb6c34cb3767ebd53a6b40e199da62b1ea4d549896fee1969?ht2_nicovideo=6-Sk4o6lclxz_1550865321541.8ukq8n_pncfcf_2t5eu7ha9vznu'
[download] Destination: 【Hatsune Miku】Bitter Choco Decoration【syudou】-sm34425299.mp4
[download]  22.6% of 75.33MiB at 115.99KiB/s ETA 08:34[download] Got server HTTP error: Downloaded 17884424 bytes, expected 78984307 bytes. Retrying (attempt 1 of 10)...
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 1913, in process_info
    success = dl(filename, info_dict)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 1852, in dl
    return fd.download(name, info)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Retrying...

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.nicovideo.jp/watch/sm34425299']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.02.18
1550865525 WARNING torsocks[11351]: [syscall] Unsupported syscall number 217. Denying the call (in tsocks_syscall() at syscall.c:568)
[debug] Python version 3.7.0 (CPython) - Linux-4.19.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[niconico] sm34425299: Downloading webpage
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://pa081d2bbc3.dmc.nico/vod/ht2_nicovideo/nicovideo-sm34425299_333ff0ef2febbf9cb6c34cb3767ebd53a6b40e199da62b1ea4d549896fee1969?ht2_nicovideo=6-x6DHyUL7B0_1550865530295.c5o3f9_pncfi9_1ag1rhpv0nji1'
[download] Resuming download at byte 17884424
[download] Destination: 【Hatsune Miku】Bitter Choco Decoration【syudou】-sm34425299.mp4
[download]  86.5% of 75.33MiB at 451.55KiB/s ETA 00:23[download] Got server HTTP error: Downloaded 68316614 bytes, expected 78984307 bytes. Retrying (attempt 1 of 10)...
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 1913, in process_info
    success = dl(filename, info_dict)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 1852, in dl
    return fd.download(name, info)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/gnu/store/6ra6z6hjg59lx058ni0nphk2xw3jac62-youtube-dl-2019.02.18/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/gnu/store/ibxxisi81v1v3y5xc3axl4x4rx6jpzw8-python-3.7.0/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Retrying...

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.nicovideo.jp/watch/sm34425299']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.02.18
1550865682 WARNING torsocks[11427]: [syscall] Unsupported syscall number 217. Denying the call (in tsocks_syscall() at syscall.c:568)
[debug] Python version 3.7.0 (CPython) - Linux-4.19.0-2-amd64-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[niconico] sm34425299: Downloading webpage
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_192kbps
[niconico] sm34425299: Downloading JSON metadata for h264_3000kbps_1080p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_2000kbps_720p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_1600kbps_540p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_600kbps_360p-aac_64kbps
[niconico] sm34425299: Downloading JSON metadata for h264_300kbps_360p-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://pa081d2bbc3.dmc.nico/vod/ht2_nicovideo/nicovideo-sm34425299_333ff0ef2febbf9cb6c34cb3767ebd53a6b40e199da62b1ea4d549896fee1969?ht2_nicovideo=6-w5KZMJL7CE_1550865687594.slz3hq_pncfmj_3whiv390v0ek'
[download] Resuming download at byte 68316614
[download] Destination: 【Hatsune Miku】Bitter Choco Decoration【syudou】-sm34425299.mp4
[download] 100% of 75.33MiB in 01:02

Remark

Please note that I use torsocks to simulate slow connection and to bypass geo
restriction (if any), but torsocks is not available on windows.

Also, the shell command for retrying requires a posix shell to run, but there's
a equivalence for windows.

Polsy · 2019-03-01T22:58:59Z

As noted in #20088 (this is regarding the issue where the download doesn't even start, rather than failing partway through) this can be worked around by having youtube-dl log in, although there's no requirement to do so in browser.

panhartstuff · 2019-03-12T03:03:12Z

Still no merge... :/
Also why is there 2 heartbeat PR? What's the difference?
#16495 #18230

yuis-ice · 2019-03-25T07:47:16Z

I wrote a bash script to continually run the command until it succeeds.

while [[ true ]]; do youtube-dl [options] "https://www.nicovideo.jp/watch/sm34815188" && break || continue ; done

remitamine · 2019-03-25T08:29:17Z

@yuis-ice you can use an until loop for a cleaner and shorter command:

until youtube-dl [options] https://www.nicovideo.jp/watch/sm34815188; do continue; done

alexvong1995 · 2019-03-26T12:32:39Z

remitamine <notifications@github.com> writes:

@yuis-ice you can use an until loop for a more clean and shorter command: until youtube-dl [options] https://www.nicovideo.jp/watch/sm34815188; do continue; done

@yuis-ice Or you can use the command I mentioned above:

until youtube-dl https://www.nicovideo.jp/watch/sm34642257; do printf 'Retrying...\n\n'; done

It prints a retrying message which can be useful when using it in a script.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

jxu mentioned this issue Jun 17, 2020

[niconico] Downloading long videos requires heartbeat signals #14582

Open

4 of 4 tasks complete

ytdl-org / youtube-dl

Unable to download from nicovideo.jp #19261

Unable to download from nicovideo.jp #19261

kateabr commented Feb 18, 2019

alexvong1995 commented Feb 22, 2019 •

edited

Polsy commented Mar 1, 2019

panhartstuff commented Mar 12, 2019

yuis-ice commented Mar 25, 2019

remitamine commented Mar 25, 2019 •

edited

alexvong1995 commented Mar 26, 2019

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

Unable to download from nicovideo.jp #19261

Unable to download from nicovideo.jp #19261

Comments

kateabr commented Feb 18, 2019

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.02.18. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Description of your issue, suggested solution and other information

alexvong1995 commented Feb 22, 2019 • edited

Background

Recommendation

Test Results

Remark

Polsy commented Mar 1, 2019

panhartstuff commented Mar 12, 2019

yuis-ice commented Mar 25, 2019

remitamine commented Mar 25, 2019 • edited

alexvong1995 commented Mar 26, 2019

Essential cookies

Always active

Analytics cookies

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2019.02.18. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

alexvong1995 commented Feb 22, 2019 •

edited

remitamine commented Mar 25, 2019 •

edited