iqiyi not downloading properly #11953

Stepin2 · 2017-02-03T08:34:04Z

Log is as follows(including 2 examples):

$ ./youtube-dl http://www.iqiyi.com/a_19rrhanifd.html#vfrm=2-4-0-1 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.iqiyi.com/a_19rrhanifd.html#vfrm=2-4-0-1', u'--verbose']
[debug] Encodings: locale ISO-8859-1, fs ISO-8859-1, out ISO-8859-1, pref ISO-8859-1
[debug] youtube-dl version 2017.02.01
[debug] Python version 2.7.3 - Linux-2.6.32_1-18-0-0-x86_64-with-redhat-4.3-Final
[debug] exe versions: none
[debug] Proxy map: {}
[iqiyi] temp_id: download video page
ERROR: Unable to extract tvid; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 369, in extract
    return self._real_extract(url)
  File "./youtube-dl/youtube_dl/extractor/iqiyi.py", line 345, in _real_extract
    r'data-player-tvid\s*=\s*[\'"](\d+)', webpage, 'tvid')
  File "./youtube-dl/youtube_dl/extractor/common.py", line 672, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract tvid; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

$ ./youtube-dl http://www.iqiyi.com/v_19rramot48.html --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.iqiyi.com/v_19rramot48.html', u'--verbose']
[debug] Encodings: locale ISO-8859-1, fs ISO-8859-1, out ISO-8859-1, pref ISO-8859-1
[debug] youtube-dl version 2017.02.01
[debug] Python version 2.7.3 - Linux-2.6.32_1-18-0-0-x86_64-with-redhat-4.3-Final
[debug] exe versions: none
[debug] Proxy map: {}
[iqiyi] temp_id: download video page
WARNING: unable to extract album title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[iqiyi] 606416900: Download playlist page 2
ERROR: An extractor error has occurred. (caused by KeyError(u'data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/extractor/common.py", line 369, in extract
    return self._real_extract(url)
  File "./youtube-dl/youtube_dl/extractor/iqiyi.py", line 340, in _real_extract
    playlist_result = self._extract_playlist(webpage)
  File "./youtube-dl/youtube_dl/extractor/iqiyi.py", line 326, in _extract_playlist
    vlist = pagelist['data']['vlist']
KeyError: u'data'
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 375, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

ThomasChr · 2017-02-03T09:46:50Z

I can confirm the problem. It seems to be that the extractor is assuming the URL is a Video-URL - but this is not the case. The url seems to be a Playlist-URL.
(It's a little bit tricky for me, because I don't speak any chinese.)
Trying to download the video URL ( http://www.iqiyi.com/v_19rramot48.html ) seems to result in just the opposite - the extractor thinks its a playlist and fails.

Stepin2 · 2017-02-03T09:57:55Z

@ThomasChr hi, the second example ( http://www.iqiyi.com/v_19rramot48.html ), what can I do so that I can extract the url for the video? Is there any solution?
Can I tell the tool what type of url I'm dealing with to avoid such problems?

ThomasChr · 2017-02-03T10:01:13Z

@Stepin2 I'm on it. I got a clue already!

yan12125 · 2017-02-03T10:33:48Z

The second example works for me:

$ youtube-dl -v "http://www.iqiyi.com/v_19rramot48.html" 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://www.iqiyi.com/v_19rramot48.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.02.01
[debug] Git HEAD: d7f9242e3
[debug] Python version 3.6.0 - Linux-4.9.5-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2
[debug] Proxy map: {}
[iqiyi] temp_id: download video page
[iqiyi] 58f77e3870a48940b827b1441933dd94: Downloading JSON metadata
[debug] Invoking downloader on 'http://cache.m.iqiyi.com/dc/dt/text/fb0a7d16x2fee3a93x777a6171/20170121/78/ff/55d6f9eaf45cdba59d5f3885bbd6321f.m3u8?np_tag=nginx_part_tag&qd_sc=c70ff2a198366ae7d575a99719f4b7c7&t_sign=-0-76f90cbd92f94a2e925d83e8ccd22cb7-606416900_04022000001000000000_4'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 74
[download] Destination: 這是什麼鬼：2017-02-01期 吃這個竟能讓女人回春-58f77e3870a48940b827b1441933dd94.mp4
[download]   8.3% of ~67.95MiB at 953.50KiB/s ETA 01:07^CException ignored in: <_io.FileIO name='這是什麼鬼：2017-02-01期 吃這個竟能讓女人回春-58f77e3870a48940b827b1441933dd94.mp4.part-Frag6.part' mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name='這是什麼鬼：2017-02-01期 吃這個竟能讓女人回春-58f77e3870a48940b827b1441933dd94.mp4.part-Frag6.part'>

ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name='這是什麼鬼：2017-02-01期 吃這個竟能讓女人回春-58f77e3870a48940b827b1441933dd94.mp4.part' mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.FileIO name='這是什麼鬼：2017-02-01期 吃這個竟能讓女人回春-58f77e3870a48940b827b1441933dd94.mp4.part' mode='wb' closefd=True>

In the first example, indeed it's a playlist while iqiyi.py doesn't correctly recognize it. #11776 may be related (broken playlist detection)

ThomasChr · 2017-02-03T10:40:00Z

I got the extractor to recognize the playlist. Now I'm hitting the next problem, but I'm on it.
I'll check #11776 afterwards.

ThomasChr · 2017-02-03T10:47:38Z

I think I'm now hitting the geo restriction-problem. Downloading the JSON for the playlist gives me:

var tvInfoJs={"code":"A00004"}
Which leads to a key error when trying to select the data object in the json:
ERROR: An extractor error has occurred. (caused by KeyError(u'data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

@yan12125 I'll make a pull request for this quick fix. I think with the geo restriction I can't do much more here :-(

yan12125 · 2017-02-03T10:49:43Z

@ThomasChr You can use --geo-verification-proxy. There's an example at #11776

ThomasChr · 2017-02-03T10:56:24Z

@yan12125 Thanks.

ThomasChr · 2017-02-03T11:24:40Z

I gave it a shot despite of all the problems with chinese* I'm having.

*Correct me if it's not chinese on the page - I didn't figure out what language it is exactly :-)

Stepin2 · 2017-02-09T09:02:12Z

@ThomasChr Hey, I found several updates of the tool but still the problem is not fixed.
What is the condition currently? Is it a problem easy to solve?

Stepin2 · 2017-02-09T09:05:38Z

I found another error as: $ ./youtube-dl http://www.iqiyi.com/v_19rrmw2r7g.html#vfrm=2-4-0-1 -v -o video
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.iqiyi.com/v_19rrmw2r7g.html#vfrm=2-4-0-1', u'-v', u'-o', u'video']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.02.07
[debug] Python version 2.7.3 - Linux-2.6.32_1-18-0-0-x86_64-with-redhat-4.3-Final
[debug] exe versions: none
[debug] Proxy map: {}
[iqiyi] temp_id: download video page
[iqiyi] a92e5a4fce9da0fd78a205c9eb1b553c: Downloading JSON metadata
[debug] Invoking downloader on u'http://cache.m.iqiyi.com/dc/dt/text/c07eefbx2fe651dex777a6171/20161109/39/42/7457237648e9533354c30736f98d147b.m3u8?np_tag=nginx_part_tag&qd_sc=cc237ce27278e3e98c52bc64553041d4&t_sign=-0-76f90cbd92f94a2e925d83e8ccd22cb7-284555600_04022000001000000000_4'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 686
[download] Destination: video
[download] 3.2% of ~929.51MiB at 2.50MiB/s ETA 10:35ERROR: unable to download video data: <urlopen error [Errno 110] Connection timed out>
Traceback (most recent call last):
File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1704, in process_info
success = dl(filename, info_dict)
File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1646, in dl
return fd.download(name, info)
File "./youtube-dl/youtube_dl/downloader/common.py", line 353, in download
return self.real_download(filename, info_dict)
File "./youtube-dl/youtube_dl/downloader/hls.py", line 121, in real_download
'http_headers': info_dict.get('http_headers'),
File "./youtube-dl/youtube_dl/downloader/common.py", line 353, in download
return self.real_download(filename, info_dict)
File "./youtube-dl/youtube_dl/downloader/http.py", line 61, in real_download
data = self.ydl.urlopen(request)
File "./youtube-dl/youtube_dl/YoutubeDL.py", line 2006, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 438, in error
result = self._call_chain(*args)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 625, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "./youtube-dl/youtube_dl/utils.py", line 884, in http_open
req)
File "/home/spider/.jumbo/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 110] Connection timed out>

ThomasChr · 2017-02-09T09:13:15Z

@Stepin2 My changes have not gone through the review-process yet - I am sorry for that but I can't do anything about it. :-(
The changes are waiting in PR #11956
You can manually apply my changes if that helps. What OS are you using? Do you feel comfortable if I just sent you the python file with the changes?

Stepin2 · 2017-02-09T09:54:45Z

@ThomasChr I use linux (centos 6); It would be great if you can just give me the file.

ThomasChr · 2017-02-09T12:06:17Z

@Stepin2 Thats great.
Just use this version:
https://github.com/ThomasChr/youtube-dl/blob/iqiyifix/youtube_dl/extractor/iqiyi.py

And tell me if it works for you. It's a little bit difficult to test if everything works when you can't read the site's content :-)

Stepin2 · 2017-02-10T03:27:43Z

@ThomasChr It finally works! Thanks a lot.

yan12125 added the bug label Feb 3, 2017

ThomasChr mentioned this issue Feb 3, 2017

[iqiyi] Fix for Issue #11953 #11956

Open

4 of 8 tasks complete

ThomasChr mentioned this issue Feb 9, 2017

Changing get_element_by_attribute() and get_element_by_class() to return a list of results. See PR #11956 #12034

Merged

4 of 8 tasks complete

ytdl-org / youtube-dl

iqiyi not downloading properly #11953

iqiyi not downloading properly #11953

Stepin2 commented Feb 3, 2017 •

edited

ThomasChr commented Feb 3, 2017

Stepin2 commented Feb 3, 2017 •

edited

ThomasChr commented Feb 3, 2017

yan12125 commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

yan12125 commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

ThomasChr commented Feb 3, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited

ThomasChr commented Feb 9, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited

ThomasChr commented Feb 9, 2017

Stepin2 commented Feb 10, 2017

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

iqiyi not downloading properly #11953

iqiyi not downloading properly #11953

Comments

Stepin2 commented Feb 3, 2017 • edited

ThomasChr commented Feb 3, 2017

Stepin2 commented Feb 3, 2017 • edited

ThomasChr commented Feb 3, 2017

yan12125 commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

yan12125 commented Feb 3, 2017

ThomasChr commented Feb 3, 2017

ThomasChr commented Feb 3, 2017 • edited

Stepin2 commented Feb 9, 2017 • edited

Stepin2 commented Feb 9, 2017 • edited

ThomasChr commented Feb 9, 2017 • edited

Stepin2 commented Feb 9, 2017 • edited

ThomasChr commented Feb 9, 2017

Stepin2 commented Feb 10, 2017

Essential cookies

Always active

Analytics cookies

Stepin2 commented Feb 3, 2017 •

edited

Stepin2 commented Feb 3, 2017 •

edited

ThomasChr commented Feb 3, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited

ThomasChr commented Feb 9, 2017 •

edited

Stepin2 commented Feb 9, 2017 •

edited