Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VEVO KeyError: 'streams' #11719

Closed
g8keepa opened this issue Jan 14, 2017 · 12 comments
Closed

VEVO KeyError: 'streams' #11719

g8keepa opened this issue Jan 14, 2017 · 12 comments

Comments

@g8keepa
Copy link

@g8keepa g8keepa commented Jan 14, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.01.14. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.01.14

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • [x ] Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl -v www.vevo.com/watch/boostee/pop-corn-clip-officiel/FR1A91600909
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'www.vevo.com/watch/boostee/pop-corn-clip-officiel/FR1A91600909']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.14
[debug] Python version 2.7.9 - Darwin-13.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
WARNING: The url doesn't specify the protocol, trying with http
[Vevo] FR1A91600909: Downloading video info
WARNING: Unable to download info: HTTP Error 503: Service Unavailable
[Vevo] Retrieving oauth token
[Vevo] FR1A91600909: Downloading api video info
[Vevo] FR1A91600909: Downloading video versions info
WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized
[Vevo] FR1A91600909: Downloading webpage
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 444, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 434, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1791, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 705, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 758, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 359, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/vevo.py", line 271, in _real_extract
TypeError: string indices must be integers

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

This error is coming up on certain files Was working previously. Any help is appreciated, thank you!

@g8keepa
Copy link
Author

@g8keepa g8keepa commented Jan 21, 2017

@dstftw I can watch the file normally using a DNS service via a browser. Had the geo-restricted issue before and it was resolved. Hopefully a fix can be determined, thanks!

@tijder
Copy link

@tijder tijder commented Jan 30, 2017

I got the same problem with some clips. When I view this video online it plays normaly. My log is:

youtube-dl -v www.vevo.com/watch/boostee/pop-corn-clip-officiel/FR1A91600909
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'www.vevo.com/watch/boostee/pop-corn-clip-officiel/FR1A91600909']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.29
[debug] Python version 2.7.9 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-Debian-8
[debug] exe versions: avconv 2.6.9, avprobe 2.6.9, ffmpeg 2.6.9, ffprobe 2.6.9, rtmpdump 2.4
[debug] Proxy map: {}
WARNING: The url doesn't specify the protocol, trying with http
[Vevo] Retrieving oauth token
[Vevo] FR1A91600909: Downloading api video info
[Vevo] FR1A91600909: Downloading video versions info
WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized
[Vevo] FR1A91600909: Downloading webpage
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 451, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 441, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1791, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 705, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 758, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 369, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/vevo.py", line 210, in _real_extract
TypeError: string indices must be integers
@g8keepa
Copy link
Author

@g8keepa g8keepa commented Feb 9, 2017

Any luck as of yet?

@g8keepa
Copy link
Author

@g8keepa g8keepa commented Feb 9, 2017

Here's an updated log from a video I just tried to pull (new error message this time, just updated):

youtube-dl -v http://www.vevo.com/watch/martin-solveig/Rejection/FRUV70500038
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://www.vevo.com/watch/martin-solveig/Rejection/FRUV70500038']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.02.07
[debug] Python version 2.7.9 - Darwin-13.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[Vevo] Retrieving oauth token
[Vevo] FRUV70500038: Downloading api video info
[Vevo] FRUV70500038: Downloading video versions info
WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized
[Vevo] FRUV70500038: Downloading webpage
ERROR: An extractor error has occurred. (caused by KeyError(u'streams',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 369, in extract
return self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/vevo.py", line 196, in _real_extract
video_versions = self._extract_json(webpage, video_id, 'streams')[video_id][0]
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/vevo.py", line 25, in _extract_json
video_id)['default'][item]
KeyError: u'streams'
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 696, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 375, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'streams',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

@yan12125 yan12125 changed the title VEVO TypeError: string indices must be integers (see log) VEVO KeyError: 'streams' Feb 22, 2017
@yan12125 yan12125 mentioned this issue Feb 22, 2017
4 of 8 tasks complete
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Feb 22, 2017

Those videos are unavailable with US proxies. Vevo said:

Oops! How Embarrassing...

We don't have permission to stream this video in your location. Please choose another video from our massive library.

Where are those videos available?

@Hrxn
Copy link

@Hrxn Hrxn commented Feb 22, 2017

Strange..
Can reproduce with http://www.vevo.com/watch/martin-solveig/Rejection/FRUV70500038
with these errors

WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized
...
ExtractorError: An extractor error has occurred. (caused by KeyError(u'streams',));

But I can see it in the Browser.

Tested with another video:
https://www.vevo.com/watch/the-chainsmokers/Paris/USQX91700259

Seems to work fine.

I think the issue is that the video is somewhat old, and therefore the metadata extraction doesn't work like the newer ones?

Edits:

Yeah, it's failing here:

[Vevo] FRUV70500038: Downloading video versions info
WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized

@tijder
Copy link

@tijder tijder commented Feb 22, 2017

@yan12125 in the Netherlands.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Feb 23, 2017

Thanks. Public Netherlands proxies are too slow for proper debugging...

@Hrxn are you in Netherlands, too?

@Hrxn
Copy link

@Hrxn Hrxn commented Feb 23, 2017

Germany...

@yan12125 yan12125 added the bug label Feb 23, 2017
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Feb 24, 2017

Sorry, @Hrxn, I didn't notice your Github profile.

I managed to create a fix from Netherlands/Germany proxies. Here's my patch:

diff --git a/youtube_dl/extractor/vevo.py b/youtube_dl/extractor/vevo.py
index c4e37f694..6fe7fccf5 100644
--- a/youtube_dl/extractor/vevo.py
+++ b/youtube_dl/extractor/vevo.py
@@ -17,12 +17,12 @@ from ..utils import (
 
 
 class VevoBaseIE(InfoExtractor):
-    def _extract_json(self, webpage, video_id, item):
+    def _extract_json(self, webpage, video_id):
         return self._parse_json(
             self._search_regex(
                 r'window\.__INITIAL_STORE__\s*=\s*({.+?});\s*</script>',
                 webpage, 'initial store'),
-            video_id)['default'][item]
+            video_id)
 
 
 class VevoIE(VevoBaseIE):
@@ -193,7 +193,14 @@ class VevoIE(VevoBaseIE):
         # https://github.com/rg3/youtube-dl/issues/9366)
         if not video_versions:
             webpage = self._download_webpage(url, video_id)
-            video_versions = self._extract_json(webpage, video_id, 'streams')[video_id][0]
+            json_data = self._extract_json(webpage, video_id)
+            if 'streams' in json_data.get('default', {}):
+                video_versions = json_data['default']['streams'][video_id][0]
+            else:
+                video_versions = [
+                    value
+                    for key, value in json_data['apollo']['data'].items()
+                    if key.startswith('%s.streams' % video_id)]
 
         uploader = None
         artist = None
@@ -207,7 +214,7 @@ class VevoIE(VevoBaseIE):
 
         formats = []
         for video_version in video_versions:
-            version = self._VERSIONS.get(video_version['version'])
+            version = self._VERSIONS.get(video_version.get('version'), 'generic')
             version_url = video_version.get('url')
             if not version_url:
                 continue
@@ -339,7 +346,7 @@ class VevoPlaylistIE(VevoBaseIE):
             if video_id:
                 return self.url_result('vevo:%s' % video_id, VevoIE.ie_key())
 
-        playlists = self._extract_json(webpage, playlist_id, '%ss' % playlist_kind)
+        playlists = self._extract_json(webpage, playlist_id)['default']['%ss' % playlist_kind]
 
         playlist = (list(playlists.values())[0]
                     if playlist_kind == 'playlist' else playlists[playlist_id])

However, I can't commit it as I didn't really test it in Python. The proxy I use involves HTTP digest authentication, and Python is broken with that.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Feb 24, 2017

it does work with this patch.

python __main__.py http://www.vevo.com/watch/boostee/pop-corn-clip-officiel/FR1A91600909
[Vevo] Retrieving oauth token
[Vevo] FR1A91600909: Downloading api video info
[Vevo] FR1A91600909: Downloading video versions info
WARNING: Failed to download video versions info: HTTP Error 401: Unauthorized
[Vevo] FR1A91600909: Downloading webpage
[Vevo] FR1A91600909: Downloading generic m3u8 information
[Vevo] FR1A91600909: Downloading generic MPD information
[Vevo] FR1A91600909: Downloading generic MPD information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 37
[download] Destination: Boostee - Pop Corn (Clip officiel)-FR1A91600909.mp4
[download]   2.7% of ~70.66MiB at 282.84KiB/s ETA 04:29
@yan12125 yan12125 closed this in 9d0c08a Feb 25, 2017
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Feb 25, 2017

Thanks @remitamine. I've pushed my patch to master without a proper _TEST.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.