Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: 'url_encoded_fmt_stream_map' #467

Closed
haroldfry opened this issue Sep 11, 2019 · 57 comments
Closed

KeyError: 'url_encoded_fmt_stream_map' #467

haroldfry opened this issue Sep 11, 2019 · 57 comments

Comments

@haroldfry
Copy link

haroldfry commented Sep 11, 2019

Seems like starting from today every video will cause this error when initialize a YouTube instance. Is there a quick way to fix this? I was using the same code yesterday and everything works fine.

python version 3.6
I've tried various pytube version such as 8.0.0, 9.5.0 and 9.5.2. None of them works and all lead to this error.

yt = YouTube('https://www.youtube.com/watch?v=p1X5A9Nmsy4')
Traceback (most recent call last):
File "", line 1, in
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 88, in init
self.prefetch_init()
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 97, in prefetch_init
self.init()
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 130, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/mixins.py", line 89, in apply_descrambler
for i in stream_data[key].split(',')
KeyError: 'url_encoded_fmt_stream_map'

@PHDPeter
Copy link

I have the same, it was working this morning and now I get the same error.
At first, I thought It might have been my update from 9.5.1 to 9.5.2 (python 3.6) but I did that this morning and it all seemed fine at the time.
I did a quick restart and then found the issues. To me this point to a change on the youtube site?

@Choromanski
Copy link

I'm assuming that YouTube has changed something big as both ytdl-org/youtube-dl and TeamNewPipe/NewPipe are having similar issues.

@Dillon7C7
Copy link

All those "free youtube downloader" websites that my co-workers often use seem to be broken as well, so YouTube definitely changed something.

@danielgordon10
Copy link

I found ytdl-org/youtube-dl@bf1317d from youtube-dl, but no idea how to port the change to pytube.

@danperazzo
Copy link

Got the same error. I thought it was only on a specific set of url's but it seems to be on every one of them :((

@choonsiong
Copy link

Same here, stuck with the KeyError: 'url_encoded_fmt_stream_map' error... 😞

@danperazzo
Copy link

danperazzo commented Sep 11, 2019

The strange part was that I started using pytube just a few hours ago. All of the sudden the commands that I was using were not working anymore.

@ndg63276
Copy link

Seems like this line:
if 'adaptive_fmts' in self.player_config_args:
needs to be something like
if 'adaptiveFormats' in self.player_config_args['player_response']:
But I can't quite put it together in the various places this comes up.

@qingguoxu
Copy link

it seems everything works again. I have the same problem this morning, but now the same script works like before. So definitely, it's Youtube side thing.

@haroldfry
Copy link
Author

confirm that everything is working again.

@genesysvip
Copy link

It's see stream_data without [url_encoded_fmt_stream_map] Key in mixins
Is youtube remove this key?

@BenTheJedi
Copy link

It seems some video has this "url_encoded_fmt_stream_map" key, and some does not.
For example:

fEnC5gwNAN0 has the key, but
BXf1j8Hz2bU does not.

So, pytube would work on the first video, but not the second video.

To find out if the video has the "url_encoded_fmt_stream_map" key, one can obtain it from a browser with the follow:
https://www.youtube.com/get_video_info?video_id={videoId}.

Sorry, that I don' t have a solution. Hopefully, get_video_info could help someone to show us a solution.

@ghost
Copy link

ghost commented Dec 18, 2019

The reason for getting this KeyError is that there are many videos in YouTube now which does not have 'url_encoded_fmt_stream_map' and 'adaptive_fmts' keys in self.player_config_args.
To make this work, instead of using these keys, 'formats' and 'adaptiveFormats' keys should be used. Whole logic to make Stream object will be changed because 'url_encoded_fmt_stream_map' and 'adaptive_fmts' keys have string values while 'formats' and 'adaptiveFormats' keys are JSON objects. Hope it gets fixed.

@bennyb518
Copy link

you can replace the apply_descrambler function inside the mixins.py file with the following

if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        stream_data[key] = [{u'url': format_item[u'url'],
                             u'type': format_item[u'mimeType'],
                             u'quality': format_item[u'quality'],
                             u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

@vade
Copy link

vade commented Dec 19, 2019

@bennyb518

Im having issues with this fix, with latest pytube im getting url not found errors on videos: DyDfgMOUjCI for example.

@cjk02025
Copy link

cjk02025 commented Dec 20, 2019

@bennyb518 attempted implementation of your code as above, but i'm receiving the error: KeyError: 'url' now...

Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 88, in init
self.prefetch_init()
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 97, in prefetch_init
self.init()
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 143, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube\mixins.py", line 101, in apply_descrambler
u'itag': format_item[u'itag']} for format_item in formats]
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube\mixins.py", line 101, in
u'itag': format_item[u'itag']} for format_item in formats]
KeyError: 'url'

@rafaelmarques7
Copy link

I am also experiencing this issue.
Are there any solutions for this?
Thank you.

@MaartenBransen
Copy link

@bennyb518 tested your fix for a few videos that didn't work before and for a few that did work, and now they all worked correctly. Thanks!

@Apfelvater
Copy link

@bennyb518 tested the fix too. Fixed it for a few videos, did not for some others. You should still commit your solution?
Also theres this fix, which, as someone commented, doesnt allways work, too:
https://github.com/nficano/pytube/pull/528/files
any1 tried both fixes combined?

@rhodso
Copy link

rhodso commented Dec 21, 2019

@bennyb518 tested that fix and it seems to work

@FidoDidoVN
Copy link

FidoDidoVN commented Dec 22, 2019

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

@giacaglia
Copy link
Contributor

giacaglia commented Dec 23, 2019

Created a fix for this bug. If you want to install the package with the fix before it gets to the main package, run the following:
pip3 install git+https://github.com/giacaglia/pytube.git

@callixte-girard
Copy link

@giacaglia it seems to work, thanks for forking and patching it !
I was wondering, what does the git+ before the url do ?

@grogowar
Copy link

@giacaglia thanks, it works, but default ordering doesn't work anymore, I always have to add .order_by('resolution').desc()

@sickzzzz
Copy link

I am facing an error like this
'list' object has no attribute 'split'

and the Key Error of 'url_encoded_fmt_stream_map' is not fixed

@ArpanSriv
Copy link

Thanks for the fix. Really appreciate it. @giacaglia
Hope it gets merged soon.

@Jakub-Kubkowski
Copy link

Hello !
After instaling: " pip3 install git+https://github.com/giacaglia/pytube.git"
I have this error for some links:
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 88, in init
self.prefetch_init()
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 97, in prefetch_init
self.init()
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 143, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/mixins.py", line 102, in apply_descrambler
u'itag': format_item[u'itag']} for format_item in formats]
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/mixins.py", line 102, in
u'itag': format_item[u'itag']} for format_item in formats]
KeyError: 'url'

How can I fix this ?

@movraxrsp
Copy link

still broke

after pip install git+, i'm getting KeyError: 'url' like jakub and also VideoUnavailable sometimes for v=bLfcwDZ5GzQ

@aryanshridhar
Copy link

I implemented the code provided by @FidoDidoVN , and it works very well.

All we need to do is to replace the code provided by @FidoDidoVN in the apply_descrambler function within the mixins.py file.

@ValentiaSulli
Copy link

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

hello!
After fixing code as above, I meet a new issue:
AttributeError: 'YouTube' object has no attribute 'download'
How can I fix this? Thank you!

@hy-pg
Copy link

hy-pg commented Jan 13, 2020

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

thank you,it works

@FidoDidoVN
Copy link

@ValentiaSulli can you show your code? may be your code is wrong. 'YouTube' object has no attribute 'download', you have to use this code: YouTube(link video).streams.first().download(yours path). For more advanced you can google how to download youtube video by pytube

@SwissCorePy
Copy link

I get this Error after benny518's solution:
Traceback (most recent call last): File "ytdl.py", line 27, in <module> main() File "ytdl.py", line 16, in main download(url) File "ytdl.py", line 20, in download YouTube(url).streams.first().download(dlpath) File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 88, in __init__ self.prefetch_init() File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 97, in prefetch_init self.init() File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 143, in init mixins.apply_descrambler(self.player_config_args, fmt) File "/anaconda3/lib/python3.6/site-packages/pytube/mixins.py", line 102, in apply_descrambler u'itag': format_item[u'itag']} for format_item in formats] File "/anaconda3/lib/python3.6/site-packages/pytube/mixins.py", line 102, in <listcomp> u'itag': format_item[u'itag']} for format_item in formats] KeyError: 'url'

@Matheuszl
Copy link

try @ bennyb518 e o somes video ainda tem erro porque os formatos não incluem a chave "url", substituem por "cipher", por isso fiz algumas alterações como essa (aprendi python há cerca de 2 meses, portanto, meu código é um pouco longo , pode ser que alguém possa melhorar, mas funciona):

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

Funcionou perfeitamente!
Worked perfectly!
TKS from Brazil

@SwissCorePy
Copy link

Thank you @FidoDidoVN. This Solution Works for me :)
greetings from switzerland

@jangroth
Copy link

jangroth commented Jan 16, 2020

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

@movraxrsp
Copy link

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

I wouldn't count on it any time soon. This repo almost seems abandoned. The last commit was in October. There are 151 issues and 49 PRs. This issue has been open since September. But the owner was making commits on other repos 3 days ago.

@jangroth
Copy link

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

I wouldn't count on it any time soon. This repo almost seems abandoned. The last commit was in October. There are 151 issues and 49 PRs. This issue has been open since September. But the owner was making commits on other repos 3 days ago.

I was wondering about this as well.

@nficano - are you the only maintainer? Is there a current status of the project? If it's 'abandoned' or 'looking for maintainers' - maybe this could be reflected in the README file, and maybe others are interested in taking over/helping out...

@hbmartin
Copy link
Contributor

fixed in my Python 3 only fork: https://github.com/hbmartin/pytube3

@pintonos
Copy link

fixed in my Python 3 only fork: https://github.com/hbmartin/pytube3

File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\pytube\cipher.py", line 271, in get_signature var, _ = transform_plan[0].split(".") ValueError: too many values to unpack (expected 2)

@hbmartin i am getting this error using your fork

@hbmartin
Copy link
Contributor

@pintonos this is a recent YouTube change, tracking work in: https://github.com/hbmartin/pytube3/issues/12

@Error404IsAmazing
Copy link

I am getting this KeyError: 'url' return after implementing @bennyb518's code. Anyone know what causes this issue and most importantly how to fix it?

@braindotai
Copy link

clone repo from https://github.com/hbmartin/pytube3
and paste pytube folder from the repo to the site-packages directory, everything should work fine

@ashwinkk23
Copy link

@braindotai Thanks!

@hbmartin
Copy link
Contributor

hbmartin commented Jan 26, 2020

paste pytube folder from the repo to the site-packages directory

Please don't do this. It will prevent you from getting pytube3 upgrades and will probably lead to weird bugs. Better to ‘pip uninstall pytube; pip install pytube3’

@braindotai
Copy link

paste pytube folder from the repo to the site-packages directory

Please don't do this. It will prevent you from getting pytube3 upgrades and will probably lead to weird bugs. Better to ‘pip uninstall pytube; pip install pytube3’

That's correct 👍

@akd6203
Copy link

akd6203 commented Feb 16, 2020

@bennyb518 it still returns error please help me out
stream_data[key] = [{u'url': format_item[u'url'],
KeyError: 'url'

@krapes
Copy link

krapes commented Feb 17, 2020

I combined the two solutions in this thread as well as added support for another error:

mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/kerri/Documents/Projects/crowdCounting/crowd/lib/python3.6/site-packages/pytube/mixins.py", line 97, in apply_descrambler
formats = json.loads(stream_data['player_response'])['streamingData']['formats']
KeyError: 'formats'

Pull Requests have been made to both this repo and that of @giacaglia. As of writing the git+https://github.com/giacaglia/pytube.git is pulled to master.

@giacaglia
Copy link
Contributor

@krapes PR is now merged to my repo. Thanks @krapes for the PR

@xislam
Copy link

xislam commented Feb 29, 2020

ImportError: cannot import name 'quote' from 'pytube.compat' (/home/islam/projects/youtube/venv/lib/python3.7/site-packages/pytube/compat.py)

@erdincay
Copy link

erdincay commented Apr 13, 2020

I combined the two solutions in this thread as well as added support for another error:

mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/kerri/Documents/Projects/crowdCounting/crowd/lib/python3.6/site-packages/pytube/mixins.py", line 97, in apply_descrambler
formats = json.loads(stream_data['player_response'])['streamingData']['formats']
KeyError: 'formats'

Pull Requests have been made to both this repo and that of @giacaglia. As of writing the git+https://github.com/giacaglia/pytube.git is pulled to master.

I hereby thank you officially in the name of the whole planet.

pip3 install git+https://github.com/giacaglia/pytube.git --upgrade

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests