KeyError: 'url_encoded_fmt_stream_map'

[haroldfry]

Seems like starting from today every video will cause this error when initialize a YouTube instance. Is there a quick way to fix this? I was using the same code yesterday and everything works fine.

python version 3.6
I've tried various pytube version such as 8.0.0, 9.5.0 and 9.5.2. None of them works and all lead to this error.

yt = YouTube('https://www.youtube.com/watch?v=p1X5A9Nmsy4')
Traceback (most recent call last):
File "", line 1, in
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 88, in init
self.prefetch_init()
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 97, in prefetch_init
self.init()
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/main.py", line 130, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/bshen/anaconda3/envs/videodata/lib/python3.6/site-packages/pytube/mixins.py", line 89, in apply_descrambler
for i in stream_data[key].split(',')
KeyError: 'url_encoded_fmt_stream_map'

[PHDPeter]

I have the same, it was working this morning and now I get the same error.
At first, I thought It might have been my update from 9.5.1 to 9.5.2 (python 3.6) but I did that this morning and it all seemed fine at the time.
I did a quick restart and then found the issues. To me this point to a change on the youtube site?

[Choromanski]

I'm assuming that YouTube has changed something big as both ytdl-org/youtube-dl and TeamNewPipe/NewPipe are having similar issues.

[Dillon7C7]

All those "free youtube downloader" websites that my co-workers often use seem to be broken as well, so YouTube definitely changed something.

[danielgordon10]

I found ytdl-org/youtube-dl@bf1317d from youtube-dl, but no idea how to port the change to pytube.

[danperazzo]

Got the same error. I thought it was only on a specific set of url's but it seems to be on every one of them :((

[choonsiong]

Same here, stuck with the KeyError: 'url_encoded_fmt_stream_map' error... 😞

[danperazzo]

The strange part was that I started using pytube just a few hours ago. All of the sudden the commands that I was using were not working anymore.

[ndg63276]

Seems like this line:
if 'adaptive_fmts' in self.player_config_args:
needs to be something like
if 'adaptiveFormats' in self.player_config_args['player_response']:
But I can't quite put it together in the various places this comes up.

[qingguoxu]

it seems everything works again. I have the same problem this morning, but now the same script works like before. So definitely, it's Youtube side thing.

[haroldfry]

confirm that everything is working again.

[genesysvip]

It's see stream_data without [url_encoded_fmt_stream_map] Key in mixins
Is youtube remove this key?

[BenTheJedi]

It seems some video has this "url_encoded_fmt_stream_map" key, and some does not.
For example:

fEnC5gwNAN0 has the key, but
BXf1j8Hz2bU does not.

So, pytube would work on the first video, but not the second video.

To find out if the video has the "url_encoded_fmt_stream_map" key, one can obtain it from a browser with the follow:
https://www.youtube.com/get_video_info?video_id={videoId}.

Sorry, that I don' t have a solution. Hopefully, get_video_info could help someone to show us a solution.

[ghost]

The reason for getting this KeyError is that there are many videos in YouTube now which does not have 'url_encoded_fmt_stream_map' and 'adaptive_fmts' keys in self.player_config_args.
To make this work, instead of using these keys, 'formats' and 'adaptiveFormats' keys should be used. Whole logic to make Stream object will be changed because 'url_encoded_fmt_stream_map' and 'adaptive_fmts' keys have string values while 'formats' and 'adaptiveFormats' keys are JSON objects. Hope it gets fixed.

[bennyb518]

you can replace the apply_descrambler function inside the mixins.py file with the following

if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        stream_data[key] = [{u'url': format_item[u'url'],
                             u'type': format_item[u'mimeType'],
                             u'quality': format_item[u'quality'],
                             u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

[vade]

@bennyb518

Im having issues with this fix, with latest pytube im getting url not found errors on videos: DyDfgMOUjCI for example.

[cjk02025]

@bennyb518 attempted implementation of your code as above, but i'm receiving the error: KeyError: 'url' now...

Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 88, in init
self.prefetch_init()
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 97, in prefetch_init
self.init()
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube_main_.py", line 143, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube\mixins.py", line 101, in apply_descrambler
u'itag': format_item[u'itag']} for format_item in formats]
File "C:\Program Files (x86)\Python37-32\lib\site-packages\pytube\mixins.py", line 101, in
u'itag': format_item[u'itag']} for format_item in formats]
KeyError: 'url'

[rafaelmarques7]

I am also experiencing this issue.
Are there any solutions for this?
Thank you.

[MaartenBransen]

@bennyb518 tested your fix for a few videos that didn't work before and for a few that did work, and now they all worked correctly. Thanks!

[Apfelvater]

@bennyb518 tested the fix too. Fixed it for a few videos, did not for some others. You should still commit your solution?
Also theres this fix, which, as someone commented, doesnt allways work, too:
https://github.com/nficano/pytube/pull/528/files
any1 tried both fixes combined?

[rhodso]

@bennyb518 tested that fix and it seems to work

[FidoDidoVN]

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

[giacaglia]

Created a fix for this bug. If you want to install the package with the fix before it gets to the main package, run the following:
pip3 install git+https://github.com/giacaglia/pytube.git

[callixte-girard]

@giacaglia it seems to work, thanks for forking and patching it !
I was wondering, what does the git+ before the url do ?

[grogowar]

@giacaglia thanks, it works, but default ordering doesn't work anymore, I always have to add .order_by('resolution').desc()

[sickzzzz]

I am facing an error like this
'list' object has no attribute 'split'

and the Key Error of 'url_encoded_fmt_stream_map' is not fixed

[ArpanSriv]

Thanks for the fix. Really appreciate it. @giacaglia
Hope it gets merged soon.

[Jakub-Kubkowski]

Hello !
After instaling: " pip3 install git+https://github.com/giacaglia/pytube.git"
I have this error for some links:
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 88, in init
self.prefetch_init()
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 97, in prefetch_init
self.init()
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/main.py", line 143, in init
mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/mixins.py", line 102, in apply_descrambler
u'itag': format_item[u'itag']} for format_item in formats]
File "/home/jacob/.local/lib/python3.6/site-packages/pytube/mixins.py", line 102, in
u'itag': format_item[u'itag']} for format_item in formats]
KeyError: 'url'

How can I fix this ?

[movraxrsp]

still broke

after pip install git+, i'm getting KeyError: 'url' like jakub and also VideoUnavailable sometimes for v=bLfcwDZ5GzQ

[aryanshridhar]

I implemented the code provided by @FidoDidoVN , and it works very well.

All we need to do is to replace the code provided by @FidoDidoVN in the apply_descrambler function within the mixins.py file.

[ValentiaSulli]

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

hello!
After fixing code as above, I meet a new issue:
AttributeError: 'YouTube' object has no attribute 'download'
How can I fix this? Thank you!

[hy-pg]

try @bennyb518 and somes video still have error because formats not include have "url" key, it replace by "cipher" so I made some change like this (I have learnt python about 2 months, so may be my code is a little long, may be someone can improve, but it work) :

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

thank you,it works

[FidoDidoVN]

@ValentiaSulli can you show your code? may be your code is wrong. 'YouTube' object has no attribute 'download', you have to use this code: YouTube(link video).streams.first().download(yours path). For more advanced you can google how to download youtube video by pytube

[SwissCorePy]

I get this Error after benny518's solution:
Traceback (most recent call last): File "ytdl.py", line 27, in <module> main() File "ytdl.py", line 16, in main download(url) File "ytdl.py", line 20, in download YouTube(url).streams.first().download(dlpath) File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 88, in __init__ self.prefetch_init() File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 97, in prefetch_init self.init() File "/anaconda3/lib/python3.6/site-packages/pytube/__main__.py", line 143, in init mixins.apply_descrambler(self.player_config_args, fmt) File "/anaconda3/lib/python3.6/site-packages/pytube/mixins.py", line 102, in apply_descrambler u'itag': format_item[u'itag']} for format_item in formats] File "/anaconda3/lib/python3.6/site-packages/pytube/mixins.py", line 102, in <listcomp> u'itag': format_item[u'itag']} for format_item in formats] KeyError: 'url'

[Matheuszl]

try @ bennyb518 e o somes video ainda tem erro porque os formatos não incluem a chave "url", substituem por "cipher", por isso fiz algumas alterações como essa (aprendi python há cerca de 2 meses, portanto, meu código é um pouco longo , pode ser que alguém possa melhorar, mas funciona):

import urllib.parse
if key == 'url_encoded_fmt_stream_map' and not stream_data.get('url_encoded_fmt_stream_map'):
        formats = json.loads(stream_data['player_response'])['streamingData']['formats']
        formats.extend(json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats'])
        try:
            stream_data[key] = [{u'url': format_item[u'url'],
                                 u'type': format_item[u'mimeType'],
                                 u'quality': format_item[u'quality'],
                                 u'itag': format_item[u'itag']} for format_item in formats]
        except:
            stream_data[key] = [{u'url': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "url=" in url_item][0].split("=")[1]),
                                  u'sp': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "sp=" in url_item][0].split("=")[1]),
                                  u's': urllib.parse.unquote([url_item for url_item in format_item[u'cipher'].split("&") if "s=" in url_item][0].split("=")[1]),
                                  u'type': format_item[u'mimeType'],
                                  u'quality': format_item[u'quality'],
                                  u'itag': format_item[u'itag']} for format_item in formats]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

Funcionou perfeitamente!
Worked perfectly!
TKS from Brazil

[SwissCorePy]

Thank you @FidoDidoVN. This Solution Works for me :)
greetings from switzerland

[jangroth]

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

[movraxrsp]

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

I wouldn't count on it any time soon. This repo almost seems abandoned. The last commit was in October. There are 151 issues and 49 PRs. This issue has been open since September. But the owner was making commits on other repos 3 days ago.

[jangroth]

Just wondering if someone could provide a PR to integrate the fix upstream? Manually patching source code is a quick workaround only.

I wouldn't count on it any time soon. This repo almost seems abandoned. The last commit was in October. There are 151 issues and 49 PRs. This issue has been open since September. But the owner was making commits on other repos 3 days ago.

I was wondering about this as well.

@nficano - are you the only maintainer? Is there a current status of the project? If it's 'abandoned' or 'looking for maintainers' - maybe this could be reflected in the README file, and maybe others are interested in taking over/helping out...

[hbmartin]

fixed in my Python 3 only fork: https://github.com/hbmartin/pytube3

[pintonos]

fixed in my Python 3 only fork: https://github.com/hbmartin/pytube3

File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\pytube\cipher.py", line 271, in get_signature var, _ = transform_plan[0].split(".") ValueError: too many values to unpack (expected 2)

@hbmartin i am getting this error using your fork

[hbmartin]

@pintonos this is a recent YouTube change, tracking work in: https://github.com/hbmartin/pytube3/issues/12

[Error404IsAmazing]

I am getting this KeyError: 'url' return after implementing @bennyb518's code. Anyone know what causes this issue and most importantly how to fix it?

[braindotai]

clone repo from https://github.com/hbmartin/pytube3
and paste pytube folder from the repo to the site-packages directory, everything should work fine

[ashwinkk23]

@braindotai Thanks!

[hbmartin]

paste pytube folder from the repo to the site-packages directory

Please don't do this. It will prevent you from getting pytube3 upgrades and will probably lead to weird bugs. Better to ‘pip uninstall pytube; pip install pytube3’

[braindotai]

paste pytube folder from the repo to the site-packages directory

Please don't do this. It will prevent you from getting pytube3 upgrades and will probably lead to weird bugs. Better to ‘pip uninstall pytube; pip install pytube3’

That's correct 👍

[akd6203]

@bennyb518 it still returns error please help me out
stream_data[key] = [{u'url': format_item[u'url'],
KeyError: 'url'

[krapes]

I combined the two solutions in this thread as well as added support for another error:

mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/kerri/Documents/Projects/crowdCounting/crowd/lib/python3.6/site-packages/pytube/mixins.py", line 97, in apply_descrambler
formats = json.loads(stream_data['player_response'])['streamingData']['formats']
KeyError: 'formats'

Pull Requests have been made to both this repo and that of @giacaglia. As of writing the git+https://github.com/giacaglia/pytube.git is pulled to master.

[giacaglia]

@krapes PR is now merged to my repo. Thanks @krapes for the PR

[xislam]

ImportError: cannot import name 'quote' from 'pytube.compat' (/home/islam/projects/youtube/venv/lib/python3.7/site-packages/pytube/compat.py)

[erdincay]

I combined the two solutions in this thread as well as added support for another error:

mixins.apply_descrambler(self.player_config_args, fmt)
File "/home/kerri/Documents/Projects/crowdCounting/crowd/lib/python3.6/site-packages/pytube/mixins.py", line 97, in apply_descrambler
formats = json.loads(stream_data['player_response'])['streamingData']['formats']
KeyError: 'formats'

Pull Requests have been made to both this repo and that of @giacaglia. As of writing the git+https://github.com/giacaglia/pytube.git is pulled to master.

I hereby thank you officially in the name of the whole planet.

pip3 install git+https://github.com/giacaglia/pytube.git --upgrade