Fix #1060 #1067

Zeecka · 2021-07-23T13:38:28Z

Here is a proposition for #1060 . The commit name says "#1342" because I was thinking about spotDL/spotify-downloader#1342 (which is the same issue) while fixing the bug.

glubsy · 2021-07-23T13:52:11Z

Why do you change "player_response" to something else? Is this really necessary? This will break downstream projects.

Zeecka · 2021-07-23T14:05:15Z

I did so because the old endpoint doesn't work anymore and the JSON output of the new endpoint has a new key name. Which is responseContext instead of player_response. The fix didn't works without renaming it (you can try it).
~~I think it would be better if someone with better understanding of the codebase can edit the fix as needed before merging. I do not feel confident putting the web request inside of in __main__.py~~ [edit] Code looks better now.

Quick check:

Old field player_response

curl "https://www.youtube.com/youtubei/v1/player" -H 'X-Goog-Api-Key: AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H "Content-Type: application/json" -X POST -d '{"context":{"client":{"clientName":"ANDROID","clientVersion":"16.05"}},"videoId":"9bZkp7q19f0"}' | jq .player_response

New field player_response

curl "https://www.youtube.com/youtubei/v1/player" -H 'X-Goog-Api-Key: AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' -H "Content-Type: application/json" -X POST -d '{"context":{"client":{"clientName":"ANDROID","clientVersion":"16.05"}},"videoId":"9bZkp7q19f0"}' | jq .responseContext

Zeecka · 2021-07-23T14:30:26Z

It turns out that local class method InnerTube.player() already had the endpoint with predefined parameters. I cleared the custom code.

Zeecka · 2021-07-23T15:21:35Z

@glubsy I got your point. I decided to alterate the new endpoint output to ensure backward compatibility.

TeddyKahwaji

Lgtm

tfdahlin · 2021-07-24T01:06:00Z

I've been wanting to swap over to using innertube exclusively for months now (see: #1022, #994), however I keep running into the same issue which is why I haven't made the jump yet, and that's still applicable here.

Innertube can't seem to bypass any age-restrictions on videos. If you test this code against this video, you'll see the problem

Changed variable name to avoid conflict with method name

tfdahlin · 2021-07-24T02:06:34Z

I made one minor change to your code, and added some stuff to innertube. Doesn't seem to fix the problem for age-gated videos though.

…cess_token as Bearer Authorization header. Need to investigate more.

tfdahlin · 2021-07-24T05:27:48Z

I need to sleep -- the oauth flow almost works. It seems to correctly get an access_token and refresh_token, but the requests are failing with a 400 error when I try to use the token in the Authorization header. If somebody can get that working, I think this may be the best path forward.

I've been using video id DkffTDkSADI to test, with code something like this:

from pytube.innertube import InnerTube

video_id = 'DkffTDkSADI'
i = InnerTube(use_oauth=True, allow_cache=True)
i.player(video_id)

Zeecka · 2021-07-24T10:13:06Z

It looks like other project got the same issue (see ytdl-org/youtube-dl#29086 ) and didn't found workaround yet. Nice work for the oAuth. The 400 error says :

{
  "error": {
    "code": 400,
    "message": "The API Key and the authentication credential are from different projects.",
    "errors": [
      {
        "message": "The API Key and the authentication credential are from different projects.",
        "domain": "global",
        "reason": "badRequest"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Note that your oAuth implementation works with a non restricted video. I don't think there is a "good" workaround to bypass that. We can either ask the user to provide an account with API key, or register an account for pytube that might be ban.

Since the whole project is broken for now, maybe should you release the fix (without oAuth ?) and open a new issue for age restricted video ?

tfdahlin · 2021-07-24T16:22:05Z

Ty for providing the output of that 400 error, it was getting too late for me to investigate further last night, but it made it much easier to debug this morning. I got OAuth working, just need to incorporate it into the main part of the library now.

tfdahlin · 2021-07-24T17:11:30Z

I'm working on carving out some of the old code now that we're accessing very differently-formatted data, it's being finnicky about though. Hopefully will have an update this evening.

…vant.

…ed without using auth.

tfdahlin · 2021-07-24T17:46:41Z

Running into a 403 error on age-restricted videos that still needs solving. Normal videos seem to work though currently.

Zeecka · 2021-07-25T10:43:45Z

Your fix works on my computer. However, I encounter a 403 error for a video that has no age restriction: _Tr1lrd4co0, dX2497jB-Ag, ZZxOdjay8Kg, GopLL_S8wzI. These video are longer than 20min however it seems to work on 8h video such as RDfjXj5EGqI. Here is my output:

from pytube import YouTube
yt = YouTube('http://youtube.com/watch?v=_Tr1lrd4co0')
yt.streams.filter(progressive=True, file_extension='mp4').first().download()

Error:

Traceback (most recent call last):
  File "/tmp/pytube/test.py", line 3, in <module>
    yt.streams.filter(progressive=True, file_extension='mp4').first().download()
  File "/tmp/pytube/pytube/streams.py", line 258, in download
    for chunk in request.stream(
  File "/tmp/pytube/pytube/request.py", line 157, in stream
    response = _execute_request(
  File "/tmp/pytube/pytube/request.py", line 37, in _execute_request
    return urlopen(request, timeout=timeout)  # nosec
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

tfdahlin · 2021-07-28T23:54:37Z

I've set this branch to default to the android client, in spite of the fact that we do lose some stream types as a result. Worth testing to see if there are any further 403 errors, but in my very brief testing I couldn't produce any. I've also bundled some miscellaneous bugfixes for other issues and linked them into this.

W/r/t signature calculation, I'm seeing something a little unusual in how signatures are being calculated:

Before: AAOSAOq0QJ8wRQIhAMy92pITfVpcKrf_vKGx9_1zKSvSYY1Aq3ZRcevK05EFAiBS-n0vzH2JId4lARhvbaNFL_aa7I7iJEVp9hym1HpE-Q==
After:  V=Q-EpH1myh9pFEJi7I7aa_L=NabvhRAl4dIJ2Hzv0n-SBiAFE50KvecRZ3qA1YYSvSKz1_9xGKv_frKcpVfTIp29yMAhIQRw8JQ0qOASOAA

It looks like the calculation is incomplete, and I'm unsure why that is. When looking at dev tools in Chrome, the signature for the video typically resembles a b64-encoded string, with the == at the end as padding. Our calculations seem to have some equal signs in the middle, which shouldn't really happen. I'll have to look into what's contained in the data, maybe we can calculate this in a different way.

MinePlayersPE · 2021-07-29T05:43:31Z

Hopefully not off-topic, since I saw that age-gating is a problem in this PR I would like to point to the new bypass method: yt-dlp/yt-dlp#574 (comment) which should have the same success rate as the old method

Zeecka · 2021-07-29T07:39:47Z

Worth testing to see if there are any further 403 errors, but in my very brief testing I couldn't produce any.

Same for me, nice job 👍🏻 .

Do you have a video sample where the signature is not already provided ?

tfdahlin · 2021-07-29T21:37:57Z

Hopefully not off-topic, since I saw that age-gating is a problem in this PR I would like to point to the new bypass method: yt-dlp/yt-dlp#574 (comment) which should have the same success rate as the old method

Definitely not off-topic, thanks for the tip! I've incorporated that bypass into this patch. Pytube will default to the normal android client. If it fails to retrieve the stream data, it will attempt to fall back to using that bypass instead. The videos that I've been testing against seem to work correctly for me, but would appreciate some additional testing.

I'm going to note once again so it doesn't get lost:
This patch defaults to using the ANDROID client because the signatureCipher isn't being calculated correctly. Because it's using the ANDROID client, some stream types are not available that are available when using the WEB client. A notable stream type that I saw was missing on the video I tested against was itag 22, which is the 720p progressive stream (typically the highest quality progressive stream available on a video). Ideally, the cipher calculation should be fixed so we can default to the WEB client again, but I think that should be pushed into a different issue/PR.

MinePlayersPE · 2021-07-30T04:48:24Z

pytube/__main__.py

+
+        # If we still can't access the video, raise an exception
+        # (tier 3 age restriction)
+        if playability_status == 'LOGIN_REQUIRED':


Note: Tier 3 videos will return an UNPLAYABLE status on embed clients instead (as they're likely just videos with embed disabled by creator)

GPRicci · 2021-07-31T00:47:48Z

I have been doing some silly tests today with the fixes in this branch, and I ran into an error when trying to retrieve the captions of a video:

Traceback (most recent call last):
  File "C:\Users\gaspa\pytube_tests\dl.py", line 14, in <module>
    c = yt.captions
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 299, in captions
    return pytube.CaptionQuery(self.caption_tracks)
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in caption_tracks
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in <listcomp>
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\captions.py", line 22, in __init__
    self.name = caption_track["name"]["simpleText"]
KeyError: 'simpleText'

I'm not sure if this error arose with this fix.

tfdahlin · 2021-08-01T00:04:48Z

I have been doing some silly tests today with the fixes in this branch, and I ran into an error when trying to retrieve the captions of a video:

Traceback (most recent call last):
  File "C:\Users\gaspa\pytube_tests\dl.py", line 14, in <module>
    c = yt.captions
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 299, in captions
    return pytube.CaptionQuery(self.caption_tracks)
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in caption_tracks
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in <listcomp>
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\captions.py", line 22, in __init__
    self.name = caption_track["name"]["simpleText"]
KeyError: 'simpleText'

I'm not sure if this error arose with this fix.

Looking into a fix for this now. I think sometimes caption details return in two different formats, not sure what causes it.

GPRicci · 2021-08-01T21:38:24Z

I have been doing some silly tests today with the fixes in this branch, and I ran into an error when trying to retrieve the captions of a video:

Traceback (most recent call last):
  File "C:\Users\gaspa\pytube_tests\dl.py", line 14, in <module>
    c = yt.captions
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 299, in captions
    return pytube.CaptionQuery(self.caption_tracks)
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in caption_tracks
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in <listcomp>
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\captions.py", line 22, in __init__
    self.name = caption_track["name"]["simpleText"]
KeyError: 'simpleText'

I'm not sure if this error arose with this fix.

Looking into a fix for this now. I think sometimes caption details return in two different formats, not sure what causes it.

With your commit, I can now download XML captions, however, conversion to SRT fails. By exploring the XML files and reading the xml_caption_to_srt function, I think the problem is that YouTube changed the format.
Here is an example:

<?xml version="1.0" encoding="utf-8" ?><timedtext format="3">
<body>
<p t="2254" d="2872">FRIDAY</p>
<p t="33434" d="1193">IT IS TIME FOR THE BOYS</p>
</body>
</timedtext>

So now, the root element timedtext contains just one child named body. So the iteration should be done for the elements of body and not root. Also, the caption elements have different attribute names: t instead of start and d instead of dur for duration.

…lity no longer works.

tfdahlin · 2021-08-03T22:58:30Z

I have been doing some silly tests today with the fixes in this branch, and I ran into an error when trying to retrieve the captions of a video:
Traceback (most recent call last):
  File "C:\Users\gaspa\pytube_tests\dl.py", line 14, in <module>
    c = yt.captions
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 299, in captions
    return pytube.CaptionQuery(self.caption_tracks)
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in caption_tracks
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\__main__.py", line 291, in <listcomp>
    return [pytube.Caption(track) for track in raw_tracks]
  File "C:\Users\gaspa\AppData\Local\Programs\Python\Python39\lib\site-packages\pytube\captions.py", line 22, in __init__
    self.name = caption_track["name"]["simpleText"]
KeyError: 'simpleText'
I'm not sure if this error arose with this fix.
Looking into a fix for this now. I think sometimes caption details return in two different formats, not sure what causes it.
With your commit, I can now download XML captions, however, conversion to SRT fails. By exploring the XML files and reading the xml_caption_to_srt function, I think the problem is that YouTube changed the format.
Here is an example:
<?xml version="1.0" encoding="utf-8" ?><timedtext format="3">
<body>
<p t="2254" d="2872">FRIDAY</p>
<p t="33434" d="1193">IT IS TIME FOR THE BOYS</p>
</body>
</timedtext>
So now, the root element timedtext contains just one child named body. So the iteration should be done for the elements of body and not root. Also, the caption elements have different attribute names: t instead of start and d instead of dur for duration.

It looks like this is inconsistent between auto-generated and user-provided captions. I'll work on a fix for captions as a separate PR, rather than trying to fix that in this one.

tfdahlin · 2021-08-03T23:01:16Z

Now that tests have been fixed, I'm going to merge this and get pypi updated.

tfdahlin · 2021-08-03T23:12:01Z

Pypi should be updated. python -m pip install --upgrade pytube

Zeecka added 2 commits July 23, 2021 15:29

Fix pytube#1342

a59b086

remove debug print

7fe0f6f

remove test file

d72cb49

Use InnerTube.player() class instead of custom code.

ad731d4

Replace responce key for backward compatibility.

a3c3c3b

Pep8 compliant

543ee37

TeddyKahwaji approved these changes Jul 23, 2021

View reviewed changes

tfdahlin added 2 commits July 23, 2021 21:34

Update __main__.py

f7612e0

Changed variable name to avoid conflict with method name

Added some additional base parameters for innertube requests.

5efda2a

tfdahlin linked an issue Jul 24, 2021 that may be closed by this pull request

HTTP Error 404: Not Found when trying get stream #1060

Closed

Nearly complete oauth flow. There's now a 400 error when including ac…

5158146

…cess_token as Bearer Authorization header. Need to investigate more.

tfdahlin linked an issue Jul 24, 2021 that may be closed by this pull request

Sign in YouTube account #1040

Closed

Zeecka mentioned this pull request Jul 24, 2021

HTTP Error 404: Not Found when trying get stream #1060

Closed

Oauth works. Just needed to not use API key + Oauth.

fbd2762

tfdahlin added 4 commits July 24, 2021 13:43

Change default to allow caching.

aed5bed

Additional args for YouTube, carve out old code that's no longer rele…

d379c86

…vant.

Add exception for age-restricted videos which can no longer be access…

9f830aa

…ed without using auth.

Changes to account for changes to YouTube object

308a270

Added renderer catch -- fixes pytube#1068

47e1997

tfdahlin linked an issue Jul 28, 2021 that may be closed by this pull request

[BUG] Additional renderer in search results #1068

Closed

Additional channel name support for URL-encoded names.

734aff7

tfdahlin linked an issue Jul 28, 2021 that may be closed by this pull request

Support for Not English channel name(/c/{channel name}) #1070

Closed

Zeecka and others added 2 commits July 29, 2021 14:06

Flake8

0b68d7f

Use embed player as a fallback for age-restricted videos.

ef7a163

MinePlayersPE reviewed Jul 30, 2021

View reviewed changes

tfdahlin mentioned this pull request Jul 31, 2021

[BUG] HTTP Error 410: Gone #1076

Closed

tfdahlin linked an issue Jul 31, 2021 that may be closed by this pull request

[BUG] HTTP Error 410: Gone #1076

Closed

tfdahlin added 2 commits July 31, 2021 20:16

Prevent cached tokens from being accidentally uploaded.

8e3612e

Fix captions bug.

9b112ee

tfdahlin linked an issue Aug 1, 2021 that may be closed by this pull request

[BUG] Captions #1074

Closed

Fix tier 3 check.

82076c2

olepoeschl mentioned this pull request Aug 2, 2021

[BUGcan't get video info #1078

Closed

tfdahlin linked an issue Aug 3, 2021 that may be closed by this pull request

[BUG] #1079

Closed

tfdahlin added 3 commits August 3, 2021 18:22

Updated test mocks, removed region-locked test because that functiona…

721a240

…lity no longer works.

Actually update test mocks.

baf799a

Fix tests, strip out some code that's no longer used from main.

8d2e9a7

tfdahlin merged commit fc9aec5 into pytube:master Aug 3, 2021

JNYH added a commit to JNYH/pytube that referenced this pull request May 16, 2022

Fix pytube#1060 (pytube#1067)

c14c460

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #1060 #1067

Fix #1060 #1067

Zeecka commented Jul 23, 2021 •

edited

glubsy commented Jul 23, 2021 •

edited

Zeecka commented Jul 23, 2021 •

edited

Zeecka commented Jul 23, 2021

Zeecka commented Jul 23, 2021

TeddyKahwaji left a comment

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

Zeecka commented Jul 24, 2021 •

edited

tfdahlin commented Jul 24, 2021 •

edited

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

Zeecka commented Jul 25, 2021

tfdahlin commented Jul 28, 2021 •

edited

MinePlayersPE commented Jul 29, 2021 •

edited

Zeecka commented Jul 29, 2021

tfdahlin commented Jul 29, 2021

MinePlayersPE Jul 30, 2021 •

edited

tfdahlin Aug 1, 2021

GPRicci commented Jul 31, 2021

tfdahlin commented Aug 1, 2021

GPRicci commented Aug 1, 2021

tfdahlin commented Aug 3, 2021

tfdahlin commented Aug 3, 2021

tfdahlin commented Aug 3, 2021

Fix #1060 #1067

Fix #1060 #1067

Conversation

Zeecka commented Jul 23, 2021 • edited

glubsy commented Jul 23, 2021 • edited

Zeecka commented Jul 23, 2021 • edited

Zeecka commented Jul 23, 2021

Zeecka commented Jul 23, 2021

TeddyKahwaji left a comment

Choose a reason for hiding this comment

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

Zeecka commented Jul 24, 2021 • edited

tfdahlin commented Jul 24, 2021 • edited

tfdahlin commented Jul 24, 2021

tfdahlin commented Jul 24, 2021

Zeecka commented Jul 25, 2021

tfdahlin commented Jul 28, 2021 • edited

MinePlayersPE commented Jul 29, 2021 • edited

Zeecka commented Jul 29, 2021

tfdahlin commented Jul 29, 2021

MinePlayersPE Jul 30, 2021 • edited

Choose a reason for hiding this comment

tfdahlin Aug 1, 2021

Choose a reason for hiding this comment

GPRicci commented Jul 31, 2021

tfdahlin commented Aug 1, 2021

GPRicci commented Aug 1, 2021

tfdahlin commented Aug 3, 2021

tfdahlin commented Aug 3, 2021

tfdahlin commented Aug 3, 2021

Zeecka commented Jul 23, 2021 •

edited

glubsy commented Jul 23, 2021 •

edited

Zeecka commented Jul 23, 2021 •

edited

Zeecka commented Jul 24, 2021 •

edited

tfdahlin commented Jul 24, 2021 •

edited

tfdahlin commented Jul 28, 2021 •

edited

MinePlayersPE commented Jul 29, 2021 •

edited

MinePlayersPE Jul 30, 2021 •

edited