Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Tier 2 age-gate bypass implementation #575

Merged
merged 17 commits into from Jul 27, 2021
Merged

[YouTube] Tier 2 age-gate bypass implementation #575

merged 17 commits into from Jul 27, 2021

Conversation

MinePlayersPE
Copy link
Contributor

@MinePlayersPE MinePlayersPE commented Jul 27, 2021

Please follow the guide below

  • You will be asked some questions, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your pull request (like that [x])
  • Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (Co-authored with yt-dlp maintainers, also major credits to @ccdffddfddfdsfedeee)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.
Implements #574 (comment) (tier-2 age-gate bypass).

@MinePlayersPE MinePlayersPE changed the title basic agegate implementation [YouTube] Tier 2 age-gate bypass implementation Jul 27, 2021
@pukkandan pukkandan marked this pull request as ready for review July 27, 2021 08:06
@pukkandan pukkandan merged commit c0bc527 into yt-dlp:master Jul 27, 2021
@pukkandan
Copy link
Member

pukkandan commented Jul 27, 2021

An update of the current situation for reference:

There are 3 tiers of age-gate

  1. Embeddable such as HtVdAasjOgU works using both the embed and this new method
  2. Some non-embeddable videos like Tq92D6wQ1mg, SkRSXFQerZs doesn't work with embed method, but does with this new one
  3. Some doesn't work with either method XgnwCQzjau8 Cr381pDsSsA, rR8s72aouNE - Not sure why

tier 3 videos did not used to work with the old get_video_info endpoint either

There doesn't seem to be any video that works in embed workaround but fails using this new method. So embed-workaround is mostly redundant rn and is kept around only in case this method breaks

There are also some non-agegated videos like tf2U5Vyj0oU which works normally, but doesn't work with clientScreen=EMBED. These do however work if we add a thirdParty={'embedUrl': any_valid_url} to the context. It is not clear if there are any normal videos that doesn't work even with this MeJVWBSsPAY however doesnt work even with this. So we'll continue using 2 seperate requests for age-gated and normal videos

Edit:
tier 2.5. HsUATh_Nc2U works with this only if thirdParty is set and is therefore not covered by this PR. Will be implemented in #577 (5dcbfdc)

@nicolaasjan
Copy link

Some doesn't work with either method XgnwCQzjau8 Cr381pDsSsA, rR8s72aouNE - Not sure why

It looks like it may depend on your location.
E.g rR8s72aouNE can be downloaded from Canada.

@pukkandan
Copy link
Member

There are also some non-agegated videos like tf2U5Vyj0oU which works normally, but doesn't work with clientScreen=EMBED. These do however work if we add a thirdParty={'embedUrl': any_valid_url} to the context.

MeJVWBSsPAY works normally, but fails with clientScreen, and fails with thirdParty too.

thanks

E.g rR8s72aouNE can be downloaded from Canada.

works from United States as well:

POST /youtubei/v1/player HTTP/1.1
Host: [www.youtube.com](http://www.youtube.com)
Authorization: Bearer ya29.a0ARrdaM947JTxOLapm4m6ULw1kwRbc_pdQXCsifKJ-5pICHKdC...

{"context":{"client":{"clientName":"ANDROID","clientVersion":"16.05"}},
"racyCheckOk":true,"videoId":"rR8s72aouNE"}

You are using ouath though?

@pukkandan
Copy link
Member

This request works with cookies, looks like same as its always been

exactly. we are talking about unauthenticated access - like I told you before in the other post

@RealDeviil
Copy link

Hello, I have been monitoring this pull request and fork, and I have not yet had a chance to test it. But I will ASAP.

@89z I think it is clear that you can download any video, age-gated or not, with a valid login cookie file (or OAuth). youtube-dl even lets you pass in a cookie file that you can read from and write to, via:

--cookies FILE File to read cookies from and dump
cookie jar in

The challenge is to avoid authentication, if possible. (I.e., avoid generating and maintaining a login cookie file.) The need for authentication introduces additional problems/challenges in some use cases (at least, it does for me) -- so it is worth exploring an alternative that accommodates anonymous access.

At the end of the day, if there is no practical solution to download all or at least the majority of age-gated videos without auth, then so be it -- I will resort to using authentication.

@MinePlayersPE
Copy link
Contributor Author

MinePlayersPE commented Jul 29, 2021

For a more complete (and arguably less messy) overview of the situation, here are the age-gate tiers turned to embedding tiers:

  1. Videos such as HtVdAasjOgU, WomAGoEh-Ss (both age-gated) and 95% of other videos can be embedded normally (and are unaffected by the other tiers).
  2. Videos such as HsUATh_Nc2U (age-gated, affected by tier 3) and tf2U5Vyj0oU (not age-gated) can only be embedded on a public site (i.e. with a domain) due to music restrictions. context => thirdParty => embedUrl must be set to a URL for embedding to work
  3. Videos like Tq92D6wQ1mg, SkRSXFQerZs cannot be embedded (only) because of age-gating. They can be embedded with non-embedded client + clientScreen=EMBED though (which also bypasses age-gating)
  4. Videos like Cr381pDsSsA (age-gated) or MeJVWBSsPAY (not age-gated) has embedding disabled by the creator. They cannot be embedded at all

EDIT: Swapped tier 2 and 3

pukkandan added a commit that referenced this pull request Jul 30, 2021
* This allows more videos like `tf2U5Vyj0oU` to become embeddable
    See #575 (comment)
* Also added tests for all types of age-gate

Closes #581
Lesmiscore added a commit to ytdl-patched/ytdl-patched that referenced this pull request Jul 31, 2021
* 'master' of https://github.com/yt-dlp/yt-dlp:
  [adobepass] Add MSO Sling TV (#596)
  [RCTIPlus] Remove PhantomJS dependency (#595)
  [youtube] Misc cleanup (#577)
  [youtube] Add `thirdParty` to agegate clients (#577) * This allows more videos like `tf2U5Vyj0oU` to become embeddable     See yt-dlp/yt-dlp#575 (comment) * Also added tests for all types of age-gate
  [youtube] simplify and de-duplicate client definitions (#577)
  [youtube] Improve age-gate detection (#577)
  [youtube] Force `hl=en` for comments (#594)
@MinePlayersPE MinePlayersPE deleted the agegate-embedplus branch August 11, 2021 07:33
nixxo pushed a commit to nixxo/yt-dlp that referenced this pull request Nov 22, 2021
* Calling the API with `clientScreen=EMBED` allows access to most age-gated videos - discovered by @ccdffddfddfdsfedeee (yt-dlp#574 (comment))
* Adds clients: (web/android/ios)_(embedded/agegate), mweb_embedded
* Renamed mobile_web to mweb

Closes yt-dlp#574

Authored by pukkandan, MinePlayersPE
nixxo pushed a commit to nixxo/yt-dlp that referenced this pull request Nov 22, 2021
* This allows more videos like `tf2U5Vyj0oU` to become embeddable
    See yt-dlp#575 (comment)
* Also added tests for all types of age-gate

Closes yt-dlp#581
@pukkandan pukkandan mentioned this pull request Mar 25, 2022
7 tasks
Nexterr-origin added a commit to Nexterr-origin/simpleTV-YouTube that referenced this pull request Mar 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants