Add support for Adobe Pass Auth TV Provider YouTubeTV

[aretwo]

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.12.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.12.14

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

you http://watchdisneyxd.go.com/atomic-puppet/video/vdka4074861/01/26-finale-a-finale-b --ap-mso YouTubeTV --ap-username PRIV --ap-password PRIV -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['http://watchdisneyxd.go.com/atomic-puppet/video/vdka4074861/01/26-finale-a-finale-b', '--ap-mso', 'YouTubeTV', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '-v']
Usage: you [OPTIONS] URL [URL...]

you: error: Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers

---

Would like to request support for YouTubeTV as an Adobe Pass MSO option. Running --ap-mso does not recognize YouTubeTV as an option that can be used. Going by the providers list https://sp.auth.adobe.com/adobe-services/config/ABC it should be able to use the service. If an account is needed, I do have an invite so access can be had.

[micahfocht]

Want to chime in here. I'm not sure exactly how to do this, but I may be able to help a bit. The login page for youtube tv is the same as the login for the standard youtube login. Hopefully, a good deal of that code could be reused.

[thieverycorp]

Looking forward to this feature

[arghc]

Is a YoutubeTV account still needed? I've got an active account...

[joeynotjoe]

I am really looking forward to this, as well.

[aretwo]

I'm still hoping that this will be worked on. Keeping an account saved in case somebody wants to pick it up.

[arcanez]

#15926

[go2tom42]

#15926

Being able to download from tv.youtube.com and adding YouTube TV as a MSO is two separate issues

I blindly fiddled with adobepass.py and got nowhere. Maybe someone with skills will give it a shot

When you login manually in a browser it goes to this order

1. https://youtube.auth-gateway.net/saml/module.php/authbypass/lastbookend.php?AuthState
2. https://accounts.youtube.com/accounts/SetSID
3. https://accounts.youtube.com/accounts/SetSID?ssdc
4. https://accounts.google.com/signin/v2/challenge/pwd?client_id
5. https://accounts.google.com/signin/oauth/identifier?client_id
6. https://youtube.auth-gateway.net/saml/module.php/authbypass/firstbookend.php?AuthState
7. https://sp.auth.adobe.com/adobe-services/authenticate/saml
8. Video plays

This might be useless
https://youtu.be/C-g4Y1pUE24

[dellis87]

Is a YoutubeTV account still needed? I've got an active account...

I've got one as well.

[connorwalks]

Is there any potential update on this? Would love to use youtube-dl instead of browser plugins but unfortunately many sites won't simply accept a cookie and require MSO so I can't download from there. Haven't seen any update on this in months

[handcraftedbits]

Like others I've been unable to use YouTube TV as a TV Provider, but I've found a really simple solution that lets me use my login cookies, and hopefully it will point the youtube-dl devs (I'm not much of a Python person myself...) in the direction of a "cookie ap-mso provider" approach that will work for (hopefully) all unsupported providers:

1. Log in to the on demand video site (e.g., Disney Junior).
2. From the same tab, export all cookies, not just the ones for the current tab, using a cookie exporter plugin. I use this one in Chrome. Save it as e.g., cookies.txt. I find it helpful to do this in an Incognito/private browser window as it keeps the number of cookies down to the bare minimum ones (I've seen youtube-dl complain about the format of cookies.txt based on certain unrelated cookies).
3. Clone the youtube-dl repo.

This is where "your mileage may vary." I can detail the change I made for Disney (go) sites, and I presume it's very similar for other extractors. Later on I'll discuss what I think is the "real" solution -- this is just a hack that'll get you going until it's properly fixed.

4. Edit youtube_dl/extractor/go.py.
5. Search for self._extract_mvpd_auth. This is where youtube-dl is attempting to sign in to the Adobe Pass site. We're going to bypass that check because the cookies we're providing should be sufficient, it's merely that youtube-dl isn't considering that we can simply bypass the login if the cookies are available. Here's a snippet of what it looks like in go.py to give some context:

if video_data.get('accesslevel') == '1':
    requestor_id = site_info.get('requestor_id', 'DisneyChannels')
    resource = site_info.get('resource_id') or self._get_mvpd_resource(
        requestor_id, title, video_id, None)
    auth = self._extract_mvpd_auth(
        url, video_id, requestor_id, resource)
    data.update({
        'token': auth,
        'token_type': 'ap',
        'adobe_requestor_id': requestor_id,
    })
else:
    self._initialize_geo_bypass({'countries': ['US']})

What this block is doing, as best as I can tell, is seeing if a video is considered "protected" (accesslevel == '1') and starting the SSO process. Otherwise, it just continues on with the download. What gave me the idea to look here is that youtube-dl has no problems downloading videos from Disney if they aren't protected (and each show has a couple of them available at any time), so the approach here is to simply make it appear as if all videos are unprotected. To do so:

6. Change if video_data.get('accesslevel') == '1' to something that should never be true e.g., if video_data.get('accesslevel') == '1000'.
7. Delete youtube_dl/extractor/go.pyc (might not be necessary, just me being paranoid I guess).
8. Run make from the top of the source tree.
9. Make sure you don't have any credentials cached -- rm -rf ~/.cache/youtube-dl
10. (From the top of the source tree) run ./youtube-dl --cookies <path_to_cookies.txt> <url, etc.>
11. Success!

I can confirm that this works with YouTube TV as my provider and the Disney video sites (Disney, Disney Junior, etc.). This is obviously a hack, but it points us in the direction of...

A Proper Solution

What I "think" the proper solution for providers like YouTube TV that aren't supported (and apparently won't be any time soon) is to make a generic "cookie" MSO provider, e.g.:

youtube-dl --ap-mso cookie --cookies <path_to_cookies.txt> <url, etc.>`

This flag would be used to indicate that _extract_mvpd_auth etc. shouldn't be called because the provided cookies are sufficient to bypass authorization checks.

Alternately, perhaps _extract_mvpd_auth could be smart enough to see if the proper cookies are available and use them.

Thoughts? I can certainly try to make a PR for this, though Python isn't really my choice of language.

[harryt04]

@handcraftedbits I was very eager to see if your solution worked. I followed the steps exactly, triple checked, even. But when I try it, this happens. Can you see if you can download this URL from your machine after performing those steps? Perhaps there was another change in youtube-dl that broke your workaround?

https://tv.youtube.com/watch/yOGRebatQvg

./youtube-dl --cookies ./yttv.txt https://tv.youtube.com/watch/yOGRebatQvg
[generic] yOGRebatQvg: Requesting header
WARNING: Falling back on generic information extractor.
[generic] yOGRebatQvg: Downloading webpage
[generic] yOGRebatQvg: Extracting information
ERROR: Unsupported URL: https://tv.youtube.com/watch/yOGRebatQvg

[dellis87]

You will need to go to the individual provider website. It only works for me on disney and not abc but it’s something!

…

Sent from my iPhone

On Feb 6, 2020, at 7:11 PM, Harrison Thomas ***@***.***> wrote:  @handcraftedbits I was very eager to see if your solution worked. I followed the steps exactly, triple checked, even. But when I try it, this happens. Can you see if you can download this URL from your machine after performing those steps? Perhaps there was another change in youtube-dl that broke your workaround? https://tv.youtube.com/watch/yOGRebatQvg ./youtube-dl --cookies ./yttv.txt https://tv.youtube.com/watch/yOGRebatQvg [generic] yOGRebatQvg: Requesting header WARNING: Falling back on generic information extractor. [generic] yOGRebatQvg: Downloading webpage [generic] yOGRebatQvg: Extracting information ERROR: Unsupported URL: https://tv.youtube.com/watch/yOGRebatQvg — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[harryt04]

@dellis87 How do you determine the provider for a show? Like I'm trying to download the bachelor, for example. Is there a way to tell on YoutubeTV's website to see who the 'individual provider' is?

[dellis87]

It would be whatever channel that show is on. Like the bachelor is ABC I think.

…

Sent from my iPhone

On Feb 6, 2020, at 8:18 PM, Harrison Thomas ***@***.***> wrote:  @dellis87 How do you determine the provider for a show? Like I'm trying to download the bachelor, for example. Is there a way to tell on YoutubeTV's website to see who the 'individual provider' is? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[handcraftedbits]

@harryt04 , in my example I was referring to using YouTube TV credentials to download a Disney video stream. It's very likely that youtube-dl doesn't support video streams originating from YouTube TV itself, and that's why you're getting an error. The example I listed is only valid for the go extractor, which is supposed to handle the Disney family of sites, and I would think ABC as well, but apparently not. Like I mentioned in my workaround, there really needs to be a more generic solution (like a "cookie" MSO provider), but I'm not familiar enough with the youtube-dl internals to take a stab at that.

@dellis87 , just curious, but what extractor does youtube-dl say it's using when you download from an ABC site? Is it not go?

[dellis87]

It is go.py. It gives error about not being able to find the files required for download or something like that.

[Sipherdrakon]

So any progress, over the 3 years this has been an issue?

[alexmerm]

So I just spent way too much time on this and finally got it working (at least for nbc.com) but it's pretty ineffieicient.
Basically, what I did was replace the _extract_mvpd_auth call and manually copy and pasting the "Short Auth" code in. Step by Step below:

1. Search the extractor you're using for "_extract_mvpd_auth" and comment it out
2. Navigate to the content you want to download in Chrome and
   a) save the cookies from the page (not 100% sure this is necessary but I did it)
   b) open the Network Tab of Developer Tools (Options->More Tools->Developer tools in chrome)
   c) and filter the requests for "ShortAuthorize" Find the latest request and copy the response
3. set whatever _extract_mvpd_auth was stored in to the copied "ShortAuthorize" response

so for example in nbc.py (specifically the nbcentertainment extractor), my code looks like this

       if video_data.get('locked'):
            resource = self._get_mvpd_resource(
                video_data.get('resourceId') or 'nbcentertainment',
                title, video_id, video_data.get('rating'))
            #query['auth'] = self._extract_mvpd_auth(url, video_id, 'nbcentertainment', resource)
            query['auth'] = '<signatureInfo>SOMETHINGENCRYPED<signatureInfo><authToken><sessionGUID>SOMETHINGENCRYPED</sessionGUID><requestorID>nbcentertainment</requestorID><resourceID><![CDATA[<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>e</title><item><title><![CDATA[[object Object]]]]]>><![CDATA[</title><guid>4004499</guid><media:rating scheme="urn:v-chip">TV-14</media:rating></item></channel></rss>]]></resourceID><ttl>420000</ttl><issueTime>2021-06-05 21:34:39 -0700</issueTime><mvpdId>YouTubeTV</mvpdId></authToken>'

For more detailed instructions on the coding side of this look at #14984 (comment)

Also: You have to repeat this process for each item you're downloading, because AFAIK the "Short Auth" code is unique to the media item

[handcraftedbits]

@alexmerm I think, similar to when I was using the go extractor, all you have to do is get the if video_data.get('locked'): block to bypass and then it'll just use cookies. Might want to give that a shot. It would confirm that what we really need is a "cookie" AP-MSO provider.

[Frekvens1]

I've been digging a little into the code, and I managed to get something working.
I want to thank @handcraftedbits for the great idea.

Note, I've only tested this with Hulu on History.com, so I'm unsure about its effect on other sites.
I also used a fork of youtube-dl, but it should work the same (https://github.com/yt-dlp/yt-dlp)

I hope this helps or inspires some of you :)

Patch

The file in question: youtube-dl/extractor/adobepass.py

There is a dictionary variable MSO_INFO = {}, add the following entry to it:

'Cookie': {
    'name': 'COOKIE'
},

Further down in the file (Around line 1500), you should find some if statements about:

if mso_id == 'Comcast_SSO':
elif mso_id == 'Philo':

I simply inserted this elif in here:

elif mso_id == 'Cookie':
    pass

The session is already authenticated in my browser, and by saving the cookies to a file, I'm actively moving my authenticated session from the browser to this program. That means no login logic is required, you're already logged in.

Getting cookies.txt

I've used Firefox with this plugin: https://github.com/lennonhill/cookies-txt.
I opened a private window and authenticated with all required services, then I saved the cookies for all sites (Important).
Since I've opened in private mode, only the cookies for the session are saved, not every site you've visited before.

Usage

You could either import the whole project directly or compile it to an .exe

If directly used (Like I do), put this in your 'ydl_opts' variable on initialization:

ydl_opts = {
    'ap_mso': 'Cookie',
    'cookiefile': 'cookies.firefox-private.txt'
}

I believe the arguments for the .exe file should be exactly like @handcraftedbits mentioned:
youtube-dl --ap-mso Cookie --cookies <path_to_cookies.txt> <url, etc.>

[handcraftedbits]

@Frekvens1 good to hear that someone with a better understanding of the codebase was able to get the cookie solution working! I think youtube-dl is more or less "dead" at this point and I've actually moved on to yt-dlp myself. Do you have a PR open in that project? Would love to see it implemented. Thanks again!

[Frekvens1]

@handcraftedbits, thanks for the feedback! I don't have the best understanding of the codebase, but thanks to you I managed to figure out something ;)

I don't have any PR open in that project, maybe we should? I would love for this feature to be default!

[Sipherdrakon]

Just because that specific link is dead does not mean the issue is resolved. Smh.

…

On Sat, May 21, 2022, 11:02 AM Zombo ***@***.***> wrote: Issue should be closed, URL is dead: https://disneynow.com/shows/atomic-puppet/season-01/episode-26-finale-a-finale-b/vdka4074861 — Reply to this email directly, view it on GitHub <#14984 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APLSCXXF6MAUTR44KEGBQGDVLD3ITANCNFSM4EIGE7YA> . You are receiving this because you commented.Message ID: ***@***.***>

[Sipherdrakon]

89z wrote: > Issue should be closed, URL is dead:
https://disneynow.com/shows/atomic-puppet/season-01/episode-26-finale-a-finale-b/vdka4074861

@Sipherdrakon so you wish to see this implemented, but you're not willing to even put for the small effort of supplying even a single example?

Whoa there Kemosabe, don't get your panties in a bunch. I was replying to the fact that you wanted this closed simply because the 4 YEAR OLD test link is dead. That is not and should never be a viable criteria to close an issue.

So let's slow your role and take a step back. I really have no preference for or against this being implemented, so "wishing" for it to be implemented is a stretch, at best. And to answer your very pretentious question, the answer is NO. I can't put forth any effort to provide a "single" example because again . . . I . . . .don't . . . .have . . . . YoutubeTV.

PS Pretty sure pukkandan already has a potential fix within yt-dlp to cover this if it can work with cookies.