Add Spectrum as an Adobe Pass provider #26792

kevinoconnor7 · 2020-10-03T17:37:04Z

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

Some services no longer allow for the older services (ex. Charter_Direct) to be used and just return a generic 401 response.
Going through their login flow manually, the new provider name appears to be Spectrum. This provider seems to work even place of services that still allow for Charter_Direct to be used.

gxam · 2020-12-21T00:09:35Z

I patched 2020.12.14 with this, but it does not work:

[debug] System config: []
[debug] User config: ['--write-thumbnail', '--add-metadata', '--cookies', '~/.youtube-dl.cookies', '--download-archive', '~/.youtube-dl.archive', '--ap-mso', 'Spectrum', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--verbose', '--write-info-json']
[debug] Custom config: []
[debug] Command-line args: ['https://play.mylifetime.com/movies/dear-christmas/full-movie']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.12.14
[debug] Python version 3.9.1 (CPython) - macOS-10.15.7-x86_64-i386-64bit
[debug] exe versions: ffmpeg git-2020-11-22-9208b72, ffprobe git-2020-11-22-9208b72, rtmpdump 2.4
[debug] Proxy map: {}
[aenetworks] movies/dear-christmas/full-movie: Downloading JSON metadata
[aenetworks] 1823671875968: Downloading JSON metadata
[aenetworks] 1823671875968: Downloading Provider Redirect Page
[aenetworks] 1823671875968: Downloading Provider Login Page
ERROR: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 824, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/aenetworks.py", line 158, in _real_extract
    auth = self._extract_mvpd_auth(
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/adobepass.py", line 1509, in _extract_mvpd_auth
    provider_refresh_redirect_url = extract_redirect_url(
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/adobepass.py", line 1370, in post_form
    post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_page, 'post url', group='url')
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1019, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1010, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I threw in a breakpoint to investigate. provider_login_page_res at adobepass.py:1514 is basically just javascript, no form in the HTML:

'<!DOCTYPE html><html lang="en"><head>\n    <meta charset="UTF-8">\n    <meta name="viewport" content="width=device-width, initial-scale=1.0">\n    <meta http-equiv="X-UA-Compatible" content="ie=edge">\n    <title>TV Everywhere</title>\n    <meta http-equiv="Cache-Control" content="max-age=0, no-cache, no-store, must-revalidate">\n    <meta http-equiv="Pragma" content="no-cache">\n    <meta http-equiv="Expires" content="0">\n    <script language="JavaScript" type="text/javascript">\n        const __MIN__ = \'.min\';\n    </script>\n\n    <script type="text/javascript">\n        // IE 11 polyfills\n        !function( d ) {\n            if( !d.currentScript ){\n                var s = d.createElement( \'script\' );\n                s.src = \'https://polyfill.io/v3/polyfill.js?flags=gated&features=default%2Ces2015%2Ces5%2Ces6%2Ces7%2CObject.entries%2CArray.prototype.find%2CArray.from%2CObject.values\';\n                d.head.appendChild( s );\n            }\n        }(document)\n    </script>\n\n      <script language="JavaScript" type="text/javascript">\n          const RelayState = "...";\n          const SAMLRequest = "...";\n      </script>\n    </head>\n<body id="temporary-override" class="kite-theme-dark">\n    <div id="root"></div>\n<script type="text/javascript" src="/vendor.b7f691fb52892f76e249.chunk.js"></script><script type="text/javascript" src="/app.b7f691fb52892f76e249.bundle.js"></script>\n\n</body></html>'

I redacted RelayState and SAMLRequest assuming they contained authentication tokens or something.

I can't tell what, if anything, in this pile of JS contains the URL adobepass is looking for.

kevinoconnor7 · 2020-12-21T17:59:13Z

Hmm yeah looks like Spectrum updated their login page recently. I'll dig into it.

kevinoconnor7 · 2020-12-21T19:07:35Z

Reworked the change to get things working again. Unfortunately the new login form is a bit special as it's all dynamically loaded via JS so we need to manually recreate the flow in code.

gxam · 2020-12-22T00:30:22Z

Awesome, nbc and aenetworks are both working for me now. This fixes both #22714 and #27328

kevinoconnor7 · 2021-01-16T17:01:17Z

@dstftw Do you need me to send a provider account?

dstftw · 2021-01-16T22:36:46Z

Yes.

kevinoconnor7 · 2021-01-23T22:33:16Z

Unfortunately my Spectrum account is in a weird state and is not allowing me to change my password. Can someone else (@gxam maybe?) share a login with @dstftw?

ohmybahgosh · 2021-06-28T10:53:00Z

If a dev needs a spectrum account login, please let me know and I'll provide one post-haste!

@dstftw let me know if this is something you're still interested in fixing, and if so, I'll send over the login for spectrum

ohmybahgosh · 2021-07-02T02:24:52Z

Bumping this.. If a developer could finish this update for the SPECTRUM MSO I'd be super grateful!

I have credentials that I will send immediately to whomever is a developer willing to take this on.

ohmybahgosh · 2021-07-04T23:21:23Z

@dstftw @kevinoconnor7 any willingness to work on this? I can provide MSO login

kevinoconnor7 · 2021-07-04T23:26:30Z

@ohmybahgosh All the work is done. From my perspective this PR is ready to merge. @dstftw just needs a Spectrum login to test it with. You can probably send it to them via the email on their github profile.

Unfortunately my Spectrum account is still in a bad state and I cannot easily change the password on it.

ohmybahgosh · 2021-07-05T13:26:56Z

Python 2.7.18

./youtube-dl/youtube-dl https://www.nbc.com/making-it/video/random-crafts-of-kindness/4386591 --verbose 
[debug] System config: []
[debug] User config: [u'--ap-mso', u'Spectrum', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'--convert-subtitles=srt', u'--add-metadata', u'--write-all-thumbnails', u'--embed-thumbnail', u'--embed-subs']
[debug] Custom config: []
[debug] Command-line args: [u'https://www.nbc.com/making-it/video/random-crafts-of-kindness/4386591', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.03.14
[debug] Python version 2.7.18 (CPython) - Linux-5.10.0-kali9-amd64-x86_64-with-debian-kali-rolling
[debug] exe versions: ffmpeg 4.3.2-0, ffprobe 4.3.2-0, phantomjs 1.9.7, rtmpdump 2.4
[debug] Proxy map: {}
[NBC] 4386591: Downloading JSON metadata
[NBC] 4386591: Downloading Provider Redirect Page
[NBC] 4386591: Downloading Provider Login Page
[NBC] 4386591: Downloading SAML Response
ERROR: Unable to download JSON metadata: HTTP Error 404:  (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "./youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "./youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Python 3.9.2

✦ ❯ ./youtube-dl https://www.nbc.com/making-it/video/random-crafts-of-kindness/4386591 --verbose --cookies /root/.cookies/nbc.txt                                                         [debug] System config: []
[debug] User config: ['--ap-mso', 'Spectrum', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--convert-subtitles=srt', '--add-metadata', '--write-all-thumbnails', '--embed-thumbnail', '--embed-subs']
[debug] Custom config: []
[debug] Command-line args: ['https://www.nbc.com/making-it/video/random-crafts-of-kindness/4386591', '--verbose', '--cookies', '/root/.cookies/nbc.txt']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.03.14
[debug] Python version 3.9.2 (CPython) - Linux-5.10.0-kali9-amd64-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg 4.3.2-0, ffprobe 4.3.2-0, phantomjs 1.9.7, rtmpdump 2.4
[debug] Proxy map: {}
[NBC] 4386591: Downloading JSON metadata
[NBC] 4386591: Downloading Provider Redirect Page
[NBC] 4386591: Downloading Provider Login Page
[NBC] 4386591: Downloading SAML Response
ERROR: Unable to download JSON metadata: HTTP Error 404:  (caused by <HTTPError 404: ''>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/opt/VID_TOOLZ/youtube-dl/./youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/opt/VID_TOOLZ/youtube-dl/./youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Some services no longer allow for the older services (ex. `Charter_Direct`) to be used and just return a generic 401 response. Going through their login flow manually, the new provider name appears to be `Spectrum`. This provider seems to work even place of services that still allow for `Charter_Direct` to be used. The Spectrum login form also needs special handling since it's dynamically loaded via JS thus we cannot simply capture the POST url and hidden form fields. Instead we need to hardcode the POST URL and recreate the request manually. Thankfully the SAMLRequest and RelayState are in the HTML response.

kevinoconnor7 · 2021-07-05T16:07:30Z

oops! There was a case-sensitivity issue in the entitlement URL that I fixed on my local branch but never pushed. Should be fixed now!

ohmybahgosh · 2021-07-05T17:10:16Z

That fixed it!!

ohmybahgosh

Fixes case sensitivity issue, and now the Spectrum Ap-mso is fully functional!

From: ytdl-org/youtube-dl#26792 Co-authored by: kevinoconnor7, ohmybahgosh

jr1221 · 2021-07-08T22:10:09Z

Are the devs reviewing this PR? Spectrum is one of the most popular TV providers in the U.S., this is a great addition! I have tested by using yt-dlp and it has worked so far, fixing unable to extract post url on NBC.

ohmybahgosh · 2021-07-08T23:04:54Z

Are the devs reviewing this PR? Spectrum is one of the most popular TV providers in the U.S., this is a great addition! I have tested by using yt-dlp and it has worked so far, fixing unable to extract post url on NBC.

I hope they are...

I issued a PR for this fix over on YT-DLP and it was tested/merged already. Has been working great!

From: ytdl-org/youtube-dl#26792 Co-authored by: kevinoconnor7, ohmybahgosh

kevinoconnor7 mentioned this pull request Oct 3, 2020

AETV.com Unable to extract video url #23363

Closed

dstftw added the tv-provider-account-needed label Oct 17, 2020

kevinoconnor7 force-pushed the adobe_pass_spectrum branch from caf9e87 to 3b7c31e Compare December 2, 2020 16:37

kevinoconnor7 force-pushed the adobe_pass_spectrum branch from 3b7c31e to 5b41146 Compare December 21, 2020 19:06

kevinoconnor7 force-pushed the adobe_pass_spectrum branch from a451d74 to 678ec23 Compare March 24, 2021 02:31

kevinoconnor7 force-pushed the adobe_pass_spectrum branch from 678ec23 to f9d0a8c Compare July 5, 2021 16:05

ohmybahgosh reviewed Jul 5, 2021

View reviewed changes

ohmybahgosh mentioned this pull request Jul 6, 2021

Add Spectrum Adobe MSO - Credit to Kevinoconnor7 yt-dlp/yt-dlp#470

Merged

9 tasks

pukkandan pushed a commit to yt-dlp/yt-dlp that referenced this pull request Jul 6, 2021

[AdobePass] Add Spectrum MSO (#470)

08625e4

From: ytdl-org/youtube-dl#26792 Co-authored by: kevinoconnor7, ohmybahgosh

nixxo pushed a commit to nixxo/yt-dlp that referenced this pull request Nov 22, 2021

[AdobePass] Add Spectrum MSO (yt-dlp#470)

cda77c4

From: ytdl-org/youtube-dl#26792 Co-authored by: kevinoconnor7, ohmybahgosh

dirkf force-pushed the master branch from 01bf89e to 4c6fba3 Compare August 26, 2022 07:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Spectrum as an Adobe Pass provider #26792

Add Spectrum as an Adobe Pass provider #26792

kevinoconnor7 commented Oct 3, 2020 •

edited

gxam commented Dec 21, 2020

kevinoconnor7 commented Dec 21, 2020

kevinoconnor7 commented Dec 21, 2020

gxam commented Dec 22, 2020

kevinoconnor7 commented Jan 16, 2021

dstftw commented Jan 16, 2021

kevinoconnor7 commented Jan 23, 2021

ohmybahgosh commented Jun 28, 2021 •

edited

ohmybahgosh commented Jul 2, 2021

ohmybahgosh commented Jul 4, 2021

kevinoconnor7 commented Jul 4, 2021

ohmybahgosh commented Jul 5, 2021 •

edited

kevinoconnor7 commented Jul 5, 2021

ohmybahgosh commented Jul 5, 2021

ohmybahgosh left a comment

jr1221 commented Jul 8, 2021

ohmybahgosh commented Jul 8, 2021

Add Spectrum as an Adobe Pass provider #26792

Are you sure you want to change the base?

Add Spectrum as an Adobe Pass provider #26792

Conversation

kevinoconnor7 commented Oct 3, 2020 • edited

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

gxam commented Dec 21, 2020

kevinoconnor7 commented Dec 21, 2020

kevinoconnor7 commented Dec 21, 2020

gxam commented Dec 22, 2020

kevinoconnor7 commented Jan 16, 2021

dstftw commented Jan 16, 2021

kevinoconnor7 commented Jan 23, 2021

ohmybahgosh commented Jun 28, 2021 • edited

ohmybahgosh commented Jul 2, 2021

ohmybahgosh commented Jul 4, 2021

kevinoconnor7 commented Jul 4, 2021

ohmybahgosh commented Jul 5, 2021 • edited

Python 2.7.18

Python 3.9.2

kevinoconnor7 commented Jul 5, 2021

ohmybahgosh commented Jul 5, 2021

ohmybahgosh left a comment

Choose a reason for hiding this comment

jr1221 commented Jul 8, 2021

ohmybahgosh commented Jul 8, 2021

kevinoconnor7 commented Oct 3, 2020 •

edited

ohmybahgosh commented Jun 28, 2021 •

edited

ohmybahgosh commented Jul 5, 2021 •

edited