Comcast_SSO failure with aetv

[jellofan]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.12.22. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2016.12.22

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones (bugs 11257 & 10803 seem related but not identical)

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl --ap-mso Comcast_SSO --ap-username <omitted> --ap-password <omitted> --verbose http://www.aetv.com/shows/going-si-ral/season-1/episode-6

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--ap-mso', u'Comcast_SSO', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'--verbose', u'http://www.aetv.com/shows/going-si-ral/season-1/episode-6']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.22
[debug] Python version 2.7.6 - Linux-3.13.0-106-generic-i686-with-Ubuntu-14.04-trusty
[debug] exe versions: none
[debug] Proxy map: {}
[aenetworks] going-si-ral/season-1/episode-6: Downloading webpage
[aenetworks] 838677059721: Downloading JSON metadata
[aenetworks] 838677059721: Downloading Provider Redirect Page
[aenetworks] 838677059721: Logging in
[aenetworks] 838677059721: Retrieving Session
ERROR: Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 405, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

---

---

Description of your issue, suggested solution and other information

Unable to download aetv.com videos using Comcast SSO. I tried several videos (including the one with --verbose info attached)

[Mr-Jake]

This was fixed by @outcoldman: #11258
I compiled youtube-dl with the fix and Comcast_SSO works with aetv.

Wait till it gets merged, or you could always compile it yourself.

[jellofan]

I pulled the repo, applied the patch, compiled, and still have the issue. I think this is something different.
Using --write-pages I see the page returned after login is a "Recovery Options" page.

Redacted version is here (removed e-mail and Name)
recovery_options_redacted.txt

[Mr-Jake]

This is what I get with the applied patch and your example link. I compiled youtube-dl 2016-12-22.

C:\youtube-dl\youtube-dl.exe "http://www.aetv.com/shows/going-si-ral/season-1/episode-6" --ap-mso Comcast_SSO --ap-username <omitted> --ap-password  <omitted> -F
[aenetworks] going-si-ral/season-1/episode-6: Downloading webpage
[aenetworks] 838677059721: Downloading JSON metadata
[aenetworks] 838677059721: Downloading Provider Redirect Page
[aenetworks] 838677059721: Confirming auto login
[aenetworks] 838677059721: Retrieving Session
[aenetworks] 838677059721: Retrieving Authorization Token
[aenetworks] 838677059721: Retrieving Media Token
[aenetworks] 838677059721: Downloading SMIL data
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[aenetworks] 838677059721: Checking video URL
[info] Available formats for 838677059721:
format code  extension  resolution note
http-162     mp4        320x180     162k
http-326     mp4        320x180     326k
http-526     mp4        448x252     526k
http-826     mp4        512x288     826k
http-1126    mp4        640x360    1126k
http-1426    mp4        768x432    1426k
http-1926    mp4        960x540    1926k
http-2618    mp4        1280x720   2618k  (best) 

I downloaded the youtube-dl source code. Found and opened the file adobepass.py. Copy and pasted in the added lines of code by outcoldman. Then compiled and that solved my Comcast issues. I'm using Windows not Linux.

Perhaps it is a different issue. Just wanted to point you to the fix in case it would help.

[jellofan]

I think my issue is different since I see the expected:

[aenetworks] 838677059721: Downloading Provider Redirect Page
[aenetworks] 838677059721: Logging in
[aenetworks] 838677059721: Retrieving Session

And I see my name & personal (non-comcast) e-mail in the login.comcast.net_login.dump. So I'm fairly confident I've been authenticated.

From an internet search it seems that in some cases Comcast presents a Recovery Options page immediately after logging in (see: http://forums.xfinity.com/t5/Email-Web-Browsing/Recvery-Options-Page/td-p/2687138). What is strange is that I don't see this page when using a browser, just when using youtube-dl.

The recovery options page (which I previously attached) contains a form, and one of the buttons is "askMeLater", so I think the solution to this issue is in adobepass.py after the login form is posted to add logic to check for the recovery page, i.e. just before the check for the "Resume" button, and submit the "askMeLater" form.

But what I'm unsure of is how to populate the "data-tracking" portion of the button:

<button 
	class="secondary button cancel" 
	type="submit"
	id="askMeLater" 
	name="_eventId"
 	value="askMeLater"
 	
 	
 	data-tracking
 >Ask me later</button>

[ForTehLose]

@jellofan so I know this isn't a permanent fix, but your issue is with the recovery options page. I had the same issue, if you have previously signed in and clicked remind me later you can't review the page on that user agent anymore. Comcast simply wont prompt you. A short term fix is to try signing in with a different browser, and thus a different agent, and verify your recovery options.

It will no longer prompt you on any user agents including the one from youtubedl

[joetex72]

@Mr-Jake and jellofan I would like to test this on AT&T network, could you point me to instructions on how to make the youtube-dl.exe for Windows after I make the changes?

[GordLacey]

I'm having the same problem (with 2018.04.03), but on Cartoon Network. Could it be that there's a CAPTCHA on the Comcast login page that's not being filled out?

youtube-dl --ap-mso Comcast_SSO --ap-username PRIVATE@comcast.net --ap-password PRIVATE https://www.cartoonnetwork.com/video/nexo-knights/weekend-at-halberts-episode.html --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--ap-mso', u'Comcast_SSO', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'https://www.cartoonnetwork.com/video/nexo-knights/weekend-at-halberts-episode.html', u'--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.04.03
[debug] Python version 2.7.10 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.1, avprobe 12.1, ffmpeg 3.3.3, ffprobe 3.3.3
[debug] Proxy map: {}
[CartoonNetwork] weekend-at-halberts: Downloading webpage
[CartoonNetwork] 598df84d240214953c1bb13eaeef2e369a59fb46: Downloading XML
[CartoonNetwork] 598df84d240214953c1bb13eaeef2e369a59fb46: Downloading Provider Redirect Page
[CartoonNetwork] 598df84d240214953c1bb13eaeef2e369a59fb46: Logging in
[CartoonNetwork] 598df84d240214953c1bb13eaeef2e369a59fb46: Retrieving Session
ERROR: Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 519, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2199, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[tv21]

This was fixed by @outcoldman: #11258
I compiled youtube-dl with the fix and Comcast_SSO works with aetv.

Wait till it gets merged, or you could always compile it yourself.

Actually that fix stopped working due to a minor URL change. For those running from source code or compiling your own, the quick fix for Comcast Xfinity is to use this version of adobepass.py:

https://raw.githubusercontent.com/bhamboss/youtube-dl/patch-1/youtube_dl/extractor/adobepass.py

That is from https://github.com/bhamboss/youtube-dl/blob/patch-1/youtube_dl/extractor/adobepass.py and I can confirm this works with Xfinity's new login if you are coming in from an Xfinity IP address.

Just to be clear, Xfinity authenticates differently if you are coming in from one of their IP addresses vs. if you are coming in from the IP of some other provider. I tested the above patch coming from an Xfinity IP. I did NOT test it from another provider's IP.

If I run diff -u on the original and patched files I get this:

$ diff -u adobepass.py.original adobepass.py.patched
--- adobepass.py.original	2018-10-17 01:22:56.000000000 -0400
+++ adobepass.py.patched	2019-04-23 08:53:36.000000000 -0400
@@ -1433,6 +1433,14 @@
                             provider_redirect_page, 'oauth redirect')
                         self._download_webpage(
                             oauth_redirect_url, video_id, 'Confirming auto login')
+                    elif 'automatically signed in with' in provider_redirect_page:
+                        # Seems like comcast is rolling up new way of automatically signing customers
+                        oauth_redirect_url = self._html_search_regex(
+                            r'continue:\s*"(https://oauth.xfinity.com/oauth/authorize\?.+)"',
+                            provider_redirect_page, 'oauth redirect (signed)')
+                        # Just need to process the request. No useful data comes back
+                        self._download_webpage(
+                            oauth_redirect_url, video_id, 'Confirming auto login')
                     else:
                         if '<form name="signin"' in provider_redirect_page:
                             provider_login_page_res = provider_redirect_page_res

[sufr3ak]

Have you noticed the url change again with Comcast?