Verizon Adobe Pass: Unable to extract SAML Login URL

[GotTheNumbers]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like this: [x])
• Use the Preview tab to see what your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.10.20. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.10.20

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--ap-mso', 'Verizon', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--prefer-ffmpeg', 'http://www.aetv.com/shows/intervention/season-15/episode-2']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2017.10.20
[debug] Python version 3.4.4 - Windows-10-10.0.14393
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[aenetworks] intervention/season-15/episode-2: Downloading webpage
[aenetworks] 642219075881: Downloading JSON metadata
[aenetworks] 642219075881: Downloading Provider Redirect Page
[aenetworks] 642219075881: Downloading SAML Login Page
ERROR: Unable to extract SAML Login URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvdixi2dy\build\youtube_dl\YoutubeDL.py", line 784, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvdixi2dy\build\youtube_dl\extractor\common.py", line 434, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvdixi2dy\build\youtube_dl\extractor\aenetworks.py", line 143, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvdixi2dy\build\youtube_dl\extractor\adobepass.py", line 1482, in _extract_mvpd_auth
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvdixi2dy\build\youtube_dl\extractor\common.py", line 797, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract SAML Login URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

---

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

-Single video: http://www.aetv.com/shows/intervention/season-15/episode-2
-Single video: http://www.history.com/shows/american-pickers/season-18/episode-2
-Single video: http://www.trutv.com/full-episodes/impractical-jokers/2111547/index.html
-Single video: http://www.spike.com/episodes/g69l7y/cops-cops-3018-season-30-ep-3018

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

---

I'm using Verizon FiOS TV adobe pass.
When I try to download from aetv.com, I get "ERROR: Unable to extract SAML Login URL;" as pasted above.
When I try to download from history.com, I get the same error.
When I try to download from trutv.com I get "ERROR: Unable to download webpage: HTTP Error 400: Bad Request (caused by HTTPError())".
When I try to download from spike.com, it works.

Is it just me, or is there an issue with these sites and/or the Verizon mso?

[enigmaquip]

I'm getting the same errors using DirecTV, it looks a little different due to the different mso

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.aetv.com/shows/intervention/season-15/episode-2', u'--ap-mso', u'DTV', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.10.20
[debug] Python version 2.7.6 - Linux-3.19.0-32-generic-x86_64-with-LinuxMint-17.3-rosa
[debug] exe versions: none
[debug] Proxy map: {}
[aenetworks] intervention/season-15/episode-2: Downloading webpage
[aenetworks] 642219075881: Downloading JSON metadata
[aenetworks] 642219075881: Downloading Provider Redirect Page
[aenetworks] 642219075881: Downloading Provider Login Page
[aenetworks] 642219075881: Logging in
ERROR: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 784, in extract_info
    ie_result = ie.extract(url)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "./youtube-dl/youtube_dl/extractor/aenetworks.py", line 143, in _real_extract
    url, video_id, requestor_id, resource)
  File "./youtube-dl/youtube_dl/extractor/adobepass.py", line 1511, in _extract_mvpd_auth
    post_form(mvpd_confirm_page_res, 'Confirming Login')
  File "./youtube-dl/youtube_dl/extractor/adobepass.py", line 1360, in post_form
    post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_page, 'post url', group='url')
  File "./youtube-dl/youtube_dl/extractor/common.py", line 806, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 797, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract post url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[enigmaquip]

I can see from my --write-pages command it's happening in adobepass.py as shown in my log for line 1511
starting at https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/adobepass.py#L1506

                    mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', {
                        mso_info.get('username_field', 'username'): username,
                        mso_info.get('password_field', 'password'): password,
                    })
                    if mso_id != 'Rogers':
                        post_form(mvpd_confirm_page_res, 'Confirming Login')

the page returned from 'logging in' is a javascript reload, not another form page
<html><head><script language="JavaScript" type="text/javascript">function reloadintop() { try{if (self.parent.frames.length != 0)self.parent.location="http://www.aetv.com/shows/intervention/season-15/episode-2"; else if (window.opener){window.opener.location="http://www.aetv.com/shows/intervention/season-15/episode-2";window.close()}}catch (Exception) {} }</script></head><body onLoad=reloadintop()></body></html>

[beren12]

I'm getting issues with disney jr now as well.

[SkiTheSlicer]

@GotTheNumbers I'm able to download the following after applying my pending pull request:
- http://www.aetv.com/shows/intervention/season-15/episode-2
- http://www.history.com/shows/american-pickers/season-18/episode-2
- http://www.spike.com/episodes/g69l7y/cops-cops-3018-season-30-ep-3018

TruTV failed with a 400 error. It's probably because TruTV is not in my subscription package, but I can download the following free episode:
- http://www.trutv.com/full-episodes/impractical-jokers/2111524/index.html

[SkiTheSlicer]

also, I believe spike worked because that particular episode wasn't behind a paywall, and thus didn't call for Verizion FiOS credentials.

[bishpuppy]

@SkiTheSlicer Thank you. I just tried your pull on an A&E video using Verizon FIOS auth and it worked. Hopefully your pull gets approved.

[malford]

I have FIOS with trutv subscription and am able to stream from a browser. Executing with @SkiTheSlicer's verizon_hba branch I am also getting a 400 error running:

youtube-dl --verbose --ap-mso Verizon --ap-username PRIVATE --ap-password PRIVATE 'http://www.trutv.com/full-episodes/at-home-with-amy-sedaris/2134844/index.html'

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--ap-mso', u'Verizon', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'http://www.trutv.com/full-episodes/at-home-with-amy-sedaris/2134844/index.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.24
[debug] Git HEAD: c163d7abc
[debug] Python version 2.7.14 - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4, ffprobe 3.4
[debug] Proxy map: {}
[TruTV] 2134844: Downloading webpage
[TruTV] Downloading XML
[TruTV] 2134844: Downloading Provider Redirect Page
ERROR: Unable to download webpage: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/Users/malford/code/youtube-dl/youtube_dl/extractor/common.py", line 503, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/Users/malford/code/youtube-dl/youtube_dl/YoutubeDL.py", line 2184, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Maybe it is unrelated though?

[beren12]

I'm getting the same error as @malford with cartoon network

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-i', u'-4', u'--ap-mso', u'Verizon', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'--all-subs', u'--write-sub', u'--download-archive', u'archive.txt', u'http://www.cartoonnetwork.com/video/we-bare-bears/bro-brawl-episode.html', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.12.14
[debug] Python version 2.7.13 - Linux-4.13.0-0.bpo.1-amd64-x86_64-with-debian-9.3
[debug] exe versions: ffmpeg 3.2.9-1, ffprobe 3.2.9-1, rtmpdump 2.4
[debug] Proxy map: {}
[CartoonNetwork] bro-brawl: Downloading webpage
[CartoonNetwork] a2b31c440c2808fe7f92310af3a21d49d5976e84: Downloading XML
[CartoonNetwork] a2b31c440c2808fe7f92310af3a21d49d5976e84: Downloading f4m manifest
WARNING: Unable to download f4m manifest: HTTP Error 403: Forbidden
[CartoonNetwork] a2b31c440c2808fe7f92310af3a21d49d5976e84: Downloading Provider Redirect Page
ERROR: Unable to download webpage: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 506, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2196, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[bishpuppy]

Looks like Verizon changed the login. The 'login url' for the json is now the logout url

[thesteveway]

Was there a fix for this? Verizon ap has never worked for me

[thesteveway]

@bishpuppy does something need to be done differently with the logout url?

[bishpuppy]

@thesteveway , the fix that @SkiTheSlicer posted worked for a while but it was never brought into the main app.

[thesteveway]

@bishpuppy @SkiTheSlicer is there any way to get it working at all? I've never gotten ap to work and I've been using y-dl for months

[ParadoxGBB]

I hit this as well, and after applying the fix @SkiTheSlicer proposed it worked briefly then stopped circa 8/25. After some investigation I tweaked his fix here:

https://bitbucket.org/ParadoxGBB/youtube-dl/commits/64bddfe15c1458a1b3461875bf9afd0a17ebeea0

You can see more details on my post at #14363.