Add support for HBOGo & HBO Now

[Jimi-James]

I'm sure you're probably already working on this, but I noticed there was no issue open about it to monitor its progress, so I'm doing that.

• I've verified and I assure that I'm running youtube-dl 2016.06.14

I have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl -v "http://play.hbogo.com/episode/urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://play.hbogo.com/episode/urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.14
[debug] Python version 3.5.1 - Linux-4.5.4-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.0.2, ffprobe 3.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[generic] urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true: Requesting header
WARNING: Falling back on generic information extractor.
[generic] urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true: Downloading webpage
[generic] urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true: Extracting information
ERROR: Unsupported URL: http://play.hbogo.com/episode/urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/common.py", line 341, in extract
    return self._real_extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/generic.py", line 2063, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: http://play.hbogo.com/episode/urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true
$ youtube-dl -v "http://play.hbogo.com/season/urn:hbo:season:GVtcU-Qbol7qlofAIAAAH"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://play.hbogo.com/season/urn:hbo:season:GVtcU-Qbol7qlofAIAAAH']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.14
[debug] Python version 3.5.1 - Linux-4.5.4-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.0.2, ffprobe 3.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[generic] urn:hbo:season:GVtcU-Qbol7qlofAIAAAH: Requesting header
WARNING: Falling back on generic information extractor.
[generic] urn:hbo:season:GVtcU-Qbol7qlofAIAAAH: Downloading webpage
[generic] urn:hbo:season:GVtcU-Qbol7qlofAIAAAH: Extracting information
ERROR: Unsupported URL: http://play.hbogo.com/season/urn:hbo:season:GVtcU-Qbol7qlofAIAAAH
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/common.py", line 341, in extract
    return self._real_extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/generic.py", line 2063, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: http://play.hbogo.com/season/urn:hbo:season:GVtcU-Qbol7qlofAIAAAH
<end of log>

---

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

-Single video: http://play.hbogo.com/episode/urn:hbo:episode:GVx-Z-wOMe8N-wwEAAAAQ?reentered=true (not sure if that ?reentered=true part matters)
-Playlist: http://play.hbogo.com/season/urn:hbo:season:GVtcU-Qbol7qlofAIAAAH
-I don't use HBO Now and so can't provide sample links for it.

Description of your issue, suggested solution and other information

Note: Both of these sites require logging in with user credentials. HBOGo logs in across multiple websites, for example, I use my parent's DISH Network subscription, so I have to login to the DISH Network site and then it redirects to HBOGo with POST data.

[yan12125]

All provided links give an HTTP 403 Forbidden error in Firefox. Maybe geo-restricted?

[Jimi-James]

I'm using Firefox. It could be because you aren't logged in. Perhaps only someone with an HBOGo subscription or HBO Now subscription can test and set up each one.

[yan12125]

Could you point out where to log in?

[Hrxn]

It's not only providing correct credentials, this is also geo-restricted. Or did they add exceptions recently? Otherwise, still not working here.

[yan12125]

I'm prompted to enter credentials with an US proxy.

[Jimi-James]

I think I'll just finally learn Python and submit a merge myself, since I have an HBOGo account and live in the US. No guarantee on how long I'll be. Someone else who has a subscription to HBO Now will have to handle HBO Now.

[Jimi-James]

So, now that I'm getting started, I've got 2 questions:

1. How can I find out what kind of authentication HBOGO uses? I don't think it's 2-factor, username/password, or a netrc file, because it's an off-site thing. You pick a TV provider, like DISH Network, and it redirects you to that site to login, and then you're redirected back with the login cookie and that lets you in. And it looks like if you haven't done that, you can't even access what would be the valid download URLs for youtube-dl to look at, so it would have to immediately login before downloading the webpage. How would I have youtube-dl support that? Are there already any supported providers that have a similar login-with-TV-provider setup? I know it's a pretty common method for these streaming sites.

In case it sheds some light, if I login with DISH Network, it redirects me to a login page where I enter a username and password into a DISH website, and then when it redirects me back to HBOGO, this is what the redirection URL looks like:
https://identity1.dishnetwork.com/saml/module.php/authbypass/lastbookend.php?AuthState=_<a 42-character string of numbers and lowercase letters goes here; I deleted it in case it's special to my account or something>%3Ahttps%3A%2F%2Fidentity1.dishnetwork.com%2Fsaml%2Fsaml2%2Fidp%2FSSOService.php%3Fspentityid%3Dhttps%253A%252F%252Fauth.hbogo.com%252Fsaml%252Fmodule.php%252Fsaml%252Fsp%252Fmetadata.php%252Fhbogo_sp%26cookieTime%3D1468444143%26RelayState%3Dhttps%253A%252F%252Fauth.hbogo.com%252Fsaml%252Fmodule.php%252Fhbo%252Flogin%252Findex.php%253Fstage%253Dli3%2526idp%253Ddish%26RequesterID%3D%255B%2522hbo%2522%255D&id=<another possibly special string, this time 10 characters, numbers and lowercase letters>&coeff=1&history=5

And to save you some time, here's what it looks like after converting all the HTML character codes with \x and echo -e:
https://identity1.dishnetwork.com/saml/module.php/authbypass/lastbookend.php?AuthState=_<a 42-character string of numbers and lowercase letters goes here; I deleted it in case it's special to my account or something>:https://identity1.dishnetwork.com/saml/saml2/idp/SSOService.php?spentityid=https://auth.hbogo.com/saml/module.php/saml/sp/metadata.php/hbogo_sp&cookieTime=1468444143&RelayState=https://auth.hbogo.com/saml/module.php/hbo/login/index.php?stage=li3&idp=dish&RequesterID=["hbo"]&id=<another possibly special string, this time 10 characters, numbers and lowercase letters>&coeff=1&history=5

2. How should I approach the issue of it being geo-restricted? Since we don't want to do anything illegal, should I just have the HBOGO provider return a message saying that it's geo-restricted if a user in a restricted country tries to download something? If so, how exactly would I go about returning that message? Would it be similar to the error message detection that I saw in extractors/common.py, where it detected that a site was blocked by Websense (in the _webpage_read_content function), except looking for a 403 Forbidden error?

[TRox1972]

@Jimi-James
1- Support for some TV providers are supported, although it's easier to use cookies; see bf83024
2- If content is geo-restricted, a geo-restriction exception should be raised.

I'm pretty sure that HBO use DRM for most of their digital content; decrypting it will probably be very difficult.

[remitamine]

1. i added support for Adobe Pass(the only supported TV provider is DirecTV) in bf83024 it works now with national geographic and A&E network sites and it can be extended for other sites as well.
   the problem is how we can support other provider, i tried to use cookies but it didn't work for me.
2. handling geo restriction should not be a problem.
3. the main problem is that HBO uses DRM for it's videos.

[Jimi-James]

For the DRM, I figured I'd just copy how the CW extractor downloads the video. The CW uses a similar kind of DRM, so its extractor seems to brute-force it by just capturing the entire stream as raw data.

[remitamine]

the m3u8 manifest used by HBO are DRM protected(if i remember correctly they are protected with Adobe Flash Access) while the ones from CW are not.

[Jimi-James]

Oh. Is HBOGO support for youtube-dl even doable, then?

[remitamine]

if you want to confirm you can see that the SAMPLE-AES was served using faxs protocol.
in the manifest file you will see something similar in the begining of the m3u8 manifest:

#EXTM3U
#EXT-X-VERSION:5
## Created with Unified Streaming Platform(version=1.6.6)
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:10
#EXT-X-FAXS-CM:<BASE64 DRM Metadata>
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="faxs://faxs.adobe.com",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1",IV=<IV>

[remitamine]

Oh. Is HBOGO support for youtube-dl even doable, then?

it's possible, but difficult(figure how to decrypt DRM pretected segements or finding a workaround to get unprotected video like in the case of DAISUKI).

[Jimi-James]

I'm probably going to need help, then, because I have no experience decrypting DRM and I don't know what an m3u8 manifest is. Is that something I'll be learning after I read more example code from the other extractors?

[remitamine]

https://en.wikipedia.org/wiki/HTTP_Live_Streaming
https://en.wikipedia.org/wiki/M3U

[Hrxn]

This is not easy, so good luck and all the best! 😉

[danfiscus]

Is there any progress on this issue? I would also like to be able to download videos from HBO Now. @Jimi-James @yan12125

I may be able to provide an HBO Now account to be used if that is necessary.

[Jimi-James]

I ended up not having the time to work on it at all. I might not be the best person with my complete lack of experience in dealing with encryption and encoding and decoding with internet videos. To survive for now, I've been using a Chromecast to watch HBOGo.

[danfiscus]

@Jimi-James Okay, unfortunately that solution doesn't work for me as I use youtube-dl to download videos entirely before I watch them since my internet is too slow to watch videos as they load. I'm not really the right person to fix this issue either, though, because I barely know Python well enough to add a site. I thought I'd tag you anyway though because no one had bumped this thread in over half a year and I wasn't sure if maybe their site changed in such a way that would make this easier to do than before

[Jimi-James]

@danfiscus Ooh, I feel you there. My internet right now is BARELY fast enough to stream (2Mb), so I massively prefer youtube-dl or other download methods when I can, which these days is with everything EXCEPT HBOGo. So I do have an incentive to try this out eventually. I at least know Python well enough to set up the skeleton of the extractor, which I've done in my fork.

[danfiscus]

@Jimi-James Is HBOGo set up the same way as HBO Now? Or are they different? I've only ever used HBO Now, so I'm not sure if it's easier to download from or if it's the same platform.

[Jimi-James]

@danfiscus They're different enough that I couldn't use the same code on both, but the real issue is that neither can be tested without a paid account in an allowed country. That's why nobody else has made extractors for them yet. So if you used your Now account for testing, that solves that.

Conversely, I've only ever used HBOGo. I don't know how identical or non-identical their frontends and backends are, but they're made by different people.

[danfiscus]

@Jimi-James anyone can get a free trial account that works for a month, that's what I'm using right now

[Jimi-James]

@danfiscus Oh, cool, so I could use that. I wouldn't be able to maintain it once my trial went out, though, so it would only be a temporary solution.

[john745]

Any progress? I managed to download the file, but only the video part - no audio - and it's encrypted so it doesn't play, it just shows a gray screen even though the time keeps passing.

[Jimi-James]

At this point, I'm officially not working on this. I brought tons of eagerness but no experience, and I have too many things going on to do anything with this in the near future. I hope someone else can!

[yan12125]

An HBO Now test URL from #13908: https://play.hbonow.com/episode/urn:hbo:episode:GVU4NYgvPQlFvjSoJAbmL

[fwzmhmd]

@yan12125 Shall i provide more urls?

[yan12125]

That should be enough

[fwzmhmd]

Question: If HBO Now isn't supported/work, why is it on the supported list @ https://rg3.github.io/youtube-dl/supportedsites.html

[dstftw]

There is no HBO Now on this list.