Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forbidden 403 error downloading from UKTV Play #16952

Closed
geekcroft opened this issue Jul 12, 2018 · 4 comments
Closed

Forbidden 403 error downloading from UKTV Play #16952

geekcroft opened this issue Jul 12, 2018 · 4 comments

Comments

@geekcroft
Copy link

@geekcroft geekcroft commented Jul 12, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.07.10. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • [x ] I've verified and I assure that I'm running youtube-dl 2018.07.10

Before submitting an issue make sure you have:

  • [x ] At least skimmed through the README, most notably the FAQ and BUGS sections
  • [x ] Searched the bugtracker for similar issues including closed ones
  • [x ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • [x ] Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl --cookies cookies.txt https://uktvplay.uktv.co.uk/shows/dara-o-briains-go-8-bit/watch-online/?video=5728551474001 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--cookies', u'cookies.txt', u'https://uktvplay.uktv.co.uk/shows/dara-o-briains-go-8-bit/watch-online/?video=5728551474001', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.10
[debug] Python version 2.7.10 (CPython) - Linux-4.2.0-35-generic-x86_64-with-Ubuntu-15.10-wily
[debug] exe versions: avconv 2.7.6-0ubuntu0.15.10.1, avprobe 2.7.6-0ubuntu0.15.10.1, ffmpeg 2.7.6-0ubuntu0.15.10.1, ffprobe 2.7.6-0ubuntu0.15.10.1
[debug] Proxy map: {}
[brightcove:new] 5728551474001: Downloading webpage
[brightcove:new] 5728551474001: Downloading JSON metadata
[brightcove:new] 5728551474001: Downloading m3u8 information
[brightcove:new] 5728551474001: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://secure.brightcove.com/services/mobile/streaming/index/rendition.m3u8?assetId=5728556754001&secure=true&expiration=1531655940000&token=0214545c20dacd5d35fd9dee74cfa3b27bd9ad85&pubId=1242911124001&videoId=5728551474001'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 270
[download] Destination: CTOO078H82H-5728551474001.mp4
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 3 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 4 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 5 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 6 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 7 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 8 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 9 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 10 of 10)...

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Trying to download from UKTV - have tried passing user/pass and using cookies.txt extracted from a logged in Chrome. Video is accessible from a browser on same box that youtube-dl is installed on, but using the cli I just get 403.

Log is truncated as it skips each fragment - but I allowed a single one in the log for visibility.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Jul 12, 2018

youtube-dl https://uktvplay.uktv.co.uk/shows/dara-o-briains-go-8-bit/watch-online/?video=5728551474001
[brightcove:new] 5728551474001: Downloading webpage
[brightcove:new] 5728551474001: Downloading JSON metadata
[brightcove:new] 5728551474001: Downloading m3u8 information
[brightcove:new] 5728551474001: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 270
[download] Destination: CTOO078H82H-5728551474001.mp4
[download]   1.2% of ~894.42MiB at 34.36KiB/s ETA 06:15:27
@geekcroft
Copy link
Author

@geekcroft geekcroft commented Jul 12, 2018

@remitamine odd as the video plays from the location I am downloading from :/

@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Jul 14, 2018

@geekcroft wrote:

Video is accessible from a browser on same box that youtube-dl is installed on

as the video plays from the location I am downloading from

Hi 😄 ; what exactly is the location referenced? Are you indeed within the UK?

First, current uktvplay extractor can by-pass the mandatory sign-in requirement, so no need to use the --cookies option (as also clearly demonstrated by @remitamine's download attempt)

Second, the extractor uses AppleHLS streams (with AES-128 encryption), produced for mobile devices, to do the actual download; I am currently unable to check the type of stream methodology UKTVPLAY implement on desktop browsers (clients), most probably it be either AdobeHDS or MPEG-DASH, no doubt both with DRM turned on 😠 ... So the fact your specific video does play in your browser (Google Chrome?) might be of little importance as far as yt-dl is concerned...

Focusing just on the mobile streams exploited by yt-dl, the master playlist URI
http://c.brightcove.com/services/mobile/streaming/index/master.m3u8?videoId=5728551474001
as well as the variant (720x404p, @1923kbps) playlist
https://secure.brightcove.com/services/mobile/streaming/index/rendition.m3u8?assetId=5728556754001&expiration=1531784640000&token=cf1d00fb8a3fdd927f1fa2e4a86bde227dc38983&videoId=5728551474001 are not geo-fenced, that is both hosts c.brightcove.com and secure.brightcove.com are accessible globally. Indeed, I am overseas myself, issuing

youtube-dl -F "https://uktvplay.uktv.co.uk/shows/dara-o-briains-go-8-bit/watch-online/?video=5728551474001"

gives the false impression I have access to listed variants:

[brightcove:new] 5728551474001: Downloading webpage
[brightcove:new] 5728551474001: Downloading JSON metadata
[brightcove:new] 5728551474001: Downloading m3u8 information
[brightcove:new] 5728551474001: Downloading m3u8 information
[info] Available formats for 5728551474001:
format code  extension  resolution note
hls-745-0    mp4        640x360     745k
hls-745-1    mp4        640x360     745k
hls-1321-0   mp4        640x360    1321k
hls-1321-1   mp4        640x360    1321k
hls-1923-0   mp4        720x404    1923k
hls-1923-1   mp4        720x404    1923k  (best)

The bitter truth is that while the AES-128 decryption key issuer URI is not geo-fenced
https://secure.brightcove.com/services/mobile/streaming/index/keyfile?assetId=5728556754001&<various other strings> (BTW the key changes after every 10 segments), the segments themselves are served from a host (hlsuktv-a.akamaihd.net) that does, in fact, geo-block, hence the 403: Forbidden errors in your log; I get the exact log if I try to fetch from my non-UK location...

So, my gut feeling is you're not in the UK, but use some means of geo-location spoofing for your browser to work; that may or may not support the HLS streams tried by yt-dl; if it's a Smart DNS service, they might have configured it to work for desktop browser streams, but not for their mobile counterparts; if you're using a commercial HTTP/Socks5 proxy (or a SSH tunnel that creates a local Socks5 proxy), you need to consult yt-dl's documentation on how to make these available to yt-dl; a full blown, system-wide, UK VPN is expected to work both in your browser and yt-dl...

In any case, this does not appear to be an issue with yt-dl; in the off-chance you're indeed located in the UK, then I hope you've enjoyed this small analysis, as nothing else comes to mind for you to try... 😜

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Jul 22, 2018

with no response from @geekcroft, I guess as @Vangelis66 suggested that he was using something specific in the browser(proxy or an extension) to access the videos.

@remitamine remitamine closed this Jul 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.