Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ITV BTCC videos support? #16139

Closed
WolfganP opened this issue Apr 9, 2018 · 6 comments
Closed

ITV BTCC videos support? #16139

WolfganP opened this issue Apr 9, 2018 · 6 comments
Labels

Comments

@WolfganP
Copy link

@WolfganP WolfganP commented Apr 9, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.04.09. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • [*] I've verified and I assure that I'm running youtube-dl 2018.04.09

Before submitting an issue make sure you have:

  • [*] At least skimmed through the README, most notably the FAQ and BUGS sections
  • [*] Searched the bugtracker for similar issues including closed ones
  • [*] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • [*] Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl -v https://pastebin.com/raw/KxD6rhpF --geo-bypass-country UK
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://pastebin.com/raw/KxD6rhpF', u'--geo-bypass-country', u'UK']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.04.09
[debug] Python version 2.7.13 (CPython) - Linux-4.9.62-v7+-armv7l-with-debian-9.4
[debug] exe versions: ffmpeg 3.2.10-1, ffprobe 3.2.10-1
[debug] Proxy map: {}
[debug] Using fake IP None (UK) as X-Forwarded-For.
[generic] KxD6rhpF: Requesting header
WARNING: Falling back on generic information extractor.
[generic] KxD6rhpF: Downloading webpage
[generic] KxD6rhpF: Extracting information
[download] Downloading playlist: Brightcove video tester
[generic] playlist Brightcove video tester: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[debug] Using fake IP None (UK) as X-Forwarded-For.
[debug] Using fake IP None (UK) as X-Forwarded-For.
[brightcove:new] 5766870719001: Downloading webpage
[brightcove:new] 5766870719001: Downloading JSON metadata
ERROR: Access to this resource is forbidden by access policy.
You might want to use a VPN or a proxy server (with --proxy) to workaround.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/brightcove.py", line 706, in _real_extract
    json_data = self._download_json(api_url, video_id, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 692, in _download_json
    encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 634, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/adobepass.py", line 1332, in _download_webpage_handle
    *args, **compat_kwargs(kwargs))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 539, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 528, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 789, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 440, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/brightcove.py", line 712, in _real_extract
    self.raise_geo_restricted(msg=message)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 743, in raise_geo_restricted
    raise GeoRestrictedError(msg, countries=countries)
GeoRestrictedError: Access to this resource is forbidden by access policy.

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

ITV separated the BTCC race videos from the hub (which also seems to be having issues as per #15925)
Lately the video are hosted at http://www.itv.com/btcc/races (ie for a particular weekend all videos are posted at individual pages like: http://www.itv.com/btcc/races/btcc-2018-all-the-action-from-brands-hatch)

Skimming the source code of this sample weekend page, I extracted the vid params and built a test page:

Question 1: is the log error above pointing just to a geo restriction error or is there anything else involved that I missed? (ie: like writing some header to force the ITV scrapper to act instead of a generic one)

<!doctype html>
<html> 
<head>
    <meta charset="UTF-8">
    <title>Brightcove video tester</title>
   <style>
    .video-js {
      height: 360px;
      width: 640px;
    }
    .video-js.vjs-mouse .vjs-big-play-button {
      background-color: red;
      opacity: .3;
    }
   </style>
   <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
</head>

<body>
	<video data-id="id3809566268703" data-video-id="5766870719001" data-account="1582188683001" data-player="HkiHLnNRx" data-embed="default" data-application-id="" data-fc-id="87459" class="video-js" controls="">
	</video>

	<script src="https://players.brightcove.net/1582188683001/HkiHLnNRx_default/index.min.js">
	</script>
</body>
</html>

Question 2: is there any way to generate a playlist of downloadable items based on pages like http://www.itv.com/btcc/races/btcc-2018-all-the-action-from-brands-hatch with youtube-dl?

@Tony-il-Capo
Copy link

@Tony-il-Capo Tony-il-Capo commented May 1, 2018

Hi @WolfganP
I'm also interested in this. Tried to get the videos from the the latest races without success.
Did you found a way to download (another method) these BTCC videos from ITV?

Thanks.

@dstftw dstftw closed this in ea1f5e5 May 2, 2018
@WolfganP
Copy link
Author

@WolfganP WolfganP commented May 2, 2018

@Tony-il-Capo I couldn't find any way to download the individual files other than going to the weekend page sources, and then manually craft a tester page for a single race file as I did above.

I was hoping there was a way to parse the weekend page and get at least a playlist that may individually be fed to youtube-dl for individual download, but this issue was closed and I couldn't find any parser around as a semi-automated workaround.

@WolfganP
Copy link
Author

@WolfganP WolfganP commented May 8, 2018

@Tony-il-Capo it works with https://github.com/rg3/youtube-dl/releases/tag/2018.05.09, with a simple cmd line:
youtube-dl -v https://www.itv.com/btcc/races/btcc-2018-all-the-action-from-donington-park --playlist-items 1-3

@dstftw you're a legend. Thx a lot for implementing the workaround so promptly.

Related question: what's the best way to parse the page and just produce a playlist file and not download the files? (so I later can instruct yt-dl it to download just certain items)
I went thru the options and tried:

--flat-playlist (no media filenames listed, just "X of Y" items)

[ITVBTCC] btcc-2018-all-the-action-from-donington-park: Downloading webpage
[download] Downloading playlist: BTCC 2018: All the action from Donington Park
[ITVBTCC] playlist BTCC 2018: All the action from Donington Park: Collected 11 video ids (downloading 11 of them)
[download] Downloading video 1 of 11
[download] Downloading video 2 of 11
[download] Downloading video 3 of 11
[download] Downloading video 4 of 11
[download] Downloading video 5 of 11
[download] Downloading video 6 of 11
[download] Downloading video 7 of 11
[download] Downloading video 8 of 11
[download] Downloading video 9 of 11
[download] Downloading video 10 of 11
[download] Downloading video 11 of 11
[download] Finished downloading playlist: BTCC 2018: All the action from Donington Park

-- get-filename (list the filenames, but not the item # so it's difficult to DL specific items later)

$ youtube-dl -v https://www.itv.com/btcc/races/btcc-2018-all-the-action-from-donington-park --get-filename
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.itv.com/btcc/races/btcc-2018-all-the-action-from-donington-park', u'--get-filename']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.05.09
[debug] Python version 2.7.13 (CPython) - Linux-4.9.62-v7+-armv7l-with-debian-9.4
[debug] exe versions: ffmpeg 3.2.10-1, ffprobe 3.2.10-1, phantomjs .
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
tg btcc 4-5778498003001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg btcc5-5778524531001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg btcc 6a-5778527050001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg pcc4-5778530146001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg f4 5-5778550788001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg f4 6-5778534190001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg clio 3-5778540079001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg clio4-5778527234001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg gjnr5-5781914140001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg gtsc5 (1)-5781929168001.mp4
[debug] Default format spec: bestvideo+bestaudio/best
tg gtsc6-5781924467001.mp4

--dump-single-json (it puts lot of detailed info on screen but can't redirect it with -o, and std unix redirect ">" produces a mixed file of info lines + json)

[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Default format spec: bestvideo+bestaudio/best
{"extractor": "ITVBTCC", "_type": "playlist", "title": "BTCC 2018: All the action from Donington Park", "extractor_key": "ITVBTCC", "webpage_url": "https://www.itv.com/btcc/races/btcc-2018-all-the-action-from-donington-park", "entries": [{"display_id": "5778498003001", "extractor": "brightcove:new", "protocol": "https", "description": null, "tags": [], "timestamp": 1525107794, "playlist_uploader": null, "tbr": 1199.0, "height": 396, "width": 704, "duration": 1520.277, "source_preference": 0, "format_id": "http-1199k-396p-1", "upload_date": "20180430", "uploader_id": "1582188683001", "subtitles": {}, "playlist_title": "BTCC 2018: All the action from Donington Park", "playlist": "BTCC 2018: All the action from Donington Park", "requested_subtitles": null, "container": "MP4", "thumbnails": [{"url": "http://newsco-a.akamaihd.net/priority/itnitvnews/shortform/HTTP/1582188683001/201804/1582188683001_5778504290001_5778498003001-th.jpg?pubId=1582188683001", "id": "0"}], "title": "tg btcc 4", "url": "https://newsco-a.akamaihd.net/priority/itnitvnews/shortform/HTTP/1582188683001/201804/1582188683001_5778507499001_5778498003001.mp4", "extractor_key": "BrightcoveNew", "vcodec": "H264", "http_headers": {"Origin": "https://www.itv.com", "Accept-Language": "en-us,en;q=0.5", "Accept-Encoding": "gzip, deflate", "X-Forwarded-For": "54.36.163.79", "Accept": "application/json;pk=BCpkADawqM2j1lcUbZzaHTq41cLhcN3GqgBE48QDOiryDpYp7ME9a_SKWQB2EPhuwE5WQS4KZZNG16_Ri7G5RKXuDCuKMBsex2M5WxlWMvgAglzmtp5u22ObRmIaNzyV21yaB_tovshcGfMmx3nlTG4bS41KIvre57lw6DDtrIDTQL9533mQclMIWiM", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 (Chrome)", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Referer": "
[...]
@Tony-il-Capo
Copy link

@Tony-il-Capo Tony-il-Capo commented May 8, 2018

Unfortunately, I can't get past the geo restriction (even using --geo-bypass , --geo-bypass-county uk)

@WolfganP
Copy link
Author

@WolfganP WolfganP commented May 8, 2018

I'm not UK based either, but I'm using an smartdns service, that may be the difference.

Interestingly, @dstftw implemented a new functionality (--geo-bypass-ip-block) that's harcoded in the ITV extractor https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/itv.py

 smuggle_url(self.BRIGHTCOVE_URL_TEMPLATE % video_id, {
                    # ITV does not like some GB IP ranges, so here are some
                    # IP blocks it accepts
                    'geo_ip_blocks': [
                        '193.113.0.0/16', '54.36.162.0/23', '159.65.16.0/21'
                    ],
                    'referrer': url,
                }),
@Tony-il-Capo
Copy link

@Tony-il-Capo Tony-il-Capo commented May 8, 2018

Didn't test that, but it's interesting.
--geo-verification-proxy url works, the hard part is to find a free and working proxy, found one, just finished downloading. Videos seem OK.

@dstftw dstftw added the fixed label May 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.