Unable to download ITV programme Viewpoint

[jefro108]

Using GiA 1.21.9 on macOS Mojave 10.14.6

I'm unable to download Viewpoint episode 1 https://www.itv.com/hub/viewpoint/10a0556a0001. I can watch it on the webpage though.

See error log at
ViewpointErrorLog.txt

[jefro108]

Well, that's strange. I tried it again a couple of times and now it works.

I've noticed this before with iTV downloads - would it be possible to introduce a number of tries like happens with BBC programmes, so that it fails a bit more gracefully?

[damndingo]

Hi,
I have the same problem here too since today.
All itv downloads fail for that reason

[SOLOSAIR]

Hi,

I have the same problem here too since today.

All itv downloads fail for that reason

all itv now failing on PVR and Use Current Webpage

[skovatch]

ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Looks like a change on ITV's end has broken youtube-dl. It fails on the command line too.

[skovatch]

It's already been reported as ytdl-org/youtube-dl#28906

[nitramlatep]

ITV have appeared to have gone DRM protection. Not sure if this means the end to be able to download via this brilliant application?

[jefro108]

ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Looks like a change on ITV's end has broken youtube-dl. It fails on the command line too.

It seems like it’s a bit random because after a few tries I was able to download the programme.

It would be great if GiA could try the download a few times like it seems to do with BBC shows, rather than falling after this error 😊

[nitramlatep]

@jefro108 are you able to download Viewpoint Episode 3? I have tried and tired but get the same error. Using Downie, the error comes up that the video is DRM protected.

[skovatch]

Automatic retry is a feature of get_iplayer, not GiA. I'm not convinced it's DRM - Downie could be assuming that because of the 500 error. Some shows do indeed still work, but there's no pattern as to what will or won't work.

[nitramlatep]

Fingers crossed you guys can find a fix. The app is superb and a life line for many. :)

[TIGHazard1]

Granted I'm a Windows user, not a Mac user but at least with Youtube-dl you can continue by using the STV Player (which is ITV in Scotland).

However because it's only the Channel 3 contractor, this would not work for ITV2, 3, 4 or CITV programmes.

[torbs723]

ITV have appeared to have gone DRM protection. Not sure if this means the end to be able to download via this brilliant application?

Agreeing with what one or two others have said. Downie may well be reporting DRM as the server error is preventing access to the HLS versions which are what GiA and yt-dl use.

I believe some streams do use DRM so there is a possibility that these are all that is being presented to Downie because of the errors.

[jefro108]

@jefro108 are you able to download Viewpoint Episode 3? I have tried and tired but get the same error. Using Downie, the error comes up that the video is DRM protected.

I was able to download Episode 2 but I've had similar problems downloading Episode 3

According to ytdl-org/youtube-dl#28906 it works to get it on STV player with youtube-dl - so I got it from https://player.stv.tv/episode/435y/viewpoint

Not only does it work but it's much higher resolution at 1080p!

@skovatch is there any possibility of adding STV player support to GiA to automate this?

[UbiGit]

Episodes 1 and 2 downloaded completely though they needed several manual retry's before the dl's were complete. Episodes 3 and 4 have been impossible and fail at the first hurdle. Logged: ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 2248, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
youtube-dl finished downloading
AppController: Downloads Finished

[torbs723]

Can't even seem to grab episode 4 from STV

[lowenna]

@torbs723 I've been able to get 3 and 4 from STV although failing on ITV. 1&2 came from ITV just fine. Haven't yet tried 5 (slow internet connection....)

[torbs723]

@lowenna Realised I had dropped back to a 2020 build of yt-dl. Up to date version downloads 3, 4 and 5 fine from STV. And much better quality too!

[lowenna]

@torbs723 yes, indeed - much better quality! That all said, I was fine with ITV quality, and had automation around it, so the STV route isn't appealing right now. Would love to figure out what's going with ITV

[rabrowne]

I've not been able to download episodes 3, 4 or 5 with log only showing :
Downloading Show 1/3:

Downloading Viewpoint: Series 1
INFO: Preparing Request for Auth Info
INFO: Metadata processed.
youtube-dl finished downloading

Downloading Show 2/3:

Downloading Viewpoint: Series 1
INFO: Preparing Request for Auth Info
INFO: Metadata processed.
youtube-dl finished downloading

Downloading Show 3/3:

Downloading Viewpoint: Series 1
INFO: Preparing Request for Auth Info
INFO: Metadata processed.
youtube-dl finished downloading
AppController: Downloads Finished

Also noticed that the show is only available for 1 day and the latest news that Episode 6 will not be broadcast due to lead actor Noel Clarke allegations. Perhaps ITV has implemented actions to limit access?!

[rabrowne]

Just run again with verbose mode for log and getting the following:

Downloading Show 1/3:

Downloading Viewpoint: Series 1
INFO: Preparing Request for Auth Info
INFO: Requesting Metadata.
DEBUG: Metadata response status code: 200
INFO: Metadata processed.
DEBUG: youtube-dl args:["https://www.itv.com/hub/viewpoint/10a0556a0003", "-f", "mp4/best", "-o", "/Volumes/video/0_New/Viewpoint/Viewpoint.s01e03.Episode 3.%(ext)s", "--verbose"]
DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator.app/Contents/MacOS:/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator.app/Contents/Resources"]
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.itv.com/hub/viewpoint/10a0556a0003', u'-f', u'mp4/best', u'-o', u'/Volumes/video/0_New/Viewpoint/Viewpoint.s01e03.Episode 3.%(ext)s', u'--verbose']
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2020.12.12
[debug] Python version 2.7.16 (CPython) - Darwin-19.6.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.2-tessus
[debug] Proxy map: {}
[debug] Using fake IP 25.161.248.252 (GB) as X-Forwarded-For.
[ITV] 10a0556a0003: Downloading webpage
[ITV] 10a0556a0003: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 2248, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
youtube-dl finished downloading

[torbs723]

Also noticed that the show is only available for 1 day and the latest news that Episode 6 will not be broadcast due to lead actor Noel Clarke allegations. Perhaps ITV has implemented actions to limit access?!

Yes due to the allegation the series is only available until Sunday. They have severely shortened the availability window.

The "500 Internal Server Error" you are seeing though is unrelated to this but is making it almost impossible to download new content from ITV

I've not been able to download episodes 3, 4 or 5

From ITV Hub you will not get them to download. You need to grab them from the STV site.

[Vangelis66]

I have posted an analysis of mine wrt to this new blockade by ITVHub people in the referenced yt-dl issue (28906, comment 830714363); the devs there, especially the Russian guy, are very "peculiar", I might get blocked for all I know, or my comment hidden/deleted (that's why I'm not linking the issue directly here 😉 ); I avoided to mention that other open source projects like GiA are reliant on yt-dl for ITV fetches, just in case...
I wish someone could figure out quickly what those (mean) ITV people did to cripple yt-dl... 💢

... I think it isn't yet another "coincidence", but I did observe, over the course of many years now, that always new draconian changes are being implemented in ITVHub (that break downloading 😞 ) just a short period prior to the Brit Awards shows being broadcast on ITV1/ITV2 (this year on May 11th); has been the case even since the era they used RTMPE streams 😠 ...

Kindest regards, stay safe 😄

[skovatch]

@Vangelis66 thank you once again for your excellent analysis of the situation. I read your comments in the yt-dl issue, and I think you've given them enough information to figure out how to react what's changed on ITV's end.

I think I've mentioned it elsewhere, but there's nothing else GiA can do here except wait for a fix and incorporate it in a future release. I'm monitoring that bug now and will update when there's something actionable for me.

[jbreef]

Been trying to submit a report on a failure detected attempting to download from itv with GiA, however, my issue was closed because it appeared to be a duplicate. Can't seem to find the duplicate issue so I guess I'm relegated to adding to this issue post. My download attempts are met with the message "youtube-dl download finished" and the request fails. Below is the log file I was able to generate. Yesterday I could simply restart the download and it would continue, but today my attempts to restart have resulted in failure.
Verbose failure.txt

[sleaux-meaux]

Use yt-dlp version 2021.12.01 or later

https://github.com/yt-dlp/yt-dlp/releases/latest

[torbs723]

the devs there, especially the Russian guy, are very "peculiar"

Is that the one that decided my message was off topic?

it may have already received the kiss of death by being labelled "geo-restricted" by the maintainers. It is difficult to know what that means (other than the maintainers stating the bleeding obvious)

They do like that "geo-restricted" label. And apparently I must NEVER question its use again...

[skovatch]

FWIW, I think @Vangelis66 pointed out that the geo-restricted label could mean you need to be in the UK or have VPN to work on it or verify any patches. There are all kinds of video providers they support that are geo-restricted -- trying to find a VPN for every possible provider is a slippery slope and I don't blame them for not wanting to bother.

@sleaux-meaux I appreciate you building a forked version for folks to use but I would strongly prefer to wait for an 'official' release before I include it in GiA. I can give it a try just to see how well it integrates, though.

[jefro108]

@sleaux-meaux I appreciate you building a forked version for folks to use but I would strongly prefer to wait for an 'official' release before I include it in GiA. I can give it a try just to see how well it integrates, though.

I've downloaded the compiled version by @sleaux-meaux and replaced it inside GiA by using Show Package Contents in Finder and GiA now downloads the itv hub programmes that wouldn't download previously! Great 😆

[torbs723]

@sleaux-meaux Is there any chance of a windows .exe version while we wait for youtube-dl to approve the patch and release it?

[cgareth]

@sleaux-meaux Is there any chance of a windows .exe version while we wait for youtube-dl to approve the patch and release it?

Or a step by step explanation of how to do this, Jefro?

[Vangelis66]

@torbs723 asked:

Is that the one that decided my message was off topic?

Yes...

They do like that "geo-restricted" label. And apparently I must NEVER question its use again...

😄

the geo-restricted label could mean you need to be in the UK or
have VPN to work on it or verify any patches.
There are all kinds of video providers they support that are geo-restricted --
trying to find a VPN for every possible provider is a slippery slope...

More slippery in the case of ITV 😠 ; being overseas, I can confirm they're being relentless at blacklisting (commercial) UK IPs, probably via a mechanism similar to EPD:

Enhanced Proxy Detection (EPD) lets you use services provided by our data provider,
GeoGuard, to apply proxy detection and location spoofing protection.
For example, you can set up the Enhanced Proxy Detection with GeoGuard behaviour
to identify requests for your content that have been redirected from an unwanted source
via a proxy. You can then allow, deny or redirect these requests.

So I can empathise with the two yt-dl developers being in quite a predicament when asked to troubleshoot geo-fenced sites... I wouldn't put any extra weight on the geo-restricted label being applied on either an issue or PR; it is what it is, i.e., according to sleaux-meaux, "stating the obvious"; he's fortunate to be UK-based, so he doesn't have to bother with ITV's strictest ACLs - but the yt-dl devs do... That's why, if you look closely at their README file in

https://github.com/ytdl-org/youtube-dl#how-can-i-speed-up-work-on-my-issue

they state:

If you can, offer proxy, VPN, or shell access to the youtube-dl developers.

@skovatch wrote:

I appreciate you building a forked version for folks to use

Ditto 👍

I would strongly prefer to wait for an 'official' release before I include it in GiA.

Yes, that is the "correct" thing to do... but let's get a "reality check", if you will 😉 :

1. sleaux-meaux hasn't yet been granted access to GH Actions for test workflows for his PR
2. The yt-dl dev who traditionally maintains itv.py (remitamine) is currently MIA 😞 ; he's probably the one to review and merge s-m's PR.

Merging this in the official way can take some time; after all, ITV.py is just one of many InfoExtractors inside yt-dl, while for GiA represents 50% of functionality...
So, having an interim "stopgap" is nice 👍 ...
Don't mean to sound disrespectful in the slightest, as the project's maintainer, it's always your call... 😄

FTR, this latest ITV breakage comes as a direct result of them moving head-on towards full-blown DRM on all their streams; I had warned about this as recently as January 13th of this year, in another exchange of mine with @torbs723:
(yt-dl issue 27582#issuecomment-759191537)

What I'm now dreading is the very probable event those last (DRM-free) ITV HLSe streams
(targeting older iOS devices) are decommissioned, then it'll be game over, sadly... 😢

So I'm currently enjoying it while it lasts... 😉

[jefro108]

an updated build of youtube-dl for GiA can be found at https://github.com/sleaux-meaux/youtube-dl-gia/releases/latest. It is youtube-dl 2021.04.26 with ITV patch applied, and --version will report 2021.04.26-gia_01. Download the youtube-dl file, run chmod a+x on it, then back up and replace the version in /Applications/Get iPlayer Automator.app/Contents/Resources.

@cgareth if you follow the instructions from @sleaux-meaux in the quote and open the GiA application contents using Show Package Contents and put the download into /Applications/Get iPlayer Automator.app/Contents/Resources (after running the chmod command using Terminal app)

So: download https://github.com/sleaux-meaux/youtube-dl-gia/releases/download/2021.04.26-gia_01/youtube-dl to your Downloads folder

Then:
Open Terminal app (/Applications/Utilities) and paste the following line into it

chmod a+x ~/Downloads/youtube-dl

Then copy and paste the youtube-dl using Finder inside the GiA app (using Show Package Contents) to the Resources folder (renaming the original to youtube-dl.bak just to be safe)

[Vangelis66]

@torbs723 wrote:

Is there any chance of a windows .exe version
while we wait for youtube-dl to approve the patch and release it?

I guess discussing Windows specifically in this tracker is probably OT...
Did you not read my comment in the yt-dl tracker (.../youtube-dl/issues/28906#issuecomment-831608015) where I mentioned info on how to run youtube-dl from source and pointed (you) to compilation instructions? Please also use your favourite search engine for more detailed answers...

I know how-to myself and have compiled a 32-bit .exe (Python 3.7.10 based) for my personal local use, but I think I would be frowned upon if I started distributing "here and there" compiled Windows executables... 🙄
Let's wait some more days, perhaps a week at most, until the devs come up with official Win binaries - if they stall, I imagine something could be arranged... No strings attached...

Apologies for the OT...

[torbs723]

@Vangelis66

Did you not read my comment in the yt-dl tracker (.../youtube-dl/issues/28906#issuecomment-831608015) where I mentioned info on how to run youtube-dl from source and pointed (you) to compilation instructions? Please also use your favourite search engine for more detailed answers...

Indeed I did have a read thank you Vangelis. Was hoping I wasn't going to have to learn to do a new thing though 😄

I know how-to myself and have compiled a 32-bit .exe (Python 3.7.10 based) for my personal local use

Now if I'm wanting to run the compiled .exe on a 32-bit system, does it need to be created with a 32-bit install of Python? I've managed to compiled it on a 64-bit system that I installed 64-bit Python on, and I've moved my new .exe over to a 32-bit machine and it says it isn't compatible. Does that mean I've ended up compiling a 64-bit version of youtube-dl?

Is there a switch I can use in the code to tell it to make it 32-bit or do I need to install a 32-bit version of Python and use that to create the .exe instead?

[Vangelis66]

@torbs723

Not wanting to further pollute this tracker with Windows-exclusive info, kindly read my reply to you over at the original yt-dl issue... 😜

[Rencroft]

@torbs723 would you mind sharing that 64-bit exe? It can be uploaded directly to my FF account here https://pastebin.com/RE32aQrK Only I will have access to it and I won't share it with anyone else, unless it's ok then I'd be happy to make a link available.

BTW a search suggests you need 32-bit Python to create a 32-bit compatible exe.

[torbs723]

@Rencroft I've realised that there's no crypto support in my original x64 and x86 versions, so downloading relies on ffmpeg. I've fixed this with my x86 now and will try to re-compile the x64 later.

[pebceb]

Thanks sleaux-meaux! Did that fix and can get my ITV shows again.

Much appreciated.

Peter

[Rencroft]

@torbs723 thanks really appreciate it. Going to take me ages to figure it out but won't miss any shows now before they expire. Your a hero.

[jefro108]

Thanks @sleaux-meaux for the update. Is it expected to get the message when I try and download directly with youtube-dl WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg and ffmpeg then takes over?
eg with the basic command ~/Downloads/youtube-dl https://www.itv.com/hub/superstore/2a5460a0104

[jbreef]

sleaux-meaux, great work on the youtube-dl. I did the replacement and the downloads from itv worked superbly. Thanks!

[jefro108]

Yes. To avoid delegation to ffmpeg you need to have the PyCrypto or PyCryptodome Python package available. GiA includes the former.

Thanks for the info. I installed PyCryptodome with the command
pip install pycryptodome
and it works perfectly now! 😊

[skovatch]

Fixed, for the time being, in 1.21.10.

[pebceb]

Does anyone have the problem where the cache take forever to update BBC shows? Any way to fix it?

I also have a problem with "use current webpage" in some versions even though I am using the correct browser as per my preferences and the page is at the front and on the show.

Thanks.

Peter

[skovatch]

Does anyone have the problem where the cache take forever to update BBC shows? Any way to fix it?

I also have a problem with "use current webpage" in some versions even though I am using the correct browser as per my preferences and the page is at the front and on the show.

Please open a new issue with steps to reproduce and information about your system -- application version, os version -- and a verbose log showing what happens when the problem occurs.