ITV Hub JSON error #28906
Comments
|
Same here, hopefully it's not DRM. |
|
I first came across this problem on the 28th April when trying to download the ITV news programmes for that day. Other programmes prior to the 28th April I haven't had a problem with the completion of the download. But then programmes such as, youtube-dl https://www.itv.com/hub/good-morning-britain/2a3211a3033 C:\Users\User\Videos\youtube-dl Downloads>youtube-dl https://www.itv.com/hub/lorraine/1a9360a2831 --verbose C:\Users\User\Videos\youtube-dl Downloads> |
|
Luckily at the moment, STV player still works, but anything from ITV2, 3, 4 or CITV would not be downloadable because STV is only the Channel 3 contractor for Scotland. And you have to hope they haven't opted out of the ITV schedule and are showing the show you want. |
500 Internal Server Errors are usually temporary. It means that for some reason, the server was unable to process the request that was sent to it. If it was DRM related, I would not expect a 500 error. Someone suggested in the other thread that if you kept trying, the programme would eventually download. That works for 403 Forbidden errors but doesn't appear to work for this. The problem seems to be that for some programmes the server is just finding it impossible to send the JSON playlist back which means that YT-DL has no way to attempt to download the programme. Why this only seems to affect some programmes and not others though is a bit of a mystery. |
|
It probably does not help, but I too am receiving this error. |
|
I've been noticing the same thing over the past day or two also. Here's another example of an identical failure. It doesn't appear retrying works, just flat out refuses to download however many times over a 12 hour window. Running same versions as @TIGHazard1. And, as reported, some downloads do work, just the ones that don't never seem to work. Have also tried with |
|
I don't own any ios devices so sadly can't test that. Is anyone else able to? |
|
I agree that many programs on ITV Hub now fail to download. On 28th April I was able to download URL's https://www.itv.com/hub/viewpoint/10a0556a0001 On 29th April I was unable to download URL https://www.itv.com/hub/viewpoint/10a0556a0003 Moreover, if I now try to download the first two above URL's they now fail with the same error message that others are reporting, namely: So ITV have changed something server side. I don't think it's DRM (yet), but maybe a missing header or auth token? |
I cannot see a 500 error being DRM related. The error shows before the download has even been attempted. If DRM made the videos impossible to download then they would fail with some sort of unable to download error not a server error. |
|
@gewyvoso I can confirm you get the http 500 error from your first link when I try yt-dl on OSX. I can also confirm that I can access the link from the itv hub with an old iphone using ios. In both cases using a VPN. Is this the info that you want? |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
is anyone able to set up man in the middle to view the traffic on working old ios? perhaps it's working in a different way from itv.py @dstftw, giving a geo tag to this is off topic, not the above two comments. one of my two linked threads gives detailed info about geo behaviour and that is different from what is occurring here. could be that it is more and different geo restriction and that you know something about it, so then please tell us that @LFS1 & @torbs723, i was surprised it did work. thank you for doing. that was one logical avenue to investigate. like i said, maybe the ytdl plugin is working differently from what itv are doing for real. if so, that could be a fix all this is academic to me, i'm not fussed, although i have heavily cached itv shows using this hls method til now. is there a place where people discuss the workings of these plugins? people seem to present good knowledge in these threads, which seems to have been picked up elsewhere. i would like to chat there, this forum here does not seem right for it |
|
@gewyvoso Thank you for the support
It is possible though that the streams we are having problems with are used by other devices possibly old old iPhones, iPads etc that maybe don't support ITV's app? |
|
this is what i mean. looking at the plugin code, it was like it was magicked into being. there was no explanation of its workings except by reading between the lines. happy to do that though, but what i would like to see is where discussions took place leading to that. that is where i can contribute and would like to, not here. until then i find myself holding back my thoughts |
|
(28934 is another one. looks like i commented there same minute you commented at 28939) |
|
If only people would spend 30 seconds doing a search before they post a new issue. |
|
Same problem here, but for me it only happens with videos that have appeared in the last day or two. I can still D/L older videos. For example, https://www.itv.com/hub/robin-of-sherwood/ROBa03a010, from 28 April, can be downloaded OK, but https://www.itv.com/hub/robin-of-sherwood/ROBa03a011, from 29 April, and https://www.itv.com/hub/robin-of-sherwood/ROBa03a012, from 30 April, both fail with the HTTP Error 500 with latest youtube-dl version 2021.04.26. e.g. [ITV] ROBa03a011: Downloading webpage |
|
@torbs723 wrote:
I was tagged (in a now hidden comment) to troubleshoot this... 😄 But first, let's discuss the While I am discussing geo-fencing, the ITV Hub people, some time in the month of March (2021), imposed even stricter geo-blocking schemes... 💢 It well appears that the ITV Hub people made some changes during these last days, certainly during last week, that make recently uploaded ITV hub shows inaccessible to Yerterday's episode: (the test was done using the very same connection/IP). Now, my deeper analysis was done focusing on Viewpoint-SE01EP03 (sadly, the whole series is due to expire imminently): So the very first thing I did was to emulate such a device in my browser, by setting up a site-specific-user-agent-override (SSUAO) for itv.com; depending on browser used, this can be achieved via an about:config setting (Firefox derived browsers) or, more commonly, via an extension (Chromium browsers) that spoofs your user-agent; please, don't ask for instructions/how-to here, it isn't the appropriate place...
Of course, ITVHub requires you to be logged-in to send any stream data to your browser (this hasn't been yet a requirement for yt-dl 👍 ); being logged in and with the above "mobile" UA, I inspected browser's Web Console (CTRL+SHIFT+K) and I clearly saw the POST XHR request to their playlist API: From that (see attached) JSON response, I extracted the master HLSe playlist URI: (the same URI can be seen and copied from within the browser's Web Console 😉 ), which looks like below: This URI is short-lived, you have to act swiftly before its auth-token expires... Feeding that directly to yt-dl, I can initiate a successful download:
I did spend a lot of time trying to emulate the above POST XHR my browser sent in a yt-dl command, by specifying all needed request headers (referer/user-agent/cookies etc.), I even tried the cookies.txt method, sadly all to no avail 😭 ... It's a real pity that I can't code (in Python or other language), I hope my findings can be put to good use by actual coders... In conclusion: ITV staff have implemented in recent days "changes" (I wouldn't rule out this having been done deliberately to thwart downloaders) that would render current itvIE useless once old shows expire, perhaps sooner if these changes propagate to all ITVhub content... My best wishes to all, my deepest thanks to the yt-dl dev team 👍 , Happy Easter Sunday to all those that observe it today... 😃 |
|
@Vangelis66 As always Vangelis, thank you so much for your detailed analysis of this issue. And thank you for your confirmation that this doesn't relate to DRM. |
|
Really appreciate the analysis @Vangelis66. Learnt something about spoofing the user agent as well. Taking it from here is beyond me but hopefuly the coding heroes can find some time and motivation to pick it up. Happy Easter Sunday. |
|
I too have just learnt about spoofing user agents and can concur with @Vangelis66 that downloads from ITV (UK based) are still possible as described above. Happy Easter. |
|
I asked a friend of mine in London to test yt-dl again fridays episode of the chase. He has confirmed that the same 500 error occurs irrespective of where you are located. I hope this helps. |
But is available with the @Vangelis66 method
|
|
OK, I think I got to the bottom of it... 🎉 https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0204 with the ones that are now "broken", e.g. (latest) "The Jonathan Ross Show -SE17E04": https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205 The point of comparison was the "data-video-variants" block; in the first case (working), it looks like below (pretty-printed, for clarity): data-video-variants="{
'dotcom':[
['hls','fairplay'],
['mpeg-dash','playready','outband-webvtt'],
['hls','fairplay','outband-webvtt'],
['hls','aes','outband-webvtt'],
['hls','aes'],
['mpeg-dash','widevine'],
['mpeg-dash','clearkey','outband-webvtt'],
['mpeg-dash','widevine','outband-webvtt'],
['mpeg-dash','playready'],
['mpeg-dash','clearkey']
],
'mobile':[
['mpeg-dash','clearkey'],
['mpeg-dash','widevine'],
['hls','fairplay'],
['hls','aes'],
['mpeg-dash','widevine','outband-webvtt']
]
}"In the second case (non-working shows, giving the "HTTP Error 500"), it looks like below: data-video-variants="{
'dotcom':[
['mpeg-dash','playready','outband-webvtt'],
['hls','fairplay','outband-webvtt'],
['mpeg-dash','widevine'],
['mpeg-dash','widevine','outband-webvtt'],
['hls','fairplay'],
['mpeg-dash','playready']
],
'mobile':[
['mpeg-dash','widevine'],
['mpeg-dash','clearkey'],
['hls','aes'],
['hls','fairplay'],
['mpeg-dash','widevine','outband-webvtt']
]
}"As you can see, the 'dotcom' group of variants was reduced from 10 to 6 members, the ones that were axed were "softer" encryption methods like (aes | clearkey): ['hls','aes','outband-webvtt'],
['hls','aes'],
['mpeg-dash','clearkey','outband-webvtt'],
['mpeg-dash','clearkey']which now leaves us with full-fledged DRM (playready | fairplay | widevine) variants within the 'dotcom' group... 😠 As far as youtube-dl/youtube_dl/extractor/itv.py Lines 88 to 94 in a0df8a0 the code above instructs it to request a ['hls','aes','outband-webvtt'] variant from within the 'dotcom' group; this variant is now absent in the non-working shows, the PlaylistAPI server can't grant As you can see, ITV Hub are going full-steam ahead with fully DRM-ing all their streams; as a most vocal opponent of DRM in every shape or form, I wish them (...) (fill-in the dots yourself, if you get my drift...). Workaround: The only remaining - 'min': ['hls', 'aes', 'outband-webvtt'],
- 'max': ['hls', 'aes', 'outband-webvtt']
+ 'min': ['hls', 'aes'],
+ 'max': ['hls', 'aes']
},
- 'platformTag': 'dotcom'
+ 'platformTag': 'mobile'... so I can fetch all these programmes currently broken (bound to increase in number, no doubt 😡 ), but without subs for them, even when subs are provided by ITV: Obviously, I now need a "real" coder that can harvest the webvtt subtitles from one of the other video-variants (either ['mpeg-dash','widevine','outband-webvtt'] from the 'mobile' group (should be easier?) or from one of the three belonging to the 'dotcom' group); any volunteers? 😄 |
|
Annoyingly I have no idea how to compile things if I made that very simple edit. I just download the .exe and run that usually. |
|
@sleaux-meaux : Many thanks 😄 !
For those hesitant to try the linked binary, VirusTotal log below: (only 1/69 score 👍 )
Preliminary tests have shown it to be working as expected: All in all, a sterling job 🥇 ... |
|
@sleaux-meaux Thank you. This works perfectly :) |
|
This worked great and solved the problem for me until I attempted any of these "https://www.itv.com/hub/ellens-game-of-games". |
|
The above (eg https://www.itv.com/hub/ellens-game-of-games/10a0434a0057, S04E09) and the show reported in #28910 are giving 404 rather than 500, so something else is up. The error occurs when POSTing /playlist/itvonline/ITV2/10_0434_0057.001 to magni.itv.com. This is the The data POSTed is (from l.125 ff. in the extractor): The |
|
No mobile streams, no downloads. That particular show may simply not be available via mobile. Episode noted in #28910 (comment) is not useful for comparison because it is no longer available, so now produces different 404. |
|
@CorrodedCoder wrote:
Yes, ALL the streams currently offered by ITVHub for any episode of "Ellen's Game Of Games" are fully DRM'ed (MPEG-DASH streams use one of Widevine/PlayReady, HLS streams use FairPlay) 😭 ; it appears the Ellen Degeneres Show (and, by association, Ellen's Game Of Games, themselves part of the full show 😉 ) is governed by a "special" rights contract, that excludes it from (older) iOS devices: https://help.itv.com/hc/en-us/articles/360003077994-Which-shows-do-rights-restrictions-apply-to-
(Considering MS have deprecated IE11, that last part might not be still valid... )
Exactly! Hence, as you said,
|
|
Great detective work, thanks for investigating this - your prior work/changes were much appreciated too. |
Considering ITV lost the rights to Pokemon or Champions League highlights a few years back, I would say that page hasn't been updated in a while. |
|
Excellent work, this method with that exact iPhone UA working today to grab the Euro 2020 Preview show :) |
Why would you need to apply the "iPhone UA" method TODAY? It is rather unfortunate that the solution to this issue, PR #28955, hasn't yet been accepted/merged officially (thus, is not present in the officially released binaries 😞 ), but third party https://github.com/sleaux-meaux/youtube-dl-gia/releases/tag/2021.04.26-gia_05 Were you not able to use one of them? 😲 |
and remove redundant tests :ci skip Ao-authored by: pukkandan, shirt-dev
|
Hello, this is my first time on here asking a question. Firstly I am not into computer programming in any way and find it hard to understand any of the above. I use youtube-dl to download BBC iplayer but had previously used it for ITV until they changed something as stated in the content of this thread. My query is, is there an easy way to download a new version of youtube-dl to allow me to download ITV shows? Is there a very basic guide with step by step instructions at all. Apologies if this is the wrong place to ask. Thanks for any help |
* commit '4c57dd270eb96c717331ed2abf13a3df9644f64c': [itv] fixed extraction (closes ytdl-org#28906)
Original PR: ytdl-org/youtube-dl#28955 (see also ytdl-org/youtube-dl#28906 (comment)) Closes #861, ytdl-org/youtube-dl#28906, ytdl-org/youtube-dl#29337, ytdl-org/youtube-dl#29190, ytdl-org/youtube-dl#28939, ytdl-org/youtube-dl#29620 Authored-by: coletdjnz, sleaux-meaux, Vangelis66
Original PR: ytdl-org/youtube-dl#28955 (see also ytdl-org/youtube-dl#28906 (comment)) Closes yt-dlp#861, ytdl-org/youtube-dl#28906, ytdl-org/youtube-dl#29337, ytdl-org/youtube-dl#29190, ytdl-org/youtube-dl#28939, ytdl-org/youtube-dl#29620 Authored-by: coletdjnz, sleaux-meaux, Vangelis66
ITV changes require media stream and subtitles to be taken from different playlist resources. Original commit 4c57dd2 by @sleaux-meaux 3 May 2021
* commit 'c6001d56b2d02d5d470a744f052f49c3a7f7b529': [ITV] Apply default vanilla UA to avoid site blocking Work-around duration parsing pending fix to utils.parse_duration Back-port ITV extractor from yt-dlp PR ytdl-org#1776 [itv] fixed extraction (closes ytdl-org#28906)
Checklist
Verbose log
Description
Several programmes on ITV Hub now fail to download. URL's such as
https://www.itv.com/hub/the-chase/1a7842a1761
https://www.itv.com/hub/tipping-point/2a1875a1123
https://www.itv.com/hub/tenable/10a0805a0021
fail to download though some like
https://www.itv.com/hub/lorraine/1a9360a2830
continue to work
The text was updated successfully, but these errors were encountered: