ITV extractor fails for some recently broadcast programmes

[Vangelis66]

Hello 😃

• I've verified and I assure that I'm running youtube-dl 2018.04.16

I have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

The URL in question is https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
Yes, it does play, proof below (you need an ITVHub account and a UK IP:

but that fact has little impact on youtube-dl, since desktop browsers use DRM'd (ClearKey+WidevineCDM) MPEG-DASH streams, that can't be decrypted+dumped by yt-dl 😢 !

OT: ITVHub have upped their tactics again recently: Not only have they blocked streams at CDN level (see #15925), now at least two of my UK VPNs fail to work at the very last step of ClearKey acquisition,

POST XHR https://itvpnp.live.ott.irdeto.com/ClearKey/getkeys?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-5159-0034-001_18

making watching in the desktop browser impossible 😠 ; a different method of geo-fencing evasion was used for the test above...

What would be of more value to the yt-dl devs is the fact pretending to be an iPad browser (user-agent spoofing) will make AppleHLS playlists show up inside Webconsole, e.g.

https://itvpnpmobile-a.akamaihd.net/2-5159-0034-001/6/1/VAR003/2-5159-0034-001_6_1_VAR003.ism/.m3u8?hdnea=st=1524407352~exp=1524428952~acl=/*~data=nohubplus~hmac=f3128ea08d398d39c6d5ac2296e85c0af232f5e46e6d44b985f51d5e30539cbc

which means (downloadable) HLS streams do exist for that show!

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)

Verbose cmd issued:
youtube-dl "https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034" -v
Cmd Prompt Window output:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/james-martins-saturday-morn
ing/2a5159a0034', '-v']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2018.04.16
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6002-SP2
[debug] exe versions: ffmpeg N-90535-gcdd6a52, ffprobe N-90535-gcdd6a52, rtmpdum
p 2.4
[debug] Using fake IP 25.71.195.241 (GB) as X-Forwarded-For.
[ITV] 2a5159a0034: Downloading webpage
[ITV] 2a5159a0034: Downloading XML
ERROR: ITV said: No match found for production id: 2/5159/0034#001
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\bu
ild\youtube_dl\YoutubeDL.py", line 789, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\bu
ild\youtube_dl\extractor\common.py", line 440, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\bu
ild\youtube_dl\extractor\itv.py", line 132, in _real_extract
youtube_dl.utils.ExtractorError: ITV said: No match found for production id: 2/5
159/0034#001

Description of your issue, suggested solution and other information

Reported first at the Get iPlayer Automator issue tracker here; GiA now uses yt-dl as an ITVHub downloader backend...
Can reproduce on Windows, too...
I hope it's an easy fix (probably caused by an ITVHub change 😠 , those people would make everything in their power to ruin things for us... )

Thanks in advance, thanks for the ITV extractor in general 👍

[Vangelis66]

@dstftw added the account-needed label

Well yes, while an account is technically needed to access streams in a desktop browser, so far youtube-dl could successfully bypass that requirement 😉 ; if you do feel an existing ITVHub account could expedite the resolution of this issue, just chime-in here and I'd be willing to forward my credentials to your public GitHub e-mail address...

And another ITVHub programme:

https://www.itv.com/hub/harry-hills-alien-fun-capsule/2a4728a0009

has emerged in the last hours, erroring out in the same manner:

youtube-dl -F "https://www.itv.com/hub/harry-hills-alien-fun-capsule/2a4728a0009"
[ITV] 2a4728a0009: Downloading webpage
[ITV] 2a4728a0009: Downloading XML
ERROR: ITV said: No match found for production id: 2/4728/0009#001

This suggests reported issue is not an isolated case, both originally reported id and this latest one have been broadcast first last Saturday (Apr 21st 2018); hence issue title has been changed...

[torbs723]

Thanks for raising this Vangelis. I was wondering what was happening with some programmes. How do you get the playlist to show in the web console though? I am on Chrome and have switched the user agent to iPad, left click on the page, Inspect but can't find anything useful.

[ravonel]

I've been hitting the same issue recently. Though I'm in the US, sometimes downloads work without issue, sometimes I need to use a VPN, and sometimes not at all, giving the same error as above. For the same series, I've hit all three situations in the last few weeks. In the case when it throws an error, I'm using a VPN and can still play the ep through my browser.

https://www.itv.com/hub/emmerdale/1a8694a8132 is an example of an error while on a VPN.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.itv.com/hub/emmerdale/1a8694a8132
']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2018.04.25
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-87353-g183fd30, ffprobe N-87353-g183fd30, phantom
js 2.1.1, rtmpdump 2.3
[debug] Proxy map: {}
[debug] Using fake IP 25.77.104.187 (GB) as X-Forwarded-For.
[ITV] 1a8694a8132: Downloading webpage
[ITV] 1a8694a8132: Downloading XML
ERROR: ITV said: No match found for production id: 1/8694/8132#001
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\YoutubeDL.py", line 789, in extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\extractor\common.py", line 440, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\extractor\itv.py", line 132, in _real_extract
youtube_dl.utils.ExtractorError: ITV said: No match found for production id: 1/8
694/8132#001

[Vangelis66]

@dstftw

Hello, again 😄
I am aware we're not supposed to comment frequently on open issues, as this can be viewed as an attempt to bump, but in fact I have now further input on this...

For starters, a third show has come up which produces the same error,
https://www.itv.com/hub/american-dad/2a4263a0229
and if I include the one posted by @ravonel
https://www.itv.com/hub/emmerdale/1a8694a8132
that would make it a total of four (4) recent programmes:

https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
https://www.itv.com/hub/harry-hills-alien-fun-capsule/2a4728a0009
https://www.itv.com/hub/american-dad/2a4263a0229
https://www.itv.com/hub/emmerdale/1a8694a8132

documented in this thread that give the error:
ERROR: ITV said: No match found for production id: <string>

In addition to the above, at least two more (recent) programmes

https://www.itv.com/hub/whos-doing-the-dishes/2a2898a0024
https://www.itv.com/hub/american-dad/2a4263a0030

produce a slightly different error:
ERROR: ITV said: The requested content is not available to this platform
but to the same effect, that is, no downloadable streams are found by yt-dl;
if it helps, verbose log for one of them:

youtube-dl "https://www.itv.com/hub/american-dad/2a4263a0030" -v =>

[ITV] 2a4263a0229: Downloading webpage
[ITV] 2a4263a0229: Downloading XML
ERROR: ITV said: No match found for production id: 2/4263/0229#001

D:\Vangelis\Installers\youtube-dl>youtube-dl "https://www.itv.com/hub/american-d
ad/2a4263a0030" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/american-dad/2a4263a0030',
'-v']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2018.04.25
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6002-SP2
[debug] exe versions: ffmpeg N-90535-gcdd6a52, ffprobe N-90535-gcdd6a52, rtmpdum
p 2.4
[debug] Proxy map: {}
[debug] Using fake IP 25.119.101.156 (GB) as X-Forwarded-For.
[ITV] 2a4263a0030: Downloading webpage
[ITV] 2a4263a0030: Downloading XML
ERROR: ITV said: The requested content is not available to this platform
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\YoutubeDL.py", line 789, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\extractor\common.py", line 440, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpad1hmn8r\bu
ild\youtube_dl\extractor\itv.py", line 132, in _real_extract
youtube_dl.utils.ExtractorError: ITV said: The requested content is not availabl
e to this platform

Again, it can be proved that downloadable Apple HLS streams (aes-128 encrypted) do exist
for both of them, but because of a change on ITV Hub, they evade yt-dl's detection...

More (helpful) background:

I have reached to the UK-based maintainer of get_flash_videos, a perl script, and raised issue monsieurvideo/get-flash-videos#224; the dev came back with a fix; it would appear the affected programmes display a different (to InvalidEntity) faultcode when the playlist service is queried for RTMPE streams, as a result GFV would not then proceed to query for HLS streams.
Looking at the code of ITV.py (and remember, I know absolutely 0 python), I see line:

            elif fault_code != 'InvalidEntity':

which might be a good place to start fixing yt-dl...

The fixed GFV script (v1.25.99.02) is now able to download HLS streams for all 4 shows with the
ERROR: ITV said: No match found for production id: <string> error message in yt-dl, but, interestingly
enough, is also able to fetch HLS streams for the two other shows with the
ERROR: ITV said: The requested content is not available to this platform ;
so possibly related bugs in yt-dl.

Thanks again for this wonderful application, take all the time you need for fixing ITV downloads...

Best regards

[Vangelis66]

@ravonel wrote:

Though I'm in the US, sometimes downloads work without issue,

Hi 😃 ; you are accessing the legacy RTMPE streams that do not geo-fence (yet?)

sometimes I need to use a VPN,

Only geo-fenced HLS streams are available for a particular show...

and sometimes not at all, giving the same error as above

In some rare occasions, there actually won't be any RTMPE/HLS streams for yt-dl to access, but most probably you are being hit by the bug I reported here... (i.e. failure to identify existing HLS streams).

and can still play the ep through my browser.

These are DRM'ed (cenc) MPEG-DASH streams that yt-dl is incapable of decrypting (even if made able to download)
Read more at #15925

[ravonel]

Sorry to bug you again about this, after updating this still wasn't working for me yesterday and I waited a day to try again, still getting the same old error, tried several different VPNs, all that can play the file in browser.

[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.itv.com/hub/coronation-street/1a0
694a9445']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2018.05.01
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-87353-g183fd30, ffprobe N-87353-g183fd30, phantom
js 2.1.1, rtmpdump 2.3
[debug] Proxy map: {}
[debug] Using fake IP 25.102.97.235 (GB) as X-Forwarded-For.
[ITV] 1a0694a9445: Downloading webpage
[ITV] 1a0694a9445: Downloading XML
ERROR: ITV said: No match found for production id: 1/0694/9445#001
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpulhs0gaq\bu
ild\youtube_dl\YoutubeDL.py", line 789, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpulhs0gaq\bu
ild\youtube_dl\extractor\common.py", line 440, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpulhs0gaq\bu
ild\youtube_dl\extractor\itv.py", line 132, in _real_extract
youtube_dl.utils.ExtractorError: ITV said: No match found for production id: 1/0
694/9445#001

[dstftw]

It's not released.