Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ITV Downloads still fail after update to version 1.21.10 #339

Open
damndingo opened this issue May 6, 2021 · 39 comments
Open

ITV Downloads still fail after update to version 1.21.10 #339

damndingo opened this issue May 6, 2021 · 39 comments

Comments

@damndingo
Copy link

Hi,
downloads from itv still stop almost instantly after starting and fail.

INFO: Loading proxy settings...
INFO: No proxy to load
INFO: Proxy load complete.
INFO: No proxy will be used
AppController: Starting Downloads

Downloading Show 1/1:

Downloading Superstore: Series 6
INFO: Preparing Request for Auth Info
INFO: Metadata processed.
youtube-dl finished downloading
AppController: Downloads Finished

@blloyd78
Copy link

blloyd78 commented May 6, 2021

Can confirm this issue - I received exactly the same results when testing.

@nitramlatep
Copy link

Same for me, from the UK, so no VPN issue. I tried the Jonathan Ross Show. Failed immediately.

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

The Python crypto support was changed in 1.2.10, and appears to be broken. If you see this before upgrading, don't upgrade and wait for a fix. If you have already upgraded, you can roll back to 1.21.9 and replace the bundled version of youtube-dl as describe in #331 (comment) . But the best course of action is probably to wait. This shouldn't affect BBC programmes.

@ghost
Copy link

ghost commented May 6, 2021

thanks @sleaux-meaux !

I had already upgraded, but your fix worked perfectly!

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

It appears that the Python 3 version of PyCryptodome was included in GiA, but that isn't digestible to Python 2.

EDIT: I think a Python 2 version of PyCryptodome can be used with both Python 2 and 3.

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

Ignore my earlier suggestion to temporarily replace /Applications/Get iPlayer Automator.app/Contents/Resources/Crypto with the equivalent folder from 1.21.9. That is not a good idea if you are using Python 3 with GiA since the old PyCrypto package won't load and ITV downloads will be delegated to ffmpeg.

@ghost
Copy link

ghost commented May 6, 2021

OK, already did that and downloaded some things. Are they compromised?

I do have Python 3 on my system but how do I know if GiA is using it (as opposed to v2).

Should I remove GiA and follow your first recommendation (install v.1.21.9 and replace YouTube.dl)?

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

Downloads won't be compromised. They either decrypt or they don't.

GiA 1.2.10 tries these locations for Python, in this order. Check versions with:

/usr/local/bin/python --version
/usr/bin/python --version

GiA 1.2.9 and earlier don't check /usr/local/bin. With verbose logging, youtube-dl will also print a line that looks this:

[debug] Python version 3.9.4 (CPython) - macOS-10.15.7-x86_64-i386-64bit

If you replaced Crypto folder and it's working, it means Python 2 is being used, so no need to change. If something goes wrong, you can always reinstall GiA and start over.

@ghost
Copy link

ghost commented May 6, 2021

thanks

for interest,
/usr/local/bin/python --version gave me nothing
/usr/bin/python --version gave me 2.7.16

python --version gave me 2.7.16
python3 --version gave me 3.9.0

@sleaux-meaux
Copy link

youtube-dl only looks for "python", so "python3" would be ignored. For completeness, you should check full paths (/usr/local/bin/python3 and /usr/bin/python3) so you know what is where.

@ghost
Copy link

ghost commented May 6, 2021

cheers.
it's in /usr/local/bin/python3

@ChrisV56
Copy link

ChrisV56 commented May 6, 2021

I am also having problems with ITV downloads, I was able to download 2 episodes of series 1 of Bancroft but the other 2 would not download and the same with episode 1 of series 2. Failed download solutions shows problem unknown, logs detail in attached file
IPA Log 6th May 21.docx
:

@skovatch
Copy link

skovatch commented May 6, 2021

It appears that the Python 3 version of PyCryptodome was included in GiA, but that isn't digestible to Python 2.

EDIT: I think a Python 2 version of PyCryptodome can be used with both Python 2 and 3.

PyCryptodome is supposed to support Python 2.6, 2.7 and 3.4 and up, which is why I didn't expect a problem. I copied over what pip installed into my Python3 installation.

For those of you still having problems, can you post a verbose log? I need to see how youtube-dl is failing for you.

@skovatch
Copy link

skovatch commented May 6, 2021

I can back out the crypto and path changes, but I'd like to understand a bit more why it's okay on my system and not others.

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

Error shown below, running under Python 2. There are separate PyCryptodome wheels installed for Python 2 and 3. The Python 3 wheel is built against Python 3 stable ABI, thus not compatible with Python 2, but the Python 2 wheel works with Python 3. Install PyCryptodome in Python 2 and copy its Crypto folder.

Downloading The Jonathan Ross Show: Series 17
INFO: Preparing Request for Auth Info
INFO: Requesting Metadata.
DEBUG: Metadata response status code: 200
INFO: Metadata processed.
DEBUG: youtube-dl args:["https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205", "-f", "mp4/best", "-o", "/Users/xxx/Movies/TV Shows/The Jonathan Ross Show - Series 17/The Jonathan Ross Show - Series 17.s17e04.Episode 4.%(ext)s", "--write-sub", "-k", "--verbose"]
DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator.app/Contents/MacOS:/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/usr/local/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator.app/Contents/Resources"]
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/__main__.py", line 16, in <module>
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/__init__.py", line 15, in <module>
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/options.py", line 8, in <module>
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/downloader/__init__.py", line 5, in <module>
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/downloader/hls.py", line 6, in <module>
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Cipher/__init__.py", line 27, in <module>
    from Crypto.Cipher._mode_ecb import _create_ecb_cipher
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Cipher/_mode_ecb.py", line 47, in <module>
    """
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/_raw_api.py", line 297, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.so': dlopen(/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/../Cipher/_raw_ecb.so, 6): image not found, Trying '_raw_ecbmodule.so': dlopen(/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/../Cipher/_raw_ecbmodule.so, 6): image not found
youtube-dl finished downloading
AppController: Downloads Finished

@sleaux-meaux
Copy link

sleaux-meaux commented May 6, 2021

Correction: Don't just roll back to PyCrypto - it doesn't seem to work with Python 3. Forgot my own advice above. Would be OK if you return to forcing use of /usr/bin/python, though.

@m1ctk
Copy link

m1ctk commented May 6, 2021

No go for me using Catalina. I was so pleased to see the prompt for the 21 update but alas it doesn't work. C'est la vie

@skovatch
Copy link

skovatch commented May 6, 2021

Can I get a volunteer to try this test build? I think I have figured out how to include python in the app so GiA isn't dependent on a system-installed python.

Link to GDrive

If you can download ITV content (or you can't) with this build let me know. I'd especially like folks running earlier versions of macOS to try it, as I'm not sure the included python runs on older versions of macOS.

@ian02054
Copy link

ian02054 commented May 6, 2021 via email

@ian02054
Copy link

ian02054 commented May 6, 2021 via email

@m1ctk
Copy link

m1ctk commented May 6, 2021

All I really want mostly is ITV news for my youtube channel. As I said I can't even watch via the kodi 'itv player' add on anymore :(

@skovatch
Copy link

skovatch commented May 6, 2021

just failed with problem unknown

What version of macOS are you using?

@sleaux-meaux
Copy link

URL: https://www.itv.com/hub/superstore/2a5460a0107

Runs on Catalina, but there is no crypto support bundled, so delegates to ffmpeg. Fails on Yosemite, with no clues in the log. I don't think you can just ship a virtualenv created with Homebrew Python. The included python binary references the Python framework in its Homebrew location, which won't exist for most people. I expect it only ran for me on Catalina because I have Homebrew Python installed.

@skovatch
Copy link

skovatch commented May 6, 2021

Bah. I thought "python venv --copies" copies in everything you need for a python distribution, but I guess not.

@ian02054
Copy link

ian02054 commented May 6, 2021 via email

@skovatch
Copy link

skovatch commented May 6, 2021

OK, I give up for now. I've reverted out the crypto and path changes and pushed a new version.

@m1ctk
Copy link

m1ctk commented May 6, 2021

URL: https://www.itv.com/hub/superstore/2a5460a0107

Runs on Catalina, but there is no crypto support bundled, so delegates to ffmpeg. Fails on Yosemite, with no clues in the log. I don't think you can just ship a virtualenv created with Homebrew Python. The included python binary references the Python framework in its Homebrew location, which won't exist for most people. I expect it only ran for me on Catalina because I have Homebrew Python installed.

It doesn't run on Catalina with me

@blloyd78
Copy link

blloyd78 commented May 6, 2021

OK, I give up for now. I've reverted out the crypto and path changes and pushed a new version.

Can confirm the rolled-back version is working perfectly on Big Sur now

@skovatch
Copy link

skovatch commented May 6, 2021

As an alternative, I could supply youtube-dl with Python 3 bundled in (via PyInstaller).

That might actually work... I have the youtube-dl sources so I built it and then pyinstaller. I was able to get a runnable youtube-dl distribution. I don't have your patch, so I'd have to get that and rebuild it. I'll work with it some more after work today.

@Vangelis66
Copy link

Vangelis66 commented May 7, 2021

@skovatch wrote:

PyCryptodome is supposed to support Python 2.6, 2.7 and 3.4 and up,
which is why I didn't expect a problem

FTR, this is only true up to pycryptodome==3.9.9; BTW, the author only provides the following wheels:

pycryptodome-3.9.9-cp27-cp27m-macosx_10_6_intel.whl (14.1 MB) 
pycryptodome-3.9.9-cp35-cp35m-macosx_10_6_intel.whl (13.3 MB) 
pycryptodome-3.9.9-cp36-cp36m-macosx_10_6_intel.whl (13.3 MB) 
pycryptodome-3.9.9-cp37-cp37m-macosx_10_6_intel.whl (14.1 MB) 
pycryptodome-3.9.9-cp38-cp38-macosx_10_9_x86_64.whl (13.3 MB) 

MacOS X 10.6 supported on Python 2.7.x/3.5.x/3.6.x/3.7.x;
MacOS X 10.9 supported on Python 3.8.x ...

Current pycryptodome==3.10.1 dropped support for Python 2.6 and 3.4, so only 2.7.x and >=3.5.x are supported; official wheels include just

pycryptodome-3.10.1-cp27-cp27m-macosx_10_9_x86_64.whl (1.5 MB) 
pycryptodome-3.10.1-cp35-abi3-macosx_10_9_x86_64.whl (1.5 MB)

i.e. MacOS X 10.6 support is dropped; the abi3 variety should be install-able on Python >= 3.5.0

The PyCryptodome module comprises C extensions (.so files - .pyd on Windows) that need to have been compiled by the same (or compatible) version of the C compiler used to compile the respective Python binaries - thus, interchange-abitlity between the cp27-cp27m and cp35-abi3 flavours isn't a given... 😞

The dead (but not buried 😉 ) PyCrypto module is only officially available as source, i.e. no wheels; it was tested against Python 2.1 through to 3.3; it, too, includes C extensions; in theory, it is possible to compile it from source (and install) on a Python 3 enabled Mac, but you should use the same compiler (OS X SDK?) your Python 3 version was built with...
Of possible relevance:
https://www.javaer101.com/en/article/14576333.html

@ghost
Copy link

ghost commented Jul 15, 2021

I saw this error again this morning.

I had 1 successful ITV download and one failed (consistently). Same show. Log below...

Get iPlayer Automator 1.21.11 Initialized.
GetITVShows: ITV Cache Update Starting 
<AppController: 0x7fb09273bf30>: Updating Programme Index Feeds...

/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/perl/bin/get_iplayer
-e86400
--type=tv,radio
--refresh
--nopurge
--profile-dir=/Users/jameshenderson/Library/Application Support/Get iPlayer Automator
.*
INFO: Indexing tv programmes (concurrent)
GetITVShows (Info): Episodes: 3550 Today Programmes: 351 
GetITVShows: Update Finished
INFO: Added 9 tv programmes to cache
INFO: Indexing radio programmes (concurrent)
INFO: Added 4 radio programmes to cache
INFO: 22219 matching programmes
<AppController: 0x7fb09273bf30>: Index Updated.
/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/perl/bin/get_iplayer
--nocopyright
-e60480000000000000
--type=tv,itv,radio
--listformat
SearchResult|<pid>|<available>|<type>|<name>|<episode>|<channel>|<seriesnum>|<episodenum>|<desc>|<thumbnail>|<web>|<available>
--long
--nopurge
--search
superstore
--profile-dir=/Users/jameshenderson/Library/Application Support/Get iPlayer Automator


INFO: Loading proxy settings...
INFO: No proxy to load
INFO: Proxy load complete.
INFO: No proxy will be used
AppController: Starting Downloads


Downloading Show 1/1:

Downloading Superstore: Series 3
INFO: Preparing Request for Auth Info
INFO: Requesting Metadata.
DEBUG: Metadata response status code: 200
INFO: Metadata processed.
DEBUG: youtube-dl args:["https://www.itv.com/hub/superstore/2a5460a0042", "-f", "mp4/best", "-o", "/Users/jameshenderson/Downloads/downloads - GiA/Superstore - Series 3/Superstore - Series 3.s03e09.Golden Globes Party.%(ext)s", "--verbose"]
DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator.app/Contents/MacOS:/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator.app/Contents/Resources"]
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.itv.com/hub/superstore/2a5460a0042', u'-f', u'mp4/best', u'-o', u'/Users/jameshenderson/Downloads/downloads - GiA/Superstore - Series 3/Superstore - Series 3.s03e09.Golden Globes Party.%(ext)s', u'--verbose']
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2021.04.26-gia_01
[debug] Python version 2.7.16 (CPython) - Darwin-20.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.2-tessus
[debug] Proxy map: {}
[debug] Using fake IP 25.139.171.87 (GB) as X-Forwarded-For.
[ITV] 2a5460a0042: Downloading webpage
[ITV] 2a5460a0042: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
youtube-dl finished downloading
AppController: Downloads Finished

@sleaux-meaux
Copy link

You cannot download episodes that are not available to mobile devices, as is the case here. If you have a suitable phone, check the episode there first. If it isn't available on your phone, it's a good bet it won't be available to download.

@ghost
Copy link

ghost commented Jul 16, 2021

OK, but does that make sense given I downloaded the episodes before and after (just checking)?

@pmhird
Copy link

pmhird commented Sep 19, 2021

I've come to this party a little late, but my ITV downloads have also stopped working, but I haven't updated GIA, still running 1.21.2 on El Capitan. It was working fine for ITV last I checked, but I don't download from them that often. Has this issue somehow affected older un-updated versions too?

@ghost
Copy link

ghost commented Sep 20, 2021

hi @pmhird I just downloaded Endeavour s08e02 just fine.

I am running GiA 1.21.12 on Big Sur.

@pmhird
Copy link

pmhird commented Sep 30, 2021

I've now updated to 1.21.12 on El Capitan. BBC still working absolutely fine, but I can't get anything to succeed from ITV. With the latest update to 1.21.12 do I still need ot re-do the little Python 2.7 trick outlined in issue 277?

@pmhird
Copy link

pmhird commented Oct 1, 2021

Can anyone help with this? It seems to be failing at the "Downloading m3u8 information" stage, suggesting it can't find a corresponding mobile format episode, but I checked and I can watch the episode (Hollington Drive S01E01) on both web browser and itvhub app. What is going on? It seems to affect all ITV shows for me, can't get anything to download at all. BBC both TV and radio are fine, but seem to take a lot longer to download after 1.21.12? I'm using El Capitan. Thanks in advance, here's the relevant bit of the log where the ITV downloads fail:

[ITV] 10a0552a0001: Downloading m3u8 information
WARNING: Failed to download m3u8 information: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)>
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper
return func(self, *args, **kwargs)
File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info
ie_result = ie.extract(url)
File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
ie_result = self._real_extract(url)
File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/itv.py", line 121, in _real_extract
self._sort_formats(formats)
File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 1374, in _sort_formats
raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
youtube-dl finished downloading
AppController: Downloads Finished

@ghost
Copy link

ghost commented Oct 1, 2021

I don’t use El Capitan, but just to confirm, are you using a VPN?

@pmhird
Copy link

pmhird commented Oct 1, 2021

Yes I am on a VPN, but the same VPN works fine for BBC and also for viewing ITV in a web browser or mobile app...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants