Skip to content

Commit

Permalink
[ivi] sign content request only when pycryptodome is available
Browse files Browse the repository at this point in the history
  • Loading branch information
@remitamine
remitamine committed Nov 15, 2019
1 parent 656c200 commit 1bba88e
Showing 1 changed file with 24 additions and 18 deletions.
42 changes: 24 additions & 18 deletions youtube_dl/extractor/ivi.py
View file
Expand Up @@ -80,38 +80,42 @@ class IviIE(InfoExtractor):
'MP4-SHQ', 'MP4-HD720', 'MP4-HD1080')

def _real_extract(self, url):
try:
from Crypto.Cipher import Blowfish
from Crypto.Hash import CMAC
except ImportError:
raise ExtractorError('pycrypto not found. Please install it.', expected=True)

video_id = self._match_id(url)

timestamp = self._download_json(
self._LIGHT_URL, video_id,
'Downloading timestamp JSON', data=json.dumps({
'method': 'da.timestamp.get',
'params': []
}).encode())['result']

data = json.dumps({
'method': 'da.content.get',
'params': [
video_id, {
'site': 's353',
'site': 's%d',
'referrer': 'http://www.ivi.ru/watch/%s' % video_id,
'contentid': video_id
}
]
}).encode()

video_json = self._download_json(
self._LIGHT_URL, video_id,
'Downloading video JSON', data=data, query={
try:
from Crypto.Cipher import Blowfish
from Crypto.Hash import CMAC

timestamp = self._download_json(
self._LIGHT_URL, video_id,
'Downloading timestamp JSON', data=json.dumps({
'method': 'da.timestamp.get',
'params': []
}).encode())['result']

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 16, 2019

Collaborator

Also if this fails extraction will fail though it may have tried s183. So it's more likely that is should try all applicable methods in a row.

data = data % 353
query = {
'ts': timestamp,
'sign': CMAC.new(self._LIGHT_KEY, timestamp.encode() + data, Blowfish).hexdigest(),

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 15, 2019

Collaborator

Does not seem to work:

PS C:\Dev\youtube-dl\master> py -2.7 .\youtube_dl\__main__.py https://www.ivi.ru/watch/137070 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.ivi.ru/watch/137070', u'-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2019.11.05
[debug] Git HEAD: 7360c06
[debug] Python version 2.7.12 (CPython) - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-85653-gb4330a0, ffprobe N-85653-gb4330a0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ivi] 137070: Downloading timestamp JSON
Traceback (most recent call last):
  File ".\youtube_dl\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\Dev\youtube-dl\master\youtube_dl\__init__.py", line 474, in main
    _real_main(argv)
  File "C:\Dev\youtube-dl\master\youtube_dl\__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 2018, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\Dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "C:\Dev\youtube-dl\master\youtube_dl\extractor\common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "C:\Dev\youtube-dl\master\youtube_dl\extractor\ivi.py", line 110, in _real_extract
    'sign': CMAC.new(self._LIGHT_KEY, timestamp.encode() + data, Blowfish).hexdigest(),
  File "C:\Python\Python27\lib\site-packages\Crypto\Hash\CMAC.py", line 225, in hexdigest
    for x in tuple(self.digest())])
  File "C:\Python\Python27\lib\site-packages\Crypto\Hash\CMAC.py", line 210, in digest
    pt = strxor(strxor(self._last_ct, partial), self._k2)
TypeError: argument 2 must be string, not bytearray

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 15, 2019

Author Collaborator

it does work for me on Python 3.8, will install PyCryptodome on python2 and check this problem.

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 15, 2019

Collaborator

FYI, it does not work for me on 3.7 either with the same error.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 15, 2019

Author Collaborator

works fine for me on both Python 3.8 and 2.7.17(not sure if it's related to the OS or PyCryptodome version), I'm using PyCryptodome 3.9.1, I'm not sure if it's better to revert the change completely until there is a better fix or just keep as it might work for some of the users.

 python2 __main__.py --proxy PROXY -v -F https://www.ivi.ru/watch/137070
[debug] System config: []
[debug] User config: [u'-f', u'best[height<=768]/bestvideo[height<=768]+bestaudio/best', u'--hls-prefer-ffmpeg', u'--external-downloader', u'curl', u'--sub-format', u'ass/tt/ttml/dfxp/scc/vtt/srt/sami/best']
[debug] Custom config: []
[debug] Command-line args: [u'--proxy', u'PROXY', u'-v', u'-F', u'https://www.ivi.ru/watch/137070']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.11.05
[debug] Git HEAD: 1bba88efc
[debug] Python version 2.7.17 (CPython) - Linux-5.3.10-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {u'http': u'PROXY', u'https': u'PROXY'}
[ivi] 137070: Downloading timestamp JSON
[ivi] 137070: Downloading video JSON
[ivi] 137070: Downloading webpage
[info] Available formats for 137070:
format code  extension  resolution note
MP4-lo       mp4        unknown    520.61MiB
MP4-hi       mp4        unknown    1.01GiB
MP4-HD720    mp4        unknown    2.31GiB (best)

python __main__.py --proxy PROXY -v -F https://www.ivi.ru/watch/137070
[debug] System config: []
[debug] User config: ['-f', 'best[height<=768]/bestvideo[height<=768]+bestaudio/best', '--hls-prefer-ffmpeg', '--external-downloader', 'curl', '--sub-format', 'ass/tt/ttml/dfxp/scc/vtt/srt/sami/best']
[debug] Custom config: []
[debug] Command-line args: ['--proxy', 'PROXY', '-v', '-F', 'https://www.ivi.ru/watch/137070']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2019.11.05
[debug] Git HEAD: 1bba88efc
[debug] Python version 3.8.0 (CPython) - Linux-5.3.10-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'http': 'PROXY', 'https': 'PROXY'}
[ivi] 137070: Downloading timestamp JSON
[ivi] 137070: Downloading video JSON
[ivi] 137070: Downloading webpage
[info] Available formats for 137070:
format code  extension  resolution note
MP4-lo       mp4        unknown    520.61MiB
MP4-hi       mp4        unknown    1.01GiB
MP4-HD720    mp4        unknown    2.31GiB (best)

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 16, 2019

Collaborator

This is due to Legrandin/pycryptodome#315.
Also from https://pycryptodome.readthedocs.io/en/latest/src/installation.html:

One must avoid having both PyCrypto and PyCryptodome installed at the same time, as they will interfere with each other.

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 16, 2019

Collaborator

Well, that's actually a problem. Some parts of youtube-dl use pycrypto, here pycryptodome is suggested, both won't work together when installed side by side since store files in the same lib/site-packages/Crypto location. So this brings to a problem that it's impossible to setup a configuration that will work in all cases leading to inability to build youtube-dl.exe working in all cases.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019

Author Collaborator

and how about one of these options(from the best to the worst in my opinion):

  • recommending/requiring only PyCryptodome, PyCrypto lacks some features and it hasn't been developed since 2014.
  • ask users to use PyCryptodomeX and changes the imports here to from Cryptodome.
  • including parts of PyCrypto needed here that weren't released(v2.7a1), I think its license is compatible with the youtube-dl license.
  • translating the JavaScript part used by ivi.ru.
  • write CMAC code from scratch.

and as the code from HLS Native Downloader is more important than the one here, I still don't have a problem with reverting this code until there is a better solution.

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 16, 2019

Collaborator

Does native hls works properly with pycryptodome(x)?
If we start suggesting pycryptodome then this may bring troubles for those who have pycrypto already installed.

ask users to use PyCryptodomeX and changes the imports here to from Cryptodome.

Can pycrypto and pycryptodomex be installed side by side and not interfere?

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019

Author Collaborator

Does native hls works properly with pycryptodome(x)?

yes, and I think from the beginning I was using PyCryptodome because when I installed python-crypto package from Arch repo, the package is just an alias for python-pycryptodome.

If we start suggesting pycryptodome then this may bring troubles for those who have pycrypto already installed.

for windows users, the package is included in the binary so this shouldn't be a problem.
for Unix systems, package managers deal with conflicts and won't allow installing incompatible packages.
for MacOS, I'm not sure.
for PIP users it will cause the problem you had before because both PyCrypto and PyCryptodome can be installed simultaneously.

Can pycrypto and pycryptodomex be installed side by side and not interfere?

I can do something like(with both PyCrypto and PyCryptodomeX):

from Crypto.Cipher import Blowfish
from Cryptodome.Hash import CMAC

print(CMAC.new(KEY, b'abcd', Blowfish).hexdigest())

and it runs fine.
but having both PyCrypto and PyCryptdomeX will further increase the executable size for windows.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019

Author Collaborator

I just checked MacPorts doesn't have pycrypto, you can only use pycryptodome.
and I couldn't find pycrypto or pycryptodome in Homebrew.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019

Author Collaborator

also, an additional benefit of using PyCryptodome over PyCrypto is that it supports AES-NI that helps to speedup HLS fragment decryption process(and reduce power consumption).

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019 •
edited

Author Collaborator

Package availability on some the major Linux distribution:

Distribution PyCrypto PyCryptodome PyCryptodomeX
Ubuntu x x
Debian x x
Fedora x x
ArchLinux x x
OpenSuse x x
Mageia x x x
CentOS x x x
Gentoo x x

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 19, 2019

Collaborator

I don't think having pycryptodomex along with pycrypto will increase Windows binary much. As ivi is the only place that needs this functionality then for now my vote goes to using pycrypto and pycryptodomex side by side.

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 21, 2019

Collaborator

It does not look to be possible to bundle pycryptodome(x) with py2exe producing working exe: Legrandin/pycryptodome#228
})
}
except ImportError:
data = data % 183
query = {}

video_json = self._download_json(
self._LIGHT_URL, video_id,
'Downloading video JSON', data=data, query=query)

error = video_json.get('error')
if error:
Expand All @@ -121,6 +125,8 @@ def _real_extract(self, url):
msg=error['message'], countries=self._GEO_COUNTRIES)
elif origin == 'NoRedisValidData':
raise ExtractorError('Video %s does not exist' % video_id, expected=True)
elif origin == 'NotAllowedError':

This comment has been minimized.

Sign in to view

Copy link
@dstftw

dstftw Nov 15, 2019

Collaborator

Technically this may not be caused by missing pycryptodome.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 15, 2019

Author Collaborator

i will change the condition to check for недоступен для просмотра на площадке s183 presense in error message.

This comment has been minimized.

Sign in to view

Copy link
@remitamine

remitamine Nov 16, 2019

Author Collaborator

done in 9e4e864.
raise ExtractorError('pycryptodome not found. Please install it.', expected=True)
raise ExtractorError(
'Unable to download video %s: %s' % (video_id, error['message']),
expected=True)
Expand Down

0 comments on commit 1bba88e

Please sign in to comment.