Add Cython and Password Support #21

Theelx · 2020-10-13T12:14:53Z

This combines the password and Cython PRs into one.

dobrosketchkun · 2020-10-13T13:14:27Z

@Theelgirl you forget about README.md:

Requires installation of FFmpeg and libmagic first, then install using pip3

But one needs a ffmpeg executable for this to work, anyway, so you should explicitly say it in there, may be?

dobrosketchkun · 2020-10-13T13:28:11Z

So, I tried patch-1 on Win8 machine.

It seems it compiled ok:

C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test\fvid-patch-1>py setup.py install


Compiling fvid/fvid_cython.pyx because it changed.
[1/1] Cythonizing fvid/fvid_cython.pyx
running install
running build
running build_ext
building 'fvid_cython' extension
creating build
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
creating build\temp.win32-3.7\Release\fvid
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo /Ox
 /W3 /GL /DNDEBUG /MD -I./fvid -Ifvid/ -IC:\Users\user\AppData\Local\Progr
ams\Python\Python37-32\include -IC:\Users\user\AppData\Local\Programs\Pyth
on\Python37-32\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC
\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "
-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x
86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\incl
ude\winrt" /EHsc /Tpfvid/fvid_cython.cpp /Fobuild\temp.win32-3.7\Release\fvid/fv
id_cython.obj
fvid_cython.cpp
creating C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test\fvid-patch-
1\build\lib.win32-3.7
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\link.exe /nologo /INC
REMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Be
lozersky\AppData\Local\Programs\Python\Python37-32\libs /LIBPATH:C:\Users\Beloze
rsky\AppData\Local\Programs\Python\Python37-32\PCbuild\win32 "/LIBPATH:C:\Progra
m Files (x86)\Microsoft Visual Studio 14.0\VC\LIB" "/LIBPATH:C:\Program Files (x
86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\
Windows Kits\8.1\lib\winv6.3\um\x86" /EXPORT:PyInit_fvid_cython build\temp.win32
-3.7\Release\fvid/fvid_cython.obj /OUT:build\lib.win32-3.7\fvid_cython.cp37-win3
2.pyd /IMPLIB:build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.lib
   Creating library build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.lib
 and object build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.exp
Generating code
Finished generating code
running install_lib
copying build\lib.win32-3.7\fvid_cython.cp37-win32.pyd -> C:\Users\user\Ap
pData\Local\Programs\Python\Python37-32\Lib\site-packages
running install_egg_info
Writing C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib\site-p
ackages\UNKNOWN-0.0.0-py3.7.egg-info
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist
.py:274: UserWarning: Unknown distribution option: 'long_description_content_typ
e'
  warnings.warn(msg)
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist
.py:274: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist
.py:274: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist
.py:274: UserWarning: Unknown distribution option: 'entry_points'
  warnings.warn(msg)
running install
running build
running build_py
creating build\lib
creating build\lib\fvid
copying fvid\cythonizer.py -> build\lib\fvid
copying fvid\fvid.py -> build\lib\fvid
copying fvid\__init__.py -> build\lib\fvid
copying fvid\__main__.py -> build\lib\fvid
running install_lib
creating C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib\site-
packages\fvid
copying build\lib\fvid\cythonizer.py -> C:\Users\user\AppData\Local\Progra
ms\Python\Python37-32\Lib\site-packages\fvid
copying build\lib\fvid\fvid.py -> C:\Users\user\AppData\Local\Programs\Pyt
hon\Python37-32\Lib\site-packages\fvid
copying build\lib\fvid\__init__.py -> C:\Users\user\AppData\Local\Programs
\Python\Python37-32\Lib\site-packages\fvid
copying build\lib\fvid\__main__.py -> C:\Users\user\AppData\Local\Programs
\Python\Python37-32\Lib\site-packages\fvid
byte-compiling C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib
\site-packages\fvid\cythonizer.py to cythonizer.cpython-37.pyc
byte-compiling C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib
\site-packages\fvid\fvid.py to fvid.cpython-37.pyc
byte-compiling C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib
\site-packages\fvid\__init__.py to __init__.cpython-37.pyc
byte-compiling C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib
\site-packages\fvid\__main__.py to __main__.cpython-37.pyc
running install_egg_info
Writing C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib\site-p
ackages\fvid-0.0.2-py3.7.egg-info

But when I tried to use it:

C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test>py -m fvid -i Lenna.png -e -o Lenna.mp4
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.
py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.
py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-p
ackages\fvid\__main__.py", line 2, in <module>
    from fvid import main
ImportError: cannot import name 'main' from 'fvid' (C:\Users\user\AppData\
Local\Programs\Python\Python37-32\lib\site-packages\fvid\__init__.py)

dobrosketchkun · 2020-10-13T13:30:08Z

Also, it seems I cannot uninstall it:

C:\Users\Belozersky\Documents\Khromov\python\fvid\mine\test>py -m pip uninstall fvid

Found existing installation: fvid 0.0.2
ERROR: Cannot uninstall 'fvid'. It is a distutils installed project and thus we
cannot accurately determine which files belong to it which would lead to only a
partial uninstall.

Theelx · 2020-10-13T13:32:56Z

Ok, I'll try and find a way to install cython without distutils. However, the "ImportError: cannot import name 'main' from 'fvid'" shouldn't be coming from cython, cython doesn't change anything there. Also, try compiling the cython directly with py cythonizer.py build_ext --inplace and tell me how that works on windows.

dobrosketchkun · 2020-10-13T13:37:49Z

Here is a log:

C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test\fvid-patch-1\fvid>py
 cythonizer.py build_ext --inplace
running build_ext
building 'fvid_cython' extension
creating build
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo /Ox
 /W3 /GL /DNDEBUG /MD -IC:\Users\user\AppData\Local\Programs\Python\Python
37-32\include -IC:\Users\user\AppData\Local\Programs\Python\Python37-32\in
clude "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\P
rogram Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program File
s (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\
8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /EHsc
/Tpfvid_cython.cpp /Fobuild\temp.win32-3.7\Release\fvid_cython.obj
fvid_cython.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\link.exe /nologo /INC
REMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Be
lozersky\AppData\Local\Programs\Python\Python37-32\libs /LIBPATH:C:\Users\Beloze
rsky\AppData\Local\Programs\Python\Python37-32\PCbuild\win32 "/LIBPATH:C:\Progra
m Files (x86)\Microsoft Visual Studio 14.0\VC\LIB" "/LIBPATH:C:\Program Files (x
86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\
Windows Kits\8.1\lib\winv6.3\um\x86" /EXPORT:PyInit_fvid_cython build\temp.win32
-3.7\Release\fvid_cython.obj /OUT:C:\Users\user\Documents\dobrosketchkun\python\f
vid\mine\test\fvid-patch-1\fvid\fvid_cython.cp37-win32.pyd /IMPLIB:build\temp.wi
n32-3.7\Release\fvid_cython.cp37-win32.lib
   Creating library build\temp.win32-3.7\Release\fvid_cython.cp37-win32.lib and
object build\temp.win32-3.7\Release\fvid_cython.cp37-win32.exp
Generating code
Finished generating code

Archive with built files:

build.zip

Theelx · 2020-10-13T13:41:34Z

That log looks like it worked. Try building this new version from scratch now and let me know how it works.

Edit: Windows is the root of all evil.

dobrosketchkun · 2020-10-13T14:04:59Z

Edit: Windows is the root of all evil.

lol

You forget about a comma before zip_safe=False

The log:

C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test\fvid-patch-1>py setup.py install


WARNING: The wheel package is not available.
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\
setuptools\dist.py:679: UserWarning: Module Cython was already imported from C:\
Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\Cyt
hon\__init__.py, but c:\users\belozersky\documents\khromov\python\fvid\mine\test
\fvid-patch-1\.eggs\cython-3.0a6-py3.7-win32.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\
setuptools\dist.py:679: UserWarning: Module cython was already imported from C:\
Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cyt
hon.py, but c:\users\belozersky\documents\khromov\python\fvid\mine\test\fvid-pat
ch-1\.eggs\cython-3.0a6-py3.7-win32.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
running install
running bdist_egg
running egg_info
creating fvid.egg-info
writing fvid.egg-info\PKG-INFO
writing dependency_links to fvid.egg-info\dependency_links.txt
writing entry points to fvid.egg-info\entry_points.txt
writing requirements to fvid.egg-info\requires.txt
writing top-level names to fvid.egg-info\top_level.txt
writing manifest file 'fvid.egg-info\SOURCES.txt'
reading manifest file 'fvid.egg-info\SOURCES.txt'
writing manifest file 'fvid.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_py
creating build
creating build\lib.win32-3.7
creating build\lib.win32-3.7\fvid
copying fvid\cythonizer.py -> build\lib.win32-3.7\fvid
copying fvid\fvid.py -> build\lib.win32-3.7\fvid
copying fvid\__init__.py -> build\lib.win32-3.7\fvid
copying fvid\__main__.py -> build\lib.win32-3.7\fvid
copying fvid\fvid_cython.cpp -> build\lib.win32-3.7\fvid
running build_ext
building 'fvid.fvid_cython' extension
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
creating build\temp.win32-3.7\Release\fvid
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo /Ox
 /W3 /GL /DNDEBUG /MD -I./fvid -Ifvid/ -IC:\Users\user\AppData\Local\Progr
ams\Python\Python37-32\include -IC:\Users\user\AppData\Local\Programs\Pyth
on\Python37-32\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC
\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "
-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x
86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\incl
ude\winrt" /EHsc /Tpfvid/fvid_cython.cpp /Fobuild\temp.win32-3.7\Release\fvid/fv
id_cython.obj
fvid_cython.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\link.exe /nologo /INC
REMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Be
lozersky\AppData\Local\Programs\Python\Python37-32\libs /LIBPATH:C:\Users\Beloze
rsky\AppData\Local\Programs\Python\Python37-32\PCbuild\win32 "/LIBPATH:C:\Progra
m Files (x86)\Microsoft Visual Studio 14.0\VC\LIB" "/LIBPATH:C:\Program Files (x
86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\
Windows Kits\8.1\lib\winv6.3\um\x86" /EXPORT:PyInit_fvid_cython build\temp.win32
-3.7\Release\fvid/fvid_cython.obj /OUT:build\lib.win32-3.7\fvid\fvid_cython.cp37
-win32.pyd /IMPLIB:build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.lib
   Creating library build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.lib
 and object build\temp.win32-3.7\Release\fvid\fvid_cython.cp37-win32.exp
Generating code
Finished generating code
creating build\bdist.win32
creating build\bdist.win32\egg
creating build\bdist.win32\egg\fvid
copying build\lib.win32-3.7\fvid\cythonizer.py -> build\bdist.win32\egg\fvid
copying build\lib.win32-3.7\fvid\fvid.py -> build\bdist.win32\egg\fvid
copying build\lib.win32-3.7\fvid\fvid_cython.cp37-win32.pyd -> build\bdist.win32
\egg\fvid
copying build\lib.win32-3.7\fvid\fvid_cython.cpp -> build\bdist.win32\egg\fvid
copying build\lib.win32-3.7\fvid\__init__.py -> build\bdist.win32\egg\fvid
copying build\lib.win32-3.7\fvid\__main__.py -> build\bdist.win32\egg\fvid
byte-compiling build\bdist.win32\egg\fvid\cythonizer.py to cythonizer.cpython-37
.pyc
byte-compiling build\bdist.win32\egg\fvid\fvid.py to fvid.cpython-37.pyc
byte-compiling build\bdist.win32\egg\fvid\__init__.py to __init__.cpython-37.pyc

byte-compiling build\bdist.win32\egg\fvid\__main__.py to __main__.cpython-37.pyc

creating stub loader for fvid\fvid_cython.cp37-win32.pyd
byte-compiling build\bdist.win32\egg\fvid\fvid_cython.py to fvid_cython.cpython-
37.pyc
creating build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\PKG-INFO -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\SOURCES.txt -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\dependency_links.txt -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\entry_points.txt -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\not-zip-safe -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\requires.txt -> build\bdist.win32\egg\EGG-INFO
copying fvid.egg-info\top_level.txt -> build\bdist.win32\egg\EGG-INFO
writing build\bdist.win32\egg\EGG-INFO\native_libs.txt
creating dist
creating 'dist\fvid-0.0.2-py3.7-win32.egg' and adding 'build\bdist.win32\egg' to
 it
removing 'build\bdist.win32\egg' (and everything under it)
Processing fvid-0.0.2-py3.7-win32.egg
creating c:\users\belozersky\appdata\local\programs\python\python37-32\lib\site-
packages\fvid-0.0.2-py3.7-win32.egg
Extracting fvid-0.0.2-py3.7-win32.egg to c:\users\belozersky\appdata\local\progr
ams\python\python37-32\lib\site-packages
Adding fvid 0.0.2 to easy-install.pth file
Installing fvid-script.py script to C:\Users\user\AppData\Local\Programs\P
ython\Python37-32\Scripts
Installing fvid.exe script to C:\Users\user\AppData\Local\Programs\Python\
Python37-32\Scripts
Installing fvid.exe.manifest script to C:\Users\user\AppData\Local\Program
s\Python\Python37-32\Scripts

Installed c:\users\belozersky\appdata\local\programs\python\python37-32\lib\site
-packages\fvid-0.0.2-py3.7-win32.egg
Processing dependencies for fvid==0.0.2
Searching for tqdm==4.28.1
Best match: tqdm 4.28.1
Adding tqdm 4.28.1 to easy-install.pth file
Installing tqdm-script.py script to C:\Users\user\AppData\Local\Programs\P
ython\Python37-32\Scripts
Installing tqdm.exe script to C:\Users\user\AppData\Local\Programs\Python\
Python37-32\Scripts
Installing tqdm.exe.manifest script to C:\Users\user\AppData\Local\Program
s\Python\Python37-32\Scripts

Using c:\users\belozersky\appdata\local\programs\python\python37-32\lib\site-pac
kages
Searching for Pillow==5.3.0
Best match: Pillow 5.3.0
Adding Pillow 5.3.0 to easy-install.pth file

Using c:\users\belozersky\appdata\local\programs\python\python37-32\lib\site-pac
kages
Searching for bitstring==3.1.7
Best match: bitstring 3.1.7
Adding bitstring 3.1.7 to easy-install.pth file

Using c:\users\belozersky\appdata\local\programs\python\python37-32\lib\site-pac
kages
Finished processing dependencies for fvid==0.0.2

I don't work either:

C:\Users\user\Documents\dobrosketchkun\python\fvid\mine\test>py -m fvid -i Lenna.
png -e -o Lenna.mp4
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.
py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.
py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-p
ackages\fvid\__main__.py", line 2, in <module>
    from fvid import main
ImportError: cannot import name 'main' from 'fvid' (C:\Users\user\AppData\
Local\Programs\Python\Python37-32\lib\site-packages\fvid\__init__.py)

New build files in the directory:

fvid-patch-1.zip

Theelx · 2020-10-13T14:11:14Z

Try changing main.py to

import sys
from fvid.fvid import main

if __name__ == '__main__':
	sys.exit(main())

Edit: No, that's dumb don't do that.

Theelx · 2020-10-13T14:18:30Z

@dobrosketchkun Try now? I figure it could have been a conflict with the package name and file name on Windows, as it works fine on Ubuntu.

dobrosketchkun · 2020-10-13T14:18:59Z

Edit: No, that's dumb don't do that.

Well, actually, I just went to another machine with Win10 and tried as it is and it's doesn't work, so I change main.py in C:\Users\user\AppData\Local\Programs\Python\Python37-32\Lib\site-packages\fvid-0.0.2-py3.7-win32.egg\fvid folder, and it works

dobrosketchkun · 2020-10-13T14:19:39Z

Should I try your new commits?

Theelx · 2020-10-13T14:21:38Z

Try the new commits, because the one I told you to do doesn't work on my Ubuntu machine, this might work on both systems.

dobrosketchkun · 2020-10-13T14:24:52Z

New commits:

first of all, again, a comma before zip_safe
And:

>py -m fvid -i 1.jpg -e -o 1.mp4
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\fvid-0.0.2-py3.7-win32.egg\fvid\__main__.py", line 2, in <module>
    from fvid_python import main
ModuleNotFoundError: No module named 'fvid_python'

doesn't work on my Ubuntu machine

maybe we just should check with the system with os module and import what we need in each case?

Theelx · 2020-10-13T14:26:02Z

Did you rename fvid.py to fvid_python.py? I'll try and add detection of the os in main.py and adjust accordingly if this doesn't work.

dobrosketchkun · 2020-10-13T14:32:44Z

Did you rename fvid.py to fvid_python.py?

Ehm, no I did not. I installed it as it is.

Anyway, the result is the same:

ModuleNotFoundError: No module named 'fvid_python'

Theelx · 2020-10-13T14:33:13Z

Ok, I'll update main.py with detection of os now.

Theelx · 2020-10-13T14:34:25Z

@dobrosketchkun Try now on windows, it may need to be tested on MacOS also (@AlfredoSequeida)

Theelx · 2020-10-16T18:13:37Z

@dobrosketchkun I just tried running this on 3.8.6, and I installed pycrypto, but I'm getting this error:

  File "fvid.py", line 309, in main
    bits = get_bits_from_file(args.input, key)
  File "fvid.py", line 63, in get_bits_from_file
    cipher = AES.new(key, AES.MODE_EAX, nonce=SALT)
AttributeError: module 'Crypto.Cipher.AES' has no attribute 'MODE_EAX'

I dont know why it suddenly stopped working, but I checked the github and it hasn't been updated since 2014. Shouldn't we be using a newer module that's actively updated for security and maintainability and such?

Edit: I realized pycryptodome will work drop-in instead, so I'll add that to setup.py

dobrosketchkun · 2020-10-24T14:55:35Z

@AlfredoSequeida any news?

AlfredoSequeida · 2020-10-24T20:12:48Z

@dobrosketchkun sorry, I'm really busy lately with school work.

Theelx · 2020-10-24T21:37:42Z

@AlfredoSequeida Do you think you can give one of us pushing and releasing permissions?

dobrosketchkun · 2020-10-24T21:57:30Z

For good for all of us, it better be @Theelgirl because I know very little about pulls, pushes, and branches.

Theelx · 2020-10-24T21:59:27Z

Ah, I'm really bad with branches but I can manage pulls and pushes if needed. However, this would also need to be released to PyPi on important updates, so if he wants to hedge any minute risk of one of us going rogue, he could let me manage PRs/pushes, and you manage PyPi releases.

AlfredoSequeida · 2020-10-25T00:43:21Z

@Theelgirl Yes! I've been pondering this for a while because I see you guys are so active. And I was actually going to ask you about it. Let me look into how to get that done and I would be glad to see what you do with it.

Theelx · 2020-10-25T00:49:35Z

Here's a useful link for PyPi:
https://pypi.org/help/#collaborator-roles
For github, to add a collaborator, you can click Settings (for the repo), then Manage Access, then "Invite Collaborator":
https://github.com/AlfredoSequeida/fvid/settings/access

AlfredoSequeida · 2020-10-25T00:56:10Z

@Theelgirl I sent you a request to add you to the project. If you wish to accept it I would be happy to have you manage it until I have more time to work on it. And I can definitely try to fit some time in when we need to update the PyPi package. Let me know what you think. Also thank you for taking interest in the project, I'm glad you like it enough to contribute the amount you have so far.

Theelx · 2020-10-25T01:00:47Z

Thanks for the invitation! I actually found this project off a Reddit link from the Python Discord, and I just happened to be using Pillow and Cython extensively, so I figured I'd drop in and help make a few optimizations, but I didn't initially expect to end up contributing as much as I have. However, even though I didn't initially plan for it, I'd love to help continue development and optimize this project, so I accepted the invitation!

Removing a python call to all(), which forced conversion of a ctuple to a python tuple, and removing white_diff and black_diff ctuple creation because I can fit all of their use cases into a single if statement.

Theelx · 2020-10-25T02:20:59Z

This most recent Cython approach is a whopping 70x faster on a large 26MB jpg of mine than the pure python approach! It takes about 32-33 seconds per frame (each frame is about 3.2 MB, and there are 105 frames) with Python, and about 0.5 seconds per frame with Cython!
~~(running Python 3.8.6 on a 3.8GHz Ryzen processor on Ubuntu 20.04)~~

~~I remember saying a while back that I expected something like a 5x speedup at most, but 70x (even though only with large files) is beyond my wildest dreams!~~

Edit: I did a dumb and measured the pure python performance with my line profiler on, and Cython without it on, Cython actually only gives a 8-9x boost compared to the current python version, and a 3-4x boost compared to a speedier python fvid that I'm about to push :(

More than doubled the python decoding speed for get_bits_from_image, and sped up ffmpeg about 33-50% by tuning for zerolatency.

Theelx · 2020-10-25T19:14:08Z

@dobrosketchkun Can you test my most recent changes to fvid on Windows and/or OS X? These work on Ubuntu 20.04, and if they work on both of those platforms, I can merge it.

dobrosketchkun · 2020-10-25T20:11:05Z

Yeah, it's working quite well and fast.

But.

I tried to check, for once, -f option and used 60. File encoded well, but upon decoding, the error appeared:

Bits are in place
100%|██████████████████████████████████████████████████████████████████████████████████| 52/52 [02:10<00:00,  2.51s/it]
Unzipping...
Traceback (most recent call last):
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\__main__.py", line 28, in <module>
    sys.exit(main())
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\fvid.py", line 288, in main
    save_bits_to_file(file_path, bits, key)
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\fvid.py", line 162, in save_bits_to_file
    bitstring = fo.read()
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\gzip.py", line 300, in read
    return self._buffer.read(size)
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\gzip.py", line 495, in read
    uncompress = self._decompressor.decompress(buf, size)
zlib.error: Error -3 while decompressing data: invalid code lengths set

Then I tried 30, and there was another error:

Traceback (most recent call last):
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\__main__.py", line 28, in <module>
    sys.exit(main())
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\fvid.py", line 288, in main
    save_bits_to_file(file_path, bits, key)
  File "D:\recovery3\python\random\fvid_env\lib\site-packages\fvid\fvid.py", line 162, in save_bits_to_file
    bitstring = fo.read()
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\gzip.py", line 300, in read
    return self._buffer.read(size)
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\gzip.py", line 478, in read
    self._read_eof()
  File "C:\Users\dobrosketchkun\AppData\Local\Programs\Python\Python39\lib\gzip.py", line 524, in _read_eof
    raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32),
gzip.BadGzipFile: CRC check failed 0x1447e231 != 0xaa41731d

The same was for -f 24

This is the same for Win and Arch.

dobrosketchkun · 2020-10-25T20:42:18Z

I tried on one file, and it is working from default up to -f 7 in my case.

Maybe the deal is about codec and its quality settings?

dobrosketchkun · 2020-10-25T21:19:33Z

Btw without the -tune zerolatency flag, it's working up to 30

Theelx · 2020-10-25T22:52:38Z

Hm ok, so I'll remove the tune flag and squash merge it. Thanks for testing!

Password commits make it incompatible with 0.0.2 and 0.0.1.

Add Cython and Password Support

Theelx added 4 commits October 13, 2020 12:14

Add Cython and Password Support

a4ec504

This combines the password and Cython PRs into one.

Create fvid_cython.pyx

c426f23

Update setup.py

5edd263

Create cythonizer.py

024543c

This was referenced Oct 13, 2020

passwords and glob ffmpeg in win #15

Closed

Add Cython Support #17

Closed

Theelx added 2 commits October 13, 2020 13:39

Update setup.py

7f5e584

Create fvid_cython.cpp

e076abd

Theelx added 2 commits October 13, 2020 14:17

Rename fvid.py to fvid_python.py

1c2d681

Update __main__.py

af170de

Update setup.py

a24b157

Theelx added 2 commits October 13, 2020 14:33

Update __main__.py

2453d6b

Rename fvid_python.py to fvid.py

b429f2c

Theelx added 4 commits October 16, 2020 18:17

Add pycryptodome

6bde757

Remove unneeded imports

ba3475f

Speedup by 40-45%

53be012

Update and rename fvid_cython.cpp to fvid_cython.c

c27314b

Speedup 35-40% over the previous speedup

29374fe

Removing a python call to all(), which forced conversion of a ctuple to a python tuple, and removing white_diff and black_diff ctuple creation because I can fit all of their use cases into a single if statement.

Theelx added 3 commits October 25, 2020 18:04

Type more stuff for a 2% speedup

5ccd42c

Update fvid_cython.c

1385b0a

Massive Python and ffmpeg speedups

0e2f668

More than doubled the python decoding speed for get_bits_from_image, and sped up ffmpeg about 33-50% by tuning for zerolatency.

Theelx added 2 commits October 25, 2020 22:53

Revert broken tuning for ffmpeg

8753269

Update version to 1.0.0

bd658aa

Password commits make it incompatible with 0.0.2 and 0.0.1.

Theelx merged commit 6881618 into AlfredoSequeida:master Oct 25, 2020

Theelx deleted the patch-1 branch October 25, 2020 22:55

dobrosketchkun mentioned this pull request Oct 28, 2020

cannot import name 'main' from 'fvid' #27

Closed

Theelx added a commit that referenced this pull request Mar 9, 2021

Merge pull request #21 from Theelgirl/patch-1

94868e4

Add Cython and Password Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Cython and Password Support #21

Add Cython and Password Support #21

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020 •

edited

Loading

dobrosketchkun commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020 •

edited

Loading

dobrosketchkun commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020 •

edited

Loading

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

Theelx commented Oct 13, 2020

Theelx commented Oct 16, 2020 •

edited

Loading

dobrosketchkun commented Oct 24, 2020

AlfredoSequeida commented Oct 24, 2020 •

edited

Loading

Theelx commented Oct 24, 2020

dobrosketchkun commented Oct 24, 2020

Theelx commented Oct 24, 2020

AlfredoSequeida commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020 •

edited

Loading

AlfredoSequeida commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020

Theelx commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

Theelx commented Oct 25, 2020 •

edited

Loading

Add Cython and Password Support #21

Add Cython and Password Support #21

Conversation

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020 • edited Loading

dobrosketchkun commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020 • edited Loading

dobrosketchkun commented Oct 13, 2020 • edited Loading

Theelx commented Oct 13, 2020 • edited Loading

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020 • edited Loading

dobrosketchkun commented Oct 13, 2020

Theelx commented Oct 13, 2020

Theelx commented Oct 13, 2020

Theelx commented Oct 16, 2020 • edited Loading

dobrosketchkun commented Oct 24, 2020

AlfredoSequeida commented Oct 24, 2020 • edited Loading

Theelx commented Oct 24, 2020

dobrosketchkun commented Oct 24, 2020

Theelx commented Oct 24, 2020

AlfredoSequeida commented Oct 25, 2020 • edited Loading

Theelx commented Oct 25, 2020 • edited Loading

AlfredoSequeida commented Oct 25, 2020 • edited Loading

Theelx commented Oct 25, 2020

Theelx commented Oct 25, 2020 • edited Loading

Theelx commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

dobrosketchkun commented Oct 25, 2020

Theelx commented Oct 25, 2020 • edited Loading

dobrosketchkun commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 13, 2020 •

edited

Loading

dobrosketchkun commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 13, 2020 •

edited

Loading

Theelx commented Oct 16, 2020 •

edited

Loading

AlfredoSequeida commented Oct 24, 2020 •

edited

Loading

AlfredoSequeida commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020 •

edited

Loading

AlfredoSequeida commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020 •

edited

Loading

Theelx commented Oct 25, 2020 •

edited

Loading