-
Notifications
You must be signed in to change notification settings - Fork 7
Improve performance with cffi #12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance with cffi #12
Conversation
Thank you for the PR!
As I'm no longer trying to actively extend this optimizer, I don't mind some additional maintenance work (which won't affect me too much).
|
I think this is possible and it would look almost the same on the Python side. I have never tried but I found this example: Calling Rust from Python/PyPy using CFFI
Thanks for your time and your comments! |
In case you are able to change, it would be neat if we could use Rust. If you don't feel experienced enough or figure it would take too long, let's leave it using C-code. Unfortunately, I'm too overwhelmed by university at the moment to delve into this issue myself. |
I'll add a duration for |
My Rust knowledge is also very basic. I'm interested in this language, but it's not at top of the list of things I want to do when I have some free time. :-) |
In that case, let's stick to the C-code for now. |
- Use mixedCase for all variables in Python (for consistency) - Prepend all ffi variables with 'ffi' - Rename the 'Bigram' struct in C to 'BigramC'
By the way, could you do a few further things: For me (=maintainability):
For users:
|
C comments added in 5e79c9a. |
I think I'm slowly running out of things to complain about ;)
cd cffi && pip install cffi && pypy3 cffi_extension_build.py ; cd .. (The last point is to return to the root, allowing for the next command to be run seamlessly.
$ pypy3 cffi_extension_build.py
generating ./_cffi_extension.c
(already up-to-date)
the current directory is '/home/florian/Downloads/8vim_keyboard_layout_calculator-improve-performance-with-cffi/cffi'
running build_ext
building '_cffi_extension' extension
gcc -pthread -DNDEBUG -O2 -fPIC -I/usr/include/pypy3.8 -c _cffi_extension.c -o ./_cffi_extension.o
_cffi_extension.c:50:14: fatal error: pyconfig.h: No such file or directory
50 | # include <pyconfig.h>
| ^~~~~~~~~~~~
compilation terminated.
Traceback (most recent call last):
File "/usr/lib/pypy3.8/distutils/unixccompiler.py", line 133, in _compile
extra_postargs)
File "/usr/lib/pypy3.8/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/pypy3.8/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/pypy3.8/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/pypy3.8/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/usr/lib/pypy3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/pypy3.8/distutils/command/build_ext.py", line 343, in run
self.build_extensions()
File "/usr/lib/pypy3.8/distutils/command/build_ext.py", line 452, in build_extensions
self._build_extensions_serial()
File "/usr/lib/pypy3.8/distutils/command/build_ext.py", line 477, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/pypy3.8/distutils/command/build_ext.py", line 537, in build_extension
depends=ext.depends)
File "/usr/lib/pypy3.8/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/usr/lib/pypy3.8/distutils/unixccompiler.py", line 135, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "cffi_extension_build.py", line 17, in <module>
ffibuilder.compile(verbose=True)
File "/usr/lib/pypy3.8/cffi/api.py", line 727, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/usr/lib/pypy3.8/cffi/recompiler.py", line 1565, in recompile
compiler_verbose, debug)
File "/usr/lib/pypy3.8/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/usr/lib/pypy3.8/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.VerificationError: CompileError: command 'gcc' failed with exit status 1 Edit:
|
My thought was that when
OK.
AFAIU,
This error message doesn't look like Which distribution do you use? You could try installing the |
Interesting, did't know about that.
You're right, it already is installed: $ gcc --version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Linux Mint (based on Ubuntu).
|
Maybe it's named Your error message looks like
and here
Maybe it's already installed but fd pyconfig.h / If you don't have fd installed, then try: find / -name pyconfig.h 2>/dev/null |
Yep, that one exists and I already had it installed.
I got this. What does it tell us? $ find / -name pyconfig.h 2>/dev/null
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/22.08/.74a00ed8035dcdea3f18a098764adfe0243a7bb5e6532caee6c8c14b2ce010a9-1H2BZ1/files/include/python3.10/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/22.08/.74a00ed8035dcdea3f18a098764adfe0243a7bb5e6532caee6c8c14b2ce010a9-1H2BZ1/files/lib/x86_64-linux-gnu/include/python3.10/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/22.08/2652112cf6f02f590602d3379a36a0e59d801a120f8ba4bb43170e75e3ab73da/files/include/python3.10/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/22.08/2652112cf6f02f590602d3379a36a0e59d801a120f8ba4bb43170e75e3ab73da/files/lib/x86_64-linux-gnu/include/python3.10/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/20.08/52034b76d4c302e58dd0c5b2b46d00b9f33a68cbe56859a264e426880e00979c/files/include/python3.8/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/20.08/52034b76d4c302e58dd0c5b2b46d00b9f33a68cbe56859a264e426880e00979c/files/lib/x86_64-linux-gnu/include/python3.8/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/21.08/.a439c06472701e00aead26fc20f474b187604a7803d942c56908743ef54e9c61-LN80X1/files/include/python3.9/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/21.08/.a439c06472701e00aead26fc20f474b187604a7803d942c56908743ef54e9c61-LN80X1/files/lib/x86_64-linux-gnu/include/python3.9/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/21.08/00ad08145692e66d6d7eff58c56299ac1e53b383877ba2c12497bc21cf16f946/files/include/python3.9/pyconfig.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/21.08/00ad08145692e66d6d7eff58c56299ac1e53b383877ba2c12497bc21cf16f946/files/lib/x86_64-linux-gnu/include/python3.9/pyconfig.h
(...Many copies in timeshift snapshots...)
/usr/include/x86_64-linux-gnu/python3.10/pyconfig.h
/usr/include/python3.10/pyconfig.h |
By the way, I just realized: Using $ cd cffi && pip install cffi && python3 cffi_extension_build.py ; cd ..
Defaulting to user installation because normal site-packages is not writeable
Collecting cffi
Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Requirement already satisfied: pycparser in /home/florian/.local/lib/python3.10/site-packages (from cffi) (2.21)
Installing collected packages: cffi
Successfully installed cffi-1.15.1
generating ./_cffi_extension.c
(already up-to-date)
the current directory is '/home/florian/Downloads/8vim_keyboard_layout_calculator-improve-performance-with-cffi/cffi'
running build_ext
building '_cffi_extension' extension
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _cffi_extension.c -o ./_cffi_extension.o
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c cffi_extension.c -o ./cffi_extension.o
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 ./_cffi_extension.o ./cffi_extension.o -o ./_cffi_extension.cpython-310-x86_64-linux-gnu.so It can then be started via The previous error only occurs when using |
OK, no matches in a |
Hey, that did the trick! :) In that case, to (probably) conclude this PR, could you...
|
Great overview! I'll update the READMEs when I'm back home. |
Great! Feel free to edit it if you feel like making any changes / have additions. |
93bcdcf
to
bf6d3f3
Compare
Fixed! |
Well then let's merge! |
Out of curiosity, I have analyzed the performance of 8vim_keyboard_layout_calculator and tested how much faster this script can be when the bottleneck functions are rewritten in C and called via
python-cffi
.This way the optimizer runs more than double as fast. See my article Speed up Python with CFFI for details.
I don't know if it makes sense merging this. Updating the C code when the Python code changes is additional maintenance work. No problem for me if you decline this pull request. At least, interested users might find this PR and apply the changes locally.