Building 'Crypto.Math._montgomery' extension fails on Solaris.

[dumol]

Hi,

And thank you for PyCryptodome! I am looking into using it to replace PyCrypto in this open-source project: https://github.com/chevah/python-package.

Unfortunately, building Crypto.Math._montgomery extension fails on Solaris with:

building 'Crypto.Math._montgomery' extension
cc -DNDEBUG -O -m64 -Kpic -DPYCRYPTO_LITTLE_ENDIAN -DLTC_NO_ASM -Isrc/ -I/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/include/python2.7 -c src/montgomery.c -o build/temp.solaris-2.11-i86pc.64bit-2.7/src/montgomery.o
"src/multiply.h", line 54: syntax error before or at: t
"src/multiply.h", line 54: warning: undefined or missing type for: t
"src/multiply.h", line 54: warning: undefined or missing type for: const
"src/multiply.h", line 54: warning: undefined or missing type for: uint64_t
"src/multiply.h", line 54: warning: undefined or missing type for: uint64_t
"src/multiply.h", line 54: warning: undefined or missing type for: size_t
cc: acomp failed for src/montgomery.c
error: command 'cc' failed with exit status 2

The above was from the 5.12 Sun C compiler bundled with Solaris Studio 12.3 on Solaris 11.2 X86 while building 64bit binaries, but the build fails similarly for Solaris 11.2 SPARC (while building 32bit binaries) and latest Solaris 10 (when building 64bit X86 binaries and 32bit SPARC binaries).

On a related note, some more warnings common to all these variations:

Testing support for 128-bit integer
Target does not support 128-bit integer
Testing support for intrin.h header
Target does not support intrin.h header
Testing support for cpuid.h header
Target does not support cpuid.h header
Warning: compiler does not support AESNI instructions
Warning: compiler does not support CLMUL instructions

"src/raw_cbc.c", line 44: warning: zero or negative subscript

"src/raw_ofb.c", line 50: warning: zero or negative subscript

"src/scrypt.c", line 83: warning: operands have incompatible pointer types: op "!="

And a warning specific to Solaris 10 builds:

"src/common.h", line 121: warning: implicit function declaration: posix_memalign

[Legrandin]

Thanks for reporting! This stuff is all easily fixable (the first error is probably due to a failure in determining the right way to define a "restrict"ed pointer; there is also a pending error to a missing posix_memalign() function).

Unfortunately, I don't have access to a Solaris machine, so it's difficult to write the right patch. Would you mind writing one? I may find a VM but I can't guarantee how long it will take to set it up...

[dumol]

Thank you for the quick feedback!

One addition: perhaps the last warning is not specific to Solaris 10. I see now on Solaris 10 (both arches) I've tried to build this with Solaris Studio 12.4, a version newer than the one on Solaris 11.

I suggest you try a Solaris 11 VM template from Oracle, such as http://www.oracle.com/technetwork/server-storage/solaris11/downloads/vm-templates-2245495.html (this is for Solaris 11.3). There are Solaris 10 VM templates as well, but Solaris 10 comes by default with OpenSSL 0.9.7d and you'll need a support contract to get patches for OpenSSL 1.0.2.

Developer Studio is available at http://www.oracle.com/technetwork/server-storage/developerstudio/downloads/index.html. For Solaris 11 you'll need an Oracle developer account to access the Oracle Developer Studio IPS repository at http://pkg-register.oracle.com. After getting the necessary certs, the commands to run would be something along the lines of:

sudo pkg set-publisher -k ~/pkg.oracle.com.key.pem -c ~/pkg.oracle.com.certificate.pem -G "*" -g https://pkg.oracle.com/solarisstudio/release solarisstudio
pkg list -a pkg://solarisstudio/*
sudo pkg install pkg://solarisstudio/developer/solarisstudio-124/cc

If interested, I'll try to get you access to our Buildbot infrastructure. We have a wide range of OS'es and pycryptodome seems to also have minor issues on SLES 11 (SP4 with or without the Security Module) and HP-UX. Otherwise, a very fine job building out-of-the box on AIX, OS X / macOS, various BSDs, Linux distros and Windows… Congratulations!

[Legrandin]

The code in master should at least compile on Solaris 11 now (though I noticed that support for AES HW acceleration would require quite a few more tweaks).

[dumol]

Thank you! I have commited above change to our PyCryptodome 3.6.3 sources and building is fine now on all our Solaris 10 and 11 slaves on both X86 and SPARC.

PyCryptodome self tests pass on Solaris 11, with these messages:

Skipping AESNI tests
Skipping test of PCLMULDQD in AES GCM

On a related note, they are quite slow on SPARC:

Ran 23331 tests in 1774.807s

However, on Solaris 10 (both X86 and SPARC), PyCryptodome's self tests bail out with:

Traceback (most recent call last):
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/SelfTest/__main__.py", line 35, in <module>
    SelfTest.run(stream=sys.stdout, verbosity=1, config=config)
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/SelfTest/__init__.py", line 61, in run
    tests = get_tests(config=config)
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/SelfTest/__init__.py", line 82, in get_tests
    from Crypto.SelfTest import Cipher; tests += Cipher.get_tests(config=config)
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/SelfTest/Cipher/__init__.py", line 31, in get_tests
    from Crypto.SelfTest.Cipher import test_AES;      tests += test_AES.get_tests(config=config)
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/SelfTest/Cipher/test_AES.py", line 29, in <module>
    from Crypto.Cipher import AES
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/Cipher/__init__.py", line 31, in <module>
    from Crypto.Cipher._mode_ctr import _create_ctr_cipher
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/Cipher/_mode_ctr.py", line 55, in <module>
    int CTR_stop_operation(void *ctrState);"""
  File "/export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/Util/_raw_api.py", line 258, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Crypto.Cipher._raw_ctr': Trying '_raw_ctr.so': ld.so.1: python: fatal: relocation error: file /export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/Util/../Cipher/_raw_ctr.so: symbol posix_memalign: referenced symbol not found, Trying '_raw_ctrmodule.so': ld.so.1: python: fatal: /export/home/buildslave/runtime/build-solaris10-x64/slave/python-package-solaris-10/build/build/python2.7-solaris10-x64/lib/python2.7/site-packages/pycryptodome-3.6.3-py2.7-solaris-2.10-i86pc.64bit.egg/Crypto/Util/../Cipher/_raw_ctrmodule.so: open failed: No such file or directory

Might this be related to the specific warning I mentioned above for Solaris 10 with Sun Studio 10.4? As far as I can tell, this appears for every .so file built:

"src/common.h", line 121: warning: implicit function declaration: posix_memalign

[Legrandin]

posix_memalign ends up being an undefined symbol in the .so files, so I can understand why it wouldn't load. I have just added on master another patch to fall back to memalign, which should be available in SunOS (I don't have the chance to try Solaris 10).

To speed up a test suite, you can use the --skip-slow-tests flag (like python setup.py --skip-slow-tests). One big reason for the slow-down is that Sun C does not seem to support 128-bit integers...

[dumol]

Have switched to using master, with revision cfa757d have now succeeded in completing both build and test phase on Solaris 10 SPARC. Great job!

However, latest changes have broken the X86 builds on both Solaris 10 and 11 with similar errors:

building 'Crypto.Hash._MD2' extension
creating build/temp.solaris-2.11-i86pc.64bit-2.7
creating build/temp.solaris-2.11-i86pc.64bit-2.7/src
cc -DNDEBUG -O -m64 -Kpic -DPYCRYPTO_LITTLE_ENDIAN -DLTC_NO_ASM -Isrc/ -I/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/include/python2.7 -c src/MD2.c -o build/temp.solaris-2.11-i86pc.64bit-2.7/src/MD2.o
"src/common.h", line 161: #error: No routines for aligned memory
cc: acomp failed for src/MD2.c
error: command 'cc' failed with exit status 2

[Legrandin]

I am afraid can't reproduce that in the Solaris 11 X86 VM I downloaded.
It compiles fine there, though it's the platform Python 32-bits, whereas you usea custom 64-bits version.

[dumol]

It should be OK, I'll try building 32bit binaries on Solaris X86 then. I'll let you know of the results. Thanks!

If interested in reproducing the issue with building a 64bit Python on Solaris, you should try our Python package at https://github.com/chevah/python-package. Just checkout the 4985-pycryptodome-instead-of-pycrypto branch and issue a ./chevah build. It should reproduce on Solaris 11.

[Legrandin]

Indeed I can reproduce it, and it is related to Python being 64 bits.

Specifically, and maybe this is actually a bug of distutils on 64 bit Solaris, I see that distutils.ccompiler.new_compiler() returns an inconsistent compiler object:

• method compiler() passes flags -m64 -xcode=pic32 -Kpic to cc, so all .o objects are 64-bit
• method link_executable() does pass add any flag to cc, and cc fails with error wrong ELF class: ELFCLASS64 - I guess it expects 32-bits objects

As result, all tests for capabilities (like for posix_memalign) fail, and PyCryptodome has no way to know how to safely do memory alignment.

[Legrandin]

There is a mini patch in the fix_solaris_64bit branch that forces linking to use whatever CFLAGS were used with CC in Unix (which could make sense). It seems to fix this problem; I'll merge after I see what CI has to say.

[dumol]

I see that the fix_solaris_64bit branch has been merged in master. Have updated our copy with this change and I'm very pleasantly surprised to see that PyCryptodome builds on all our Solaris 10/11 X86/SPARC slave for both 32bit and 64bit binaries and the whole suite of self tests passes everywhere. This is a fantastic job, thank you!

I will continue with several other minor issues separately, but you might want to look into fixing the warnings below, as they are present in a Solaris 11 VM as well. Of course, on Solaris they are minor, but they are fatal on HP-UX and that particular environment is much harder to replicate (Itanium servers). I will copy them below for your convenience:

building 'Crypto.Cipher._raw_cbc' extension
cc -DNDEBUG -O -m64 -Kpic -DPYCRYPTO_LITTLE_ENDIAN -DLTC_NO_ASM -DHAVE_POSIX_MEMALIGN -Isrc/ -I/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/include/python2.7 -c src/raw_cbc.c -o build/temp.solaris-2.11-i86pc.64bit-2.7/src/raw_cbc.o
"src/raw_cbc.c", line 44: warning: zero or negative subscript
cc -G -L/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/lib/ -m64 -L/usr/lib/64 -R/usr/lib/64 -L/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/lib/ -m64 -L/usr/lib/64 -R/usr/lib/64 -m64 build/temp.solaris-2.11-i86pc.64bit-2.7/src/raw_cbc.o -o build/lib.solaris-2.11-i86pc.64bit-2.7/Crypto/Cipher/_raw_cbc.so

building 'Crypto.Cipher._raw_ofb' extension
cc -DNDEBUG -O -m64 -Kpic -DPYCRYPTO_LITTLE_ENDIAN -DLTC_NO_ASM -DHAVE_POSIX_MEMALIGN -Isrc/ -I/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/include/python2.7 -c src/raw_ofb.c -o build/temp.solaris-2.11-i86pc.64bit-2.7/src/raw_ofb.o
"src/raw_ofb.c", line 50: warning: zero or negative subscript
cc -G -L/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/lib/ -m64 -L/usr/lib/64 -R/usr/lib/64 -L/srv/buildslave/runtime/build-solaris11-x64/slave/python-package-solaris-11/build/build/python2.7-solaris11-x64/lib/ -m64 -L/usr/lib/64 -R/usr/lib/64 -m64 build/temp.solaris-2.11-i86pc.64bit-2.7/src/raw_ofb.o -o build/lib.solaris-2.11-i86pc.64bit-2.7/Crypto/Cipher/_raw_ofb.so

For the reference, here's how the fatal error on HP-UX looks (the compiler is HP C/aC++ B3910B A.06.27):

/opt/aCC/bin/cc -w +DD32 +z -g -DNDEBUG -O +z -DPYCRYPTO_BIG_ENDIAN -DLTC_NO_ASM -DHAVE_MEMALIGN -Isrc/ -I/srv/buildslave/runtime/build-hpux1131-ia64/slave/python-package-hpux-dev/build/build/python2.7-hpux1131-ia64/include/python2.7 -c src/raw_cbc.c -o build/temp.hp-ux-B.11.31-ia64-2.7/src/raw_cbc.o
"src/raw_cbc.c", line 44: error #2094: the size of an array must be greater than zero
      uint8_t iv[0];
                 ^

1 error detected in the compilation of "src/raw_cbc.c".
error: command '/opt/aCC/bin/cc' failed with exit status 2

Thank you once more!

[dumol]

I can confirm revision 61abaac fixes the build on HP-UX as well. I couldn't ran the tests because of a segmentation fault in CFFI though (both latest stable version and current master). Will try to contact CFFI upstream, hopefully there will be a way to solve this.

If only upstream maintainers would be half as attentive as you are for half of the issues I report… Thank you again, you've been the best maintainer I've ever worked with!

[frispete]

If Helder would get a beer for every problem, he solved, he would probably getting into the danger of alcoholism ;-)