New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation on Visual Studio/MSVC 9 #44

Closed
hynek opened this Issue Dec 9, 2015 · 10 comments

Comments

Projects
None yet
2 participants
@hynek
Contributor

hynek commented Dec 9, 2015

Hi,

context: I’ve built friendly CFFI-base Python wrappers that work over all Python version and implementations (PyPy…). I also vendor Argon2 and want to offer binary packages (called wheels) for Windows.

This may become a story of multiple tickets I’m afraid since I’m not too skilled WRT Windows. Anyhow but the first problem I’ve run into is that Argon2 currently can’t be compiled using MSVC 9.0 (which I believe is VS 2008) which unfortunately is the one you have to use for Python 2.7 extensions (Microsoft even put it online just for that reason: Microsoft Visual C++ Compiler for Python 2.7
)

The first problem I ran into is that you’re including inttypes.h which VS 2008 lacks. I was able to work around that one by dropping http://msinttypes.googlecode.com/svn/trunk/stdint.h into the repo and change the includes.

As such, I was able to compile most of the project:

building 'libargon2' library
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/argon2.c /Fobuild\temp.win32-2.7\libargon2
/src/argon2.obj
argon2.c
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/blake2/blake2b.c /Fobuild\temp.win32-2.7\l
ibargon2/src/blake2/blake2b.obj
blake2b.c
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/core.c /Fobuild\temp.win32-2.7\libargon2/s
rc/core.obj
core.c
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/encoding.c /Fobuild\temp.win32-2.7\libargo
n2/src/encoding.obj
encoding.c
libargon2/src/encoding.c(407) : warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprint
f_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdio.h(36
6) : see declaration of 'sprintf'
libargon2/src/encoding.c(409) : warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprint
f_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdio.h(36
6) : see declaration of 'sprintf'
libargon2/src/encoding.c(411) : warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprint
f_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdio.h(36
6) : see declaration of 'sprintf'
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/opt.c /Fobuild\temp.win32-2.7\libargon2/sr
c/opt.obj
opt.c
libargon2/src/opt.c(18) : fatal error C1083: Cannot open include file: 'immintrin.h': No such file or directory
error: command 'C:\\Users\\Hynek Schlawack\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\
\Bin\\cl.exe' failed with exit status 2

And I really don’t know how to fix that one…


To be clear: this means it’s currently impossible to build Python 2 (still 90+% of the landscape) packages on Windows.


Steps to reproduce (tested on Windows 10):

  1. Install Microsoft Visual C++ Compiler for Python 2.7
  2. Install Python 2.7.11
  3. git clone https://github.com/hynek/argon2_cffi and cd into it.
  4. git submodules update to pull in your library.
  5. Install virtualenv: C:\Python27\Scripts\pip.exe install virtualenv
  6. create a virtualenv: C:\Python27\Scripts\virtualenv.exe venv
  7. try to build a wheel using that virtualenv: .\venv\Scripts\python.exe .\setup.py bdist_wheel

Behold the fail.


I’m sorry. :(

I really hope we can get this working on all platforms as I’d like to establish Argon2 in the Python community. But that impossible if it doesn’t work on 2.7/Windows…

And tomorrow I’ll try to get it working on Python 3. :-/

@sneves

This comment has been minimized.

Contributor

sneves commented Dec 9, 2015

Thanks for the very good bug report. Luckily, this seems straightforward to fix, though it's annoying that Python is still sticking to such an old Visual Studio compiler.

While a solution is not committed, you can try to fix the immintrin.h situation by replacing it with intrin.h. Alternatively, you can only build the reference code (omit opt.c and include ref.c) as a stopgap measure.

@hynek

This comment has been minimized.

Contributor

hynek commented Dec 9, 2015

intrin.h indeed fixes that problem!

New problem:

libargon2/src/opt.c(40) : error C2275: '__m128i' : illegal use of this type as an expression
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\emmintrin.
h(45) : see declaration of '__m128i'
libargon2/src/opt.c(40) : error C2146: syntax error : missing ';' before identifier 't1'
libargon2/src/opt.c(40) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(40) : error C2440: '=' : cannot convert from '__m128i' to 'int'
libargon2/src/opt.c(40) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(40) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(40) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 1
libargon2/src/opt.c(40) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(40) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(40) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 2
libargon2/src/opt.c(46) : error C2275: '__m128i' : illegal use of this type as an expression
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\emmintrin.
h(45) : see declaration of '__m128i'
libargon2/src/opt.c(46) : error C2146: syntax error : missing ';' before identifier 't1'
libargon2/src/opt.c(46) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(46) : error C2440: '=' : cannot convert from '__m128i' to 'int'
libargon2/src/opt.c(46) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(46) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(46) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 1
libargon2/src/opt.c(46) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(46) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(46) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 2
error: command 'C:\\Users\\Hynek Schlawack\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\
\Bin\\cl.exe' failed with exit status 2

Would you like a new issue or are we going just to roll with it? :)

P.S. The reason for the annoying old compiler is that Python 2.7 is from 2010 and we need binary compatibility…

@sneves

This comment has been minimized.

Contributor

sneves commented Dec 9, 2015

OK, intrin.h on Visual Studio 2008 does not seem to include all the other intrinsics headers like it does now. So we need to include each header individually. Namely:

#include <intrin.h>
#include <emmintrin.h>
#include <tmmintrin.h>

You may also need to make similar adjustments in src/blake2/blamka-round-opt.h.

@hynek

This comment has been minimized.

Contributor

hynek commented Dec 9, 2015

Hm I’ve added all three includes into both src/blake2/blamka-round-opt.h and opt.c (which should be unnecessary since opt.c is including it…) and I’m still getting:

opt.c
libargon2/src/opt.c(42) : error C2275: '__m128i' : illegal use of this type as an expression
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\emmintrin.
h(45) : see declaration of '__m128i'
libargon2/src/opt.c(42) : error C2146: syntax error : missing ';' before identifier 't1'
libargon2/src/opt.c(42) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(42) : error C2440: '=' : cannot convert from '__m128i' to 'int'
libargon2/src/opt.c(42) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(42) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(42) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 1
libargon2/src/opt.c(42) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(42) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(42) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 2
libargon2/src/opt.c(48) : error C2275: '__m128i' : illegal use of this type as an expression
        C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\emmintrin.
h(45) : see declaration of '__m128i'
libargon2/src/opt.c(48) : error C2146: syntax error : missing ';' before identifier 't1'
libargon2/src/opt.c(48) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(48) : error C2440: '=' : cannot convert from '__m128i' to 'int'
libargon2/src/opt.c(48) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(48) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(48) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 1
libargon2/src/opt.c(48) : error C2065: 't1' : undeclared identifier
libargon2/src/opt.c(48) : error C2440: 'function' : cannot convert from 'int' to '__m128i'
libargon2/src/opt.c(48) : warning C4024: '_mm_unpacklo_epi64' : different types for formal and actual parameter 2
error: command 'C:\\Users\\Hynek Schlawack\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\
\Bin\\cl.exe' failed with exit status 2

(of course the line numbers are woefully off now due to my tampering.

Line 42 would be:

        BLAKE2_ROUND(state[8 * i + 0], state[8 * i + 1], state[8 * i + 2],
                     state[8 * i + 3], state[8 * i + 4], state[8 * i + 5],
                     state[8 * i + 6], state[8 * i + 7]);

(the last line of the statement to be precise)

@sneves

This comment has been minimized.

Contributor

sneves commented Dec 9, 2015

Hmm, I think you may need to add /arch:SSE2 to the compiler flags, but this is just a guess. I'll setup VS 2008 later to investigate this.

@sneves

This comment has been minimized.

Contributor

sneves commented Dec 9, 2015

It was actually something much simpler---a C99-style declaration was still existing in the BLAKE2_ROUND macros. Fixed in latest commit.

@hynek

This comment has been minimized.

Contributor

hynek commented Dec 10, 2015

I could only do a quick check, but the inttypes.h and stdint.h problems are still there, right?

@hynek

This comment has been minimized.

Contributor

hynek commented Dec 10, 2015

OK got to play some more. Let me start with good news: Python 3.5 works. :)

To sum up, currently the following things break for Python 2.7/Windows/VS 2008:

  • #include <intrin.h> and #include <immintrin.h> in blamka-round-opt.h (replace by the three you mentioned fixes this)
  • #include <immintrin.h> in opt.c (commented out completely since it includes blamka-round-opt.h
  • #include <stdint.h> all over the place (dropped in from http://msinttypes.googlecode.com/)
  • #include <inttypes.h> core.c (dropped in from http://msinttypes.googlecode.com/)
  • which brings me to my current blocker:
C:\Users\Hynek Schlawack\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -Ilibargon2/src -Ilibargon2/src/blake2 /Tclibargon2/src/opt.c /Fobuild\temp.win32-2.7\libargon2/sr
c/opt.obj
opt.c
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2143: syntax error : missing ')' before '*'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2143: syntax error : missing '{' before '*'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2371: 'uint8_t' : redefinition; different basic ty
pes
        libargon2/src\stdint.h(90) : see declaration of 'uint8_t'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : warning C4228: nonstandard extension used : qualifiers af
ter comma in declarator list are ignored
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2143: syntax error : missing ';' before '*'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2371: 'uint8_t' : redefinition; different basic ty
pes
        libargon2/src\stdint.h(90) : see declaration of 'uint8_t'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2143: syntax error : missing ';' before '*'
c:\users\hynek schlawack\argon2_cffi\libargon2\src\opt.h(24) : error C2059: syntax error : ')'
error: command 'C:\\Users\\Hynek Schlawack\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\
\Bin\\cl.exe' failed with exit status 2

And the line it objects is the following:

void fill_block(__m128i *state, const uint8_t *ref_block, uint8_t *next_block);

Sorry for being completely useless here. Windows is wayyyy out of my comfort zone. :(

@sneves

This comment has been minimized.

Contributor

sneves commented Dec 10, 2015

I think the last commit should compile fine everywhere now. Let me know if anything else comes up.

I have not yet included stdint.h and inttypes.h, and not sure I will. This is fixable at the compiler switch level, by pointing it to the alternative versions. For example, downloading both of those files from http://msinttypes.googlecode.com to src\extra we can compile genkat.c as

cl /nologo /Ox /W4 /D_CRT_SECURE_NO_WARNINGS /Isrc\extra src\genkat.c src\argon2.c src\encoding.c src\thread.c src\core.c src\opt.c src\blake2\blake2b.c 
@hynek

This comment has been minimized.

Contributor

hynek commented Dec 10, 2015

Yeah I don’t think you actually can include them since they’re “New BSD” and you want to stay CC0 I presume.

Anyhow, I’ve done what you suggested and I’m happy to report great success: https://jenkins.cryptography.io/job/argon2_cffi-wheel-builder/TOXENV=py27,label=windows/12/console

Thanks for your assistance, it’s very appreciated!

@hynek hynek closed this Dec 10, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment