Compile error 7.81.0 with OpenSSL 1.0.2u (VC14, Winbuild, multissl)

[Jan-E]

@bagder
When I try to compile Curl 7.81.0 with both OpenSSL 1.0.2u and Schannel on Windows using VC14 I run into a compile error. This is my command line:

nmake /f Makefile.vc mode=static VC=14 MACHINE=x86 WITH_DEVEL=../../win32build.vc14 USE_IDN=yes WITH_SSL=dll ENABLE_SCHANNEL=yes ENABLE_NGHTTP2=yes WITH_SSH2=static WITH_ZLIB=dll GEN_PDB=yes

And this is the error:

        cl /O2 /DNDEBUG /MD /DCURL_STATICLIB /I. /I ../lib /I../include /nologo /W4 /EHsc /DWIN32 /FD /c /DBUILDING_LIBCURL /I"../../win32build.vc14/include" /DUSE_OPENSSL /I"../../win32build.vc14/include\openssl" /DHAVE_ZLIB_H /DHAVE_ZLIB /DHAVE_LIBZ  /I"../../win32build.vc14/include" /DHAVE_LIBSSH2 /DHAVE_LIBSSH2_H /DLIBSSH2_WIN32 /DLIBSSH2_LIBRARY /DUSE_LIBSSH2 /I../../win32build.vc14/include/libssh2  /DUSE_WIN32_IDN /DWANT_IDN_PROTOTYPES  /DUSE_IPV6  /DUSE_WINDOWS_SSPI /DUSE_SCHANNEL /DUSE_NGHTTP2 /I"../../win32build.vc14/include" /Zi /Fd"..\builds\libcurl-vc14-x86-release-static-ssl-dll-zlib-dll-ssh2-static-ipv6-sspi-schannel-nghttp2-dll-obj-lib\libcurl_a.pdb" /DCURL_WITH_MULTI_SSL /Fo"..\builds\libcurl-vc14-x86-release-static-ssl-dll-zlib-dll-ssh2-static-ipv6-sspi-schannel-nghttp2-dll-obj-lib/smb.obj"  ..\lib\smb.c
smb.c
N:\php-sdk\win32build.vc14\include\openssl/pem.h(159): error C2059: syntax error: '('
N:\php-sdk\win32build.vc14\include\openssl/pem.h(163): error C2059: syntax error: '}'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(181): error C2061: syntax error: identifier 'PEM_USER'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(184): error C2143: syntax error: missing '{' before '*'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(184): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
N:\php-sdk\win32build.vc14\include\openssl/pem.h(184): warning C4218: nonstandard extension used: must specify at least a storage class or a type
N:\php-sdk\win32build.vc14\include\openssl/pem.h(204): error C2059: syntax error: '}'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2059: syntax error: '<parameter-list>'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2143: syntax error: missing ')' before '('
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2059: syntax error: ')'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2143: syntax error: missing ')' before 'constant'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2091: function returns function
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2143: syntax error: missing '{' before 'constant'
N:\php-sdk\win32build.vc14\include\openssl/pem.h(454): error C2059: syntax error: 'constant'

Lines 157-163 of pem.h are these

typedef struct pem_recip_st {
    char *name;
    X509_NAME *dn;
    int cipher;
    int key_enc;
    /*      char iv[8]; unused and wrong size */
} PEM_USER;

The error occurs on the line with X509_NAME.
The error is related to this commit: c148f0f

2 possible solutions:

1. set ENABLE_SCHANNEL=no (but I do use the Schannel in libcurl_a.lib)
2. revert the part of the commit that caused the error (kludgy):

diff -r -u "lib/vtls/openssl.h" "lib/vtls/openssl.h"
--- "lib/vtls/openssl.h"
+++ "lib/vtls/openssl.h"
@@ -26,15 +26,11 @@
 
 #ifdef USE_OPENSSL
 /*
- * This header should only be needed to get included by vtls.c, openssl.c
- * and ngtcp2.c
+ * This header should only be needed to get included by vtls.c and openssl.c
  */
 
-#include <openssl/x509v3.h>
 #include "urldata.h"
 
-CURLcode Curl_ossl_verifyhost(struct Curl_easy *data, struct connectdata *conn,
-                              X509 *server_cert);
 extern const struct Curl_ssl Curl_ssl_openssl;
 
 #endif /* USE_OPENSSL */

I know this really is an edge case (and I can work around it), but I want to report it anyway.

[bagder]

Reverting that will instead break the build when ngtcp2 is used.

[Jan-E]

Looking at c148f0f (briefly), would a solution with #ifdef ENABLE_QUIC in lib/vtls/openssl.c and lib/vtls/openssl.h be a way out?

[bagder]

It wouldn't solve it, since you could build curl with quic support and still support OpenSSL + Schannel for TLS.

[bagder]

I'm trying a different take as you can see in #8246

[Jan-E]

I will test it. BTW: it did not error when building with OpenSSL 1.1.1m, even if I stiil use VC14 for a Windows version of PHP 7.1 with an upgraded OpenSSl.

[Jan-E]

Is ENABLE_QUIC missing in the winbuild way of building curl or just undocumented in https://github.com/curl/curl/blob/master/winbuild/README.md ?

[bagder]

Both I guess. QUIC/HTTP/3 is still an experimental feature and is not enabled by default anywhere.

[Jan-E]

Tested #8246 with VC14, x86. Building fine with either OpenSSL 1.0.2u or OpenSSL 1.1.1m. Will test further with VC9, VC11, VC14, VC15, VS16, x86 and x64. Looking OK.

[Jan-E]

Your affort (pun intended) was successful. I have rebuilt 12 different curl versions without compile errors. Of course, I cannot test QUIC support, because it is not supported in winbuild.

[bagder]

The ngtcp2 CI builds turned green as well so let's merge this.