New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evp_cipher_ctx_st definition not installing #962

Closed
aressler38 opened this Issue Apr 12, 2016 · 11 comments

Comments

Projects
None yet
5 participants
@aressler38

aressler38 commented Apr 12, 2016

Hi, I'm getting an error when compiling a simple app after compiling openssl through the git repo.

error: invalid use of incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’

I'm looking at my install directory, /usr/local/include/openssl, and I can see that the EVP_CIPHER_CTX is typedef'd in /usr/local/include/openssl/ossl_typ.h, but the definition of the struct is nowhere to be found in the install directory.

When I examine the git repo, I can see the definition in openssl/crypto/evp/evp_locl.h, but I don't see the crypto directory installed into /usr/local/include anywhere.

My application will work fine when I install through my package manager, but I'd like to be able to use the latest version, so I'm trying to install from the repo (master branch 1.1.0-pre5-dev).

I've tried

./config
make
make test
sudo make install

./Configure linux-x86_64
make 
make test
sudo make install

./config shared

...

Nothing has worked. All the tests are successful. I'd really appreciate any insights.

I'm using Debian GNU/Linux 8 and gcc 4.9.2

For what it's worth, I downloaded openssl-SNAP-20160412, ran ./config, and I still got the same error when compiling my app (openssl compiled and installed fine).

Here's the output of the ./config command

Operating system: x86_64-whatever-linux2
Configuring for linux-x86_64
Configuring OpenSSL version 1.1.0-pre5-dev (0x0x10100005L)
no-crypto-mdebug [default] OPENSSL_NO_CRYPTO_MDEBUG (skip dir)
no-crypto-mdebug-backtrace [forced] OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE (skip dir)
no-dynamic-engine [forced]
no-ec_nistp_64_gcc_128 [default] OPENSSL_NO_EC_NISTP_64_GCC_128 (skip dir)
no-egd [default] OPENSSL_NO_EGD (skip dir)
no-heartbeats [default] OPENSSL_NO_HEARTBEATS (skip dir)
no-md2 [default] OPENSSL_NO_MD2 (skip dir)
no-rc5 [default] OPENSSL_NO_RC5 (skip dir)
no-sctp [default] OPENSSL_NO_SCTP (skip dir)
no-shared [default]
no-ssl-trace [default] OPENSSL_NO_SSL_TRACE (skip dir)
no-ssl3 [default] OPENSSL_NO_SSL3 (skip dir)
no-ssl3-method [default] OPENSSL_NO_SSL3_METHOD (skip dir)
no-unit-test [default] OPENSSL_NO_UNIT_TEST (skip dir)
no-weak-ssl-ciphers [default] OPENSSL_NO_WEAK_SSL_CIPHERS (skip dir)
no-zlib [default]
no-zlib-dynamic [default]
Configuring for linux-x86_64
CC =gcc
CFLAG =-Wall -O3 -pthread -m64 -DL_ENDIAN -Wa,--noexecstack
SHARED_CFLAG =-fPIC
DEFINES =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_DYNAMIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM POLY1305_ASM
LFLAG =
PLIB_LFLAG =
EX_LIBS =-ldl
APPS_OBJ =
CPUID_OBJ =x86_64cpuid.o
UPLINK_OBJ =
BN_ASM =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o
EC_ASM =ecp_nistz256.o ecp_nistz256-x86_64.o
DES_ENC =des_enc.o fcrypt_b.o
AES_ENC =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o
BF_ENC =bf_enc.o
CAST_ENC =c_enc.o
RC4_ENC =rc4-x86_64.o rc4-md5-x86_64.o
RC5_ENC =rc5_enc.o
MD5_OBJ_ASM =md5-x86_64.o
SHA1_OBJ_ASM =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o
RMD160_OBJ_ASM=
CMLL_ENC =cmll-x86_64.o cmll_misc.o
MODES_OBJ =ghash-x86_64.o aesni-gcm-x86_64.o
PADLOCK_OBJ =e_padlock-x86_64.o
CHACHA_ENC =chacha-x86_64.o
POLY1305_OBJ =poly1305-x86_64.o
BLAKE2_OBJ =
PROCESSOR =
RANLIB =/usr/bin/ranlib
ARFLAGS =
PERL =/usr/bin/perl

SIXTY_FOUR_BIT_LONG mode

Configured for linux-x86_64.

@mattcaswell

This comment has been minimized.

Member

mattcaswell commented Apr 12, 2016

One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 version is that many types have been made opaque, i.e. applications are no longer allowed to look inside the internals of the structures. The biggest impact on applications is that:

  1. You cannot instantiate these structures directly on the stack. So instead of:
EVP_CIPHER_CTX ctx;

you must instead do:

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
....
EVP_CIPHER_CTX_free(ctx);
  1. You must only use the provided accessor functions to access the internals of the structure.
@aressler38

This comment has been minimized.

aressler38 commented Apr 12, 2016

Thanks @mattcaswell

S4mw1s3 pushed a commit to S4mw1s3/encrypted-session-nginx-module that referenced this issue Dec 23, 2016

Sam Van Den Berge
bugfix: don't initiate ctx directly on the stack
One of the primary differences between master (OpenSSL 1.1.0) and the
1.0.2 version is that many types have been made opaque, i.e.
applications are no longer allowed to look inside the internals of the
structures. The biggest impact on applications is that:

1) You cannot instantiate these structures directly on the stack. So
instead of:

EVP_CIPHER_CTX ctx;

you must instead do:

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
....
EVP_CIPHER_CTX_free(ctx);

2) You must only use the provided accessor functions to access the
internals of the structure.

Source:
openssl/openssl#962 (comment)

Signed-off-by: Sam Van Den Berge <sam.van.den.berge@telenet.be>

S4mw1s3 pushed a commit to S4mw1s3/encrypted-session-nginx-module that referenced this issue Dec 30, 2016

Sam Van Den Berge
bugfix: don't initiate ctx directly on the stack
One of the primary differences between master (OpenSSL 1.1.0) and the
1.0.2 version is that many types have been made opaque, i.e.
applications are no longer allowed to look inside the internals of the
structures. The biggest impact on applications is that:

1) You cannot instantiate these structures directly on the stack. So
instead of:

EVP_CIPHER_CTX ctx;

you must instead do:

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
....
EVP_CIPHER_CTX_free(ctx);

2) You must only use the provided accessor functions to access the
internals of the structure.

Source:
openssl/openssl#962 (comment)

Signed-off-by: Sam Van Den Berge <sam.van.den.berge@telenet.be>

joseriosneto added a commit to joseriosneto/qupzilla that referenced this issue May 1, 2017

Fixed compilation for Openssl 1.1.0
Most of libcrypto and libssl internal structures were made
opaque in this version not allowing to instatiate them in
the stack.

More info:
    * https://www.openssl.org/news/openssl-1.1.0-notes.html
    * openssl/openssl#962 (comment)

nowrep added a commit to QupZilla/qupzilla that referenced this issue May 1, 2017

Fixed compilation for Openssl 1.1.0
Most of libcrypto and libssl internal structures were made
opaque in this version not allowing to instatiate them in
the stack.

More info:
    * https://www.openssl.org/news/openssl-1.1.0-notes.html
    * openssl/openssl#962 (comment)

a17r added a commit to a17r/gentoo that referenced this issue May 7, 2017

www-client/qupzilla: Fix build with >=dev-libs/openssl-1.1.0
See also:
openssl/openssl#962 (comment)

Package-Manager: Portage-2.3.5, Repoman-2.3.1

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue May 7, 2017

www-client/qupzilla: Fix build with >=dev-libs/openssl-1.1.0
See also:
openssl/openssl#962 (comment)

Closes: #4560

Package-Manager: Portage-2.3.5, Repoman-2.3.1

@hirenvadalia hirenvadalia referenced this issue Jul 13, 2017

Merged

PP-856: Implement AppVeyor CI for Windows #332

3 of 9 tasks complete

brunoaduarte added a commit to brunoaduarte/modwifi-tools that referenced this issue May 9, 2018

kusumi added a commit to kusumi/DragonFlyBSD that referenced this issue Jul 8, 2018

lib/libdmsg: Unbreak using new API EVP_CIPHER_CTX_new()
The upstream OpenSSL no longer publicly expose definition of
EVP_CIPHER_CTX (struct evp_cipher_ctx_st).

Due to this change clients need to have it as a pointer instead
of as a value, and allocate or free EVP_CIPHER_CTX instance by
EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free().

openssl/openssl#962 (comment)

Above APIs are available in our OpenSSL too, so we should move
on to use these, otherwise upgrading OpenSSL will break libdmsg
compilation at some point in the future. This also makes it more
portable against systems using newer version of OpenSSL.

Note that this diff is missing EVP_CIPHER_CTX_free() (was not
sure where it should be freed at).
@Dravion

This comment has been minimized.

Dravion commented Oct 8, 2018

So, all existing Software must be rewritten just to be able to use 1.1.x ?
Do you have any idea how many Million lines of code are affected by this crap??

@mattcaswell

This comment has been minimized.

Member

mattcaswell commented Oct 8, 2018

So, all existing Software must be rewritten just to be able to use 1.1.x ?
Do you have any idea how many Million lines of code are affected by this crap??

Yes, applications must be changed to use 1.1.x. This was not a change we made lightly or without an understanding of the impact on applications. It was much discussed both internally within the project and also in public forums over a long period. Largely speaking the user community has been very supportive of this change - although of course there are always people on both sides of the fence.

There are a number of problems with non-opaque structures:

  1. The internal members of structures become part of the API, which means it is impossible for us to make any changes to any of them. This in turn makes refactoring, changing, or even just bug fixing any OpenSSL code very difficult - meaning that the code tends to ossify over time.
  2. Changing the size or order of members in a structure has implications for ABI compatibility further restricting our ability to maintain the code
  3. It is does not create a clear separation between internal implementation details and the application interface - meaning that applications tend to rely on specific implementation details. This makes applications brittle and further restricts the ability of the OpenSSL team to maintain the code
  4. In order to deal with the above problems the OpenSSL maintainers were forced into tortuous contortions in the code in order to implement workarounds.

Making this change was necessary for us to have a healthy OpenSSL moving forwards. Without it, it would have been impossible for us to make major improvements to the code such as the state machine refactor. Implementing TLSv1.3 (as in 1.1.1) would have been extremely difficult (maybe impossible) without making this change. While it does require applications to make changes the vast majority of those changes are quite straightforward and simple to do. Very many applications have already done so.

@Dravion

This comment has been minimized.

Dravion commented Oct 8, 2018

Thats no excuse. I just downloaded the latest version of LibreSSL 2.8.1 which was forked as result of the Heartbleed security meltdown in OpenSSL and to my surprise, it works perfectly without changing 1 Line of code of our codebase. Sorry OpenSSL Team, but you messed it up again big time.

@mattcaswell

This comment has been minimized.

Member

mattcaswell commented Oct 9, 2018

Sorry you feel that way. It's worth noting that OpenSSL 1.0.2 continues to support (until 31st December 2019), non-opaque structures.

@cbarrett69

This comment has been minimized.

cbarrett69 commented Oct 10, 2018

Doesn't this also mean that the context pointer can no longer be wrapped in a smart pointer? I currently have

{
std::unique_ptr<EVP_CIPHER_CTX> ctx(new EVP_CIPHER_CTX());
/* stuff that can throw an exception */
}

@mattcaswell

This comment has been minimized.

Member

mattcaswell commented Oct 11, 2018

Doesn't this also mean that the context pointer can no longer be wrapped in a smart pointer?

It can still be done. We do this in the ossl_shim test code (borrowed from BoringSSL), for example see:

bssl::UniquePtr<SSL> ssl(SSL_new(ssl_ctx));

Where UniquePtr is defined like this:

#define BORINGSSL_MAKE_DELETER(type, deleter) \
namespace internal { \
template <> \
struct DeleterImpl<type> { \
static void Free(type *ptr) { deleter(ptr); } \
}; \
}
// This makes a unique_ptr to STACK_OF(type) that owns all elements on the
// stack, i.e. it uses sk_pop_free() to clean up.
#define BORINGSSL_MAKE_STACK_DELETER(type, deleter) \
namespace internal { \
template <> \
struct DeleterImpl<STACK_OF(type)> { \
static void Free(STACK_OF(type) *ptr) { \
sk_##type##_pop_free(ptr, deleter); \
} \
}; \
}
// Holds ownership of heap-allocated BoringSSL structures. Sample usage:
// bssl::UniquePtr<BIO> rsa(RSA_new());
// bssl::UniquePtr<BIO> bio(BIO_new(BIO_s_mem()));
template <typename T>
using UniquePtr = std::unique_ptr<T, internal::Deleter<T>>;
BORINGSSL_MAKE_DELETER(BIO, BIO_free)
BORINGSSL_MAKE_DELETER(EVP_PKEY, EVP_PKEY_free)
BORINGSSL_MAKE_DELETER(DH, DH_free)
BORINGSSL_MAKE_DELETER(X509, X509_free)
BORINGSSL_MAKE_DELETER(SSL, SSL_free)
BORINGSSL_MAKE_DELETER(SSL_CTX, SSL_CTX_free)
BORINGSSL_MAKE_DELETER(SSL_SESSION, SSL_SESSION_free)

@richsalz

This comment has been minimized.

Contributor

richsalz commented Oct 11, 2018

In other words, you can't just delete a pointer to an OpenSSL object, you have to call the specific free/cleanup function. This requirement hasn't changed because the pointers were made to opaque structures.

@cbarrett69

This comment has been minimized.

cbarrett69 commented Oct 11, 2018

So, wouldn't a simple implementation for cipher context (or anything else with the new/free design) just be
{
std::unique_ptr<EVP_CIPHER_CTX, void(*)(EVP_CIPHER_CTX *)> ctx(EVP_CIPHER_CTX_new(), EVP_CIPHER_CTX_free);
}

Seems to work, just a little unsure because it wasn't suggested. Maybe it will help someone who comes across this in a search.

@richsalz

This comment has been minimized.

Contributor

richsalz commented Oct 11, 2018

Yes that should work. My point is that this kind of thing was always required. (Except now, yeah, you can't have stack-local variables.) Add a reference using borrow, and most of the existing code might not need a change. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment