OpenSSL 3.0: SSL ECDHE Kex fails when OpenSSL Engine with EC methods is set in config file

[ElMostafaIdrassi]

Bug description

I've created a sample OpenSSL EC engine : it implements EC_KEY_METHOD, EVP_PKEY_ASN1_METHOD and EVP_PKEY_METHOD for EC. The engine does not do anything fancy, it just duplicates OpenSSL's default EC_KEY_METHOD, EVP_PKEY_ASN1_METHOD and EVP_PKEY_METHOD and then sets them.

Once the engine is set in a configuration file and the config exported as OPENSSL_CONF, any attempt to perform SSL authentication using OpenSSL 3.0 with ECDHE-RSA-AES256-GCM-SHA384 set as cipher fails with:

80DB1A8B1D7F0000:error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:ssl/s3_lib.c:4748:
80DB1A8B1D7F0000:error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:ssl/statem/statem_clnt.c:2146:

N.B: ECDHE-RSA-AES256-GCM-SHA384 makes use of the Elliptic Curve P-256.

Environment

$ cat /etc/centos-release 
CentOS Stream release 9

$ openssl version -a
OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
built on: Thu Nov 24 00:00:00 2022 UTC
platform: linux-x86_64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\"" -DREDHAT_FIPS_VERSION="\"3.0.7-3bdd18e92af0bdc9\"" -DSYSTEM_CIPHERS_FILE="/etc/crypto-policies/back-ends/openssl.config"
OPENSSLDIR: "/etc/pki/tls"
ENGINESDIR: "/usr/lib64/engines-3"
MODULESDIR: "/usr/lib64/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfeda32034f8bffff:0x9c27a9

Tested commands

$ export OPENSSL_CONF=/var/tmp/testengine.conf
$ openssl s_client -cipher 'ECDHE-RSA-AES256-GCM-SHA384' -connect samsung.com:443
CONNECTED(00000003)
...
80CB9438877F0000:error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:ssl/s3_lib.c:4748:
80CB9438877F0000:error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:ssl/statem/statem_clnt.c:2146:

$ export OPENSSL_CONF=/var/tmp/testengine.conf
$ curl -v --ciphers ECDHE-RSA-AES256-GCM-SHA384 https://samsung.com:443
*   Trying 211.45.27.231:443...
* Connected to samsung.com (211.45.27.231) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ECDHE-RSA-AES256-GCM-SHA384
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
...
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, internal error (592):
* error:0A080006:SSL routines::EVP lib
* Closing connection 0
curl: (35) error:0A080006:SSL routines::EVP lib

The commands were also tested using manually build OpenSSL 3.0.7 and CURL 7.86.0 to confirm this is not a defect linked to the CentOS 9 versions. Additionally, this was also tested using CURL programmatically, and the same issue with the same error message occurs.

Note that the same commands executed in a distribution that uses OpenSSL 1.1.1 work with no issues.

Bug details

It looks like when the server supports only a NIST curve based Key Exchange algorithm (e.g. ECDHE-RSA-AES256-GCM-SHA384), the call to tls_process_ske_ecdhe fails. I was able to locate where the failure happens: here is the tree of calls, leading down to default_fixup_args returning 0 when it should not, because ctx->p2 is NULL.

tls_process_key_exchange(...)
  tls_process_ske_ecdhe(...)
    ssl_generate_param_group(...)
      1/ EVP_PKEY_CTX_new_from_name(NULL, "EC", NULL)
          int_ctx_new(...)
            e = ENGINE_get_pkey_meth_engine(id)
            e != NULL
            pmeth = ENGINE_get_pkey_meth(e, id);
            pmeth != NULL
            ret->keymgmt = keymgmt; (NULL)
            ret->engine = e; (!= NULL)
            ret->pmeth = pmeth; (!= NULL)
            ret->keytype = "id-ecPublicKey"
      2/ EVP_PKEY_CTX_set_group_name(pctx, "prime256v1")
          params[0] = 
            key = "group"
            data_type = 4
            data_size = 10
            return_size = -1
          EVP_PKEY_CTX_set_params(pctx, params)
            evp_pkey_ctx_state(ctx)
              EVP_PKEY_CTX_IS_GEN_OP(ctx) but ctx->op.keymgmt.genctx == NULL
                =>  returns EVP_PKEY_STATE_LEGACY
            evp_pkey_ctx_set_params_to_ctrl(pctx, params)
              evp_pkey_ctx_setget_params_to_ctrl(pctx, SET params)
                translation = lookup_evp_pkey_ctx_translation(&tmpl);
                  action_type = SET
                  keytype1 = 408
                  keytype2 = 0
                  optype = 6
                  ctrl_num = 4097
                  ctrl_str = "ec_paramgen_curve"
                  param_key = "group"
                  param_data_type = 4
                  fixup_args = fix_ec_paramgen_curve_nid
                ctx =
                  action_type = SET
                  p1 = 0
                  p2 = NULL
                  sz = 0
                fixup = translation->fixup_args
                ret = fixup(PRE_PARAMS_TO_CTRL, translation, &ctx);
                  fix_ec_paramgen_curve_nid()
                    ret = default_fixup_args(state, translation, ctx)
                      case PRE_PARAMS_TO_CTRL:
                        if (state == PRE_PARAMS_TO_CTRL && ctx->action_type == SET)
                          switch (translation->param_data_type)
                            case OSSL_PARAM_UTF8_STRING:
                              OSSL_PARAM_get_utf8_string(ctx->params, ctx->p2 = NULL, ctx->sz = 0);
                                get_string_internal(ctx->params, ctx->p2, &ctx->sz, NULL, OSSL_PARAM_UTF8_STRING)
                                  if ((val == NULL && used_len == NULL) || p == NULL || p->data_type != type)
                                    return 0;
                                    !! FAILURE !!

[t8m]

Yeah, the PRE_PARAMS_TO_CTRL case in default_fixup_args() does not seem to be implemented correctly.

@levitte might want to look

[levitte]

The surprising thing with this is that we do test EVP_PKEY_CTX_set_group_name(), among others by pretty thorough ssl test programs, but also explicitly here:

openssl/test/evp_extra_test.c

Line 1917 in c809334

|| !TEST_int_gt(EVP_PKEY_CTX_set_group_name(pctx, "P-256"), 0)

So, what I wonder is what's being done differently in this case

[levitte]

.... ah! We haven't really tested much using legacy methods, have we? Everything we try has moved to providers, so the param→ctrl translations doesn't get as exercised as they probably should...

[ulrichb]

The fix for this issue was released in OpenSSL v3.0.9+ (backport commit ff56f28) resp. 3.1.1+ (backport commit d1b2c9b).

Unfortunately Ubuntu LTS 22.04 is on OpenSSL 3.0.2 and RedHat EL 9 is on OpenSSL 3.0.7.

@ElMostafaIdrassi Did you find any workaround for this issue which could be used on distros without having OpenSSL 3.0.9+?

[omedirk]

I'm having the same issue. The fix is nice, but it's not available for us.
we compiled 3.0.12 but got into other issues there.
(i am coming from issue : EST issue on iotedge

hope somebody has a clue how to work around this..