Potential method name clash with LibreSSL

[jwalch]

I am working on a project where I have to integrate both OpenSSL & OpenSSH and I have been experiencing a similar symptom to what it is described with the try-ciphers script in #9524 . I can SSH into my system if it's AESGCM, but not with AESCTR.

I've been trying to debug-by-printf (ugh) this situation from OpenSSH side, and I think I have some idea of what happened.

I added some tracing to various routines within OpenSSH. I discovered that the context was getting called with some IV, then we would encrypt / decrypt a handful of times, then OpenSSH tries to marshal the state (key, IV, etc) via a blob to an inferior process. When it does this, I observed it was sending out the original IV, and not the running IV.

In the LibreSSL compatibility layer of OpenSSH , there is an EVP_CIPHER_CTX_get_iv() routine. This overlaps with an OpenSSL routine of the same name, of course... so with the right defines / configuration it calls into the OpenSSL one instead. But looking at the LibreSSL routine is instructive...

int
EVP_CIPHER_CTX_get_iv(const EVP_CIPHER_CTX *ctx, unsigned char *iv, size_t len)
{
	if (ctx == NULL)
		return 0;
	if (EVP_CIPHER_CTX_iv_length(ctx) < 0)
		return 0;
	if (len != (size_t)EVP_CIPHER_CTX_iv_length(ctx))
		return 0;
	if (len > EVP_MAX_IV_LENGTH)
		return 0; /* sanity check; shouldn't happen */
	/*
	 * Skip the memcpy entirely when the requested IV length is zero,
	 * since the iv pointer may be NULL or invalid.
	 */
	if (len != 0) {
		if (iv == NULL)
			return 0;
# ifdef HAVE_EVP_CIPHER_CTX_IV
		memcpy(iv, EVP_CIPHER_CTX_iv(ctx), len);
# else
		memcpy(iv, ctx->iv, len);
# endif /* HAVE_EVP_CIPHER_CTX_IV */
	}
	return 1;
}

It seems, at least to me, that the above is meant to get the running IV and not the original IV.

When I modified the OpenSSH call to EVP_CIPHER_CTX_get_iv() and instead had it call EVP_CIPHER_CTX_get_iv_state(), it fixed the problem as observed in my setup.

There is some subsequent discussion in #9524 about the merits of changing
the name(s) before beta. I'll let project members move salient discussion points along those lines into this issue as needed...

[paulidale]

We will address the names of these APIs because the current names are confusing.

[slontis]

Perhaps use one function that has a parameter that indicates running or original iv.

[kroeckx]

I'm not sure that that is what we agreed on

[slontis]

What did you hear :)

[t8m]

There was no definite agreement about what to change and how but there at least is an agreement that the current API definition of EVP_CIPHER_CTX_get_iv() and EVP_CIPHER_CTX_get_iv_state() is confusing and needs adjustments.

The proposed choices were:

1. Rename EVP_CIPHER_CTX_get_iv to EVP_CIPHER_CTX_get_original_iv and EVP_CIPHER_CTX_get_iv_state to EVP_CIPHER_CTX_get_running_iv

2. Keep just EVP_CIPHER_CTX_get_iv and add a parameter to it to indicate whether the original or running iv should be returned.

In case of 2 there still will be name clash with LibreSSL but it will result in syntax error in compilation so the users can easily see that their code needs to handle LibreSSL and OpenSSL differently.

[marckleinebudde]

In https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-December/039003.html @daztucker thought about:

3. Rename EVP_CIPHER_CTX_get_iv to EVP_CIPHER_CTX_get_original_iv and EVP_CIPHER_CTX_get_iv_state to EVP_CIPHER_CTX_get_iv

[t8m]

In https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-December/039003.html @daztucker thought about:

3. Rename `EVP_CIPHER_CTX_get_iv` to `EVP_CIPHER_CTX_get_original_iv` and `EVP_CIPHER_CTX_get_iv_state` to `EVP_CIPHER_CTX_get_iv`

That was discussed but not acceptable for OTC. Please note that the objective of the change that OTC aims for is that the current APIs are confusing and not the clash with LibreSSL. This would not remove the confusion it would just make it the other way around.

[marckleinebudde]

Ok.

Proposal 1. looks good to me, as libressl can keep their EVP_CIPHER_CTX_get_iv() and the libressl compat layer in openssl can call EVP_CIPHER_CTX_get_running_iv().

[djmdjm]

I see that 3.0.0-alpha10 did not fix this problem. It would be good if it could be fixed before 3.0.0 leaves alpha, otherwise it will be more difficult for OpenSSH to support OpenSSL-3.x