Encryption failure during handshake when using libssh2 and wolfssl

[brucsc]

I'm getting a -44 error code and message of "Unable to ask for ssh-userauth service" when using wolfssl as the encryption engine during ssh handshake. I've tracked the error to this code on line 584 of openssl.c:

if(IS_LAST(firstlast)) {
  /* This is the last block.
     encrypt: compute tag, if applicable
     decrypt: verify tag, if applicable
     in!=NULL is equivalent to EVP_CipherUpdate
     in==NULL is equivalent to EVP_CipherFinal */
#ifdef HAVE_OPAQUE_STRUCTS
  ret = EVP_Cipher(*ctx, NULL, NULL, 0); /* final */
#else
  ret = EVP_Cipher(ctx, NULL, NULL, 0); /* final */
#endif

The comment indicates the expectation that EVP_CipherFinal should be equivalent to EVP_Cipher when finalizing the encryption. The EVP_Cipher method resolves to wolfSSL_EVP_Cipher on line 7080 of the wolfssl/wolfcrypt/src/evp.c file. There a check of NULL on the incoming buffers returns a WOLFSSL_FATAL_ERROR. After modifying the code to follow the comment, EVP_CipherFinal is used instead of EVP_Cipher, then the encryption seems to work as expected:

int final_length = 0;
ret = EVP_CipherFinal(*ctx, buf + aadlen, &final_length); /* final */

I do not know enough about these internals to know if this is the correct solution or not and would appreciate any assistance in getting this fixed. The comment suggests a break in the interface expectations with wolfssl.

To Reproduce
Extract libssh2 1.11.0, wolfssl 5.6.4. Build wolfssl. Build libssh2 to use wolfssl. Attempt to connect to an ssh server. In my case OpenSSH 8.9.

Expected behavior
For the encryption to work and the ssh handshake to complete.

Version

• OS and version: Linux Ubuntu 22.04.3 LTS
• libssh2 version: 1.11.0
• crypto backend and version: wolfSSL 5.6.4

Additional context
Add any other context about the problem here.

[vszakats]

initial report: #1020

[vszakats]

@brucsc: Can you make PR out of your fix proposal?

You can also find earlier discussions about this in #1020, with
the patch necessary to re-enable CI tests for wolfSSL, in this comment:
#1020 (comment)

[kareem-wolfssl]

This should be fixed with my colleague's PR here: wolfSSL/wolfssl#7143
This PR was not merged until wolfSSL 5.7.0, please try updating to 5.7.0 and try again.

[vszakats]

@kareem-wolfssl Can you confirm this is the same issue as #1020, or are they different?

[kareem-wolfssl]

@vszakats as best as I can tell this is the same issue, #1020 was also caused by the line EVP_Cipher(*ctx, NULL, NULL, 0);. So the linked PR/5.7.0 should resolve both issues.

[vszakats]

Thanks, also for the info about the fix!

I prefer @MichaelBuckley's PR, which replaces that API call with one that works with all wolfSSL versions. The Linux CI runner is still at 5.2.0.

[brucsc]

Great, I was very uncomfortable with what I presented in the initial question since it seemed like I was covering over an issue instead of fixing it.