wolfSSL: libssh2_session_handshake failed: -12/-44 DECRYPT/ENCRYPT

[vszakats]

Describe the bug
When running tests/test_ssh2, either of these errors happen:

• libssh2_session_handshake failed (-12) LIBSSH2_ERROR_DECRYPT: Failed to get response to ssh-userauth request
  https://github.com/libssh2/libssh2/actions/runs/4851558817/jobs/8645524071#step:7:1087

• libssh2_session_handshake failed (-44) LIBSSH2_ERROR_ENCRYPT: Unable to ask for ssh-userauth service
  https://github.com/libssh2/libssh2/actions/runs/4847366472/jobs/8637553577#step:7:1052
  https://github.com/libssh2/libssh2/actions/runs/4846389013/jobs/8635903998#step:7:96

This is the first packet with encryption.

To Reproduce
Run tests/test_ssh2 against an sshd server.

Expected behavior
Same result as with OpenSSL/LibreSSL backends:
https://github.com/libssh2/libssh2/actions/runs/4847366472/jobs/8637553403#step:7:72
https://github.com/libssh2/libssh2/actions/runs/4847366472/jobs/8637553480#step:7:72
https://github.com/libssh2/libssh2/actions/runs/4851558817/jobs/8645523936#step:7:71

Version:

• OS and version: macOS (possibly others)
• libssh2 version: 1.10.1-DEV
• crypto backend: wolfSSL v5.6.0

[vszakats]

Reverting the AES-GCM patch #797 / 3c953c0 fixes it.

The issue might be inside openssl.c / _libssh2_cipher_crypt(). Possibly with the interpretation of the return value of EVP_Cipher(), or EVP_CIPHER_CTX_ctrl().

@dfandrich Can you take a look?

The issue happens with both AES-GCM disabled for wolfSSL (as in master) or enabled.

UPDATE: Now enabled in master: df513c0.

[dfandrich]

I'll take a look.

[migueldeicaza]

I also have one user getting this error with OpenSSL, the trace looks like this:

https://gist.github.com/migueldeicaza/da684d07f91716ddefafe6ef298c3d64

[migueldeicaza]

Update: reverting locally the patches for 0048f30 and 3c953c0 did not solve the problem.

Here is one additional data point: this only happens on an M1 Mac connecting to an M1 Mac, or a CentOS machine. It does not happen from an Intel Mac to those systems, and does not happen with iPhone/iPad on device or simulator.

[migueldeicaza]

In case someone else runs into this: the issue on the M1 Mac was not related to the AES-GCM patch, that's why it did not make a difference. For some reason the version of OpenSSL I was using on Mac on ARM did not include the LIBSSH2_EC_CURVE_NISTP256 (NID_X9_62_prime256v1) curve, so the call to _libssh2_ecsa_curve_name_with_octal_new was failing which in turn made curve25519_sha256 fail.

Mind you, this code was not being triggered with the currently public version of libssh2

[kareem-wolfssl]

Hi @vszakats ,

I was able to reproduce the failures you're seeing here. Some failures are being caused by our AES-GCM streaming support not being enabled, you can enable this support while building wolfSSL by passing --enable-aesgcm-stream to ./configure. I will add this to --enable-libssh2 once I've confirmed it's working as expected.
I am still working on debugging the remaining test failures.

I would strongly recommend you manually build wolfSSL with --enable-libssh2 --enable-aesgcm-stream for your tests. The test appears to be using the Brew package of wolfSSL, which does not enable DES3, DSA or AES-CTR: https://github.com/Homebrew/homebrew-core/blob/7fd778f54e75e1f9f85f41486094a7182225d839/Formula/wolfssl.rb These algorithms are enabled by --enable-libssh2, and if they aren't enabled they will cause additional test failures.

[vszakats]

@kareem-wolfssl, thanks for jumping in!

Opened a PR to detect AES-GCM via WOLFSSL_AESGCM_STREAM (instead of HAVE_AESGCM): #1045
Does this look correct to you?

I think it'd be useful for most users if the Homebrew package came enabled with everything necessary for AES-GCM. At the moment it lists --enable-aesgcm twice, perhaps one of those was to meant to enable stream support? (as for DSA and DES3, I think the they are OK as disabled by default, libssh2 will adapt.)

Building wolfSSL from source in CI requires constant maintenance (bumping on each upgrade). Also makes CI builds much slower and complex. I'd like to avoid this and rely on package defaults. This is also what users will encounter on this platform by default.

This particular issue is also present with AES-GCM disabled, so keeping it open.

[kareem-wolfssl]

Happy to help!

It is correct, but to be safe I'd recommend checking both. I left a comment in the review.

wolfSSL does not control any distributions' packages, you will have to check with the Homebrew project on modifying their wolfSSL package.

You are correct that with the way libssh2 is set up, your other tests should still pass while skipping DSA/DES3, but this isn't ideal as it isn't giving full test coverage. In general, we expect you to be using --enable-libssh2 when building wolfSSL against libssh2, and it is possible we may add more features under this flag in the future, like AES-GCM streaming. We expect users to do this as well.
Building wolfSSL is really just pulling our current latest release, ./configure it with --enable-libssh2, make and make install. It should safe to have this automatically use our latest release, and you are welcome to open a bug upstream if this ever fails after a new release from us.

However, if you aren't able to do this I understand. We aim to automatically test all of our open source integrations to catch issues like this. I will see if we currently have a libssh2 test and if not, look into setting one up.

[kareem-wolfssl]

I have found another issue, wolfSSL does not like this call in openssl.c: ret = EVP_Cipher(*ctx, NULL, NULL, 0);
We don't support NULL src or dest and just return BAD_FUNC_ARG. Looks like we need to call EVP_CipherFinal instead here for wolfSSL, I am working on determining the right way to do that.

[vszakats]

In theory easy, but in practice every sub-build is a source of issues, maintenance, error cases and flakyness. Starting from how to extract the current latest version. In Homebrew, It's a one-time effort with wider benefits – seems more practical.

I might give it a try adding wolfSSL Linux builds, though it will be an older version. [edit: added in 801aebc]

DES3/DSA is tested with other crypto-backends, which seems enough for libssh2 purposes.

Very off topic: I'd be fantastic to have wolfSSL's ed25519 support exposed via its OpenSSL compatibility interface. That way we could enable it with libssh2.

[kareem-wolfssl]

Understood. I will still look into getting this added to our tests, to hopefully avoid situations like this in the future.

Looking at the libssh2 code, it looks like you just need ED25519 supported in the EVP APIs, as well as a couple of EVP APIs we don't currently support like EVP_PKEY_new_raw_public_key. We should be able to add this. I will log a feature request for this.

[vszakats]

@kareem-wolfssl This is great news, thanks for that!

Meanwhile, I've added (801aebc) Linux builds with wolfSSL to our CI. Version v5.2.0. This platform has the ability to run our dockerized runtime tests. All of them fail, so for now I've disabled them. They can be enabled (incl. logging) for testing with this patch:

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b42ae5af..3eddc7cf 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -57,6 +57,9 @@ jobs:
           [ "${{ matrix.crypto_backend }}" == 'wolfSSL' ] && pkg='libwolfssl-dev'
           [ "${{ matrix.crypto_backend }}" == 'Libgcrypt' ] && pkg='libgcrypt-dev'
           sudo apt-get install -y "${pkg}"
+          if [ "${{ matrix.crypto_backend }}" == 'wolfSSL' ]; then
+            echo 'TOOLCHAIN_OPTION=-DENABLE_DEBUG_LOGGING=ON' >> $GITHUB_ENV
+          fi
 
       - name: 'install mbedTLS from source'
         if: ${{ matrix.crypto_backend == 'mbedTLS' }}
@@ -100,9 +103,10 @@ jobs:
         if: ${{ matrix.build == 'cmake' }}
         run: cmake --build bld --parallel 3 --target package
       - name: 'cmake tests'
-        if: ${{ matrix.build == 'cmake' && matrix.crypto_backend != 'wolfSSL' }}
+        if: ${{ matrix.build == 'cmake' }}
         timeout-minutes: 10
         run: |
+          export FIXTURE_TRACE_ALL_CONNECT=1
           export OPENSSH_SERVER_IMAGE=ghcr.io/libssh2/ci_tests_openssh_server:$(git rev-parse --short=20 HEAD:tests/openssh_server)
           cd bld && ctest -VV --output-on-failure