GitHub actions on MacOS failing since OPENSSL 1.0.2 removal

[ycrumeyrolle]

Related to actions/runner-images#2089

I have an issue since the 14th December on my dotnet unit test workflow running on MacOS :
The active test run was aborted. Reason: Test host process crashed : No usable version of libssl was found
This occurs on netcoreapp3.1 with SDK 3.1.404.

Current runner version: '2.275.1' Operating System Mac OS X 10.15.7 19H15 Virtual Environment Environment: macos-10.15 Version: 20201212.1

[scalablecory]

@safern can you help route this?

[safern]

This is happening in dotnet test (VSTest) host so I would start with @nohwnd

[nohwnd]

@ycrumeyrolle are you still facing that issue? I tried making a pipeline based on the info you shared and it works. https://github.com/nohwnd/macOSTestIssue/runs/1702266623?check_suite_focus=true I am not using any special version of test platform, not even the latest packages.

If you are still seeing the issue, plese check if there is something significantly different from what you are doing.

[ycrumeyrolle]

I am trying to reproduce the issue with your sample application. Without success for the moment...

[ycrumeyrolle]

@vcsjones @bartonjs we are able to reproduce the issue:
https://github.com/nohwnd/macOSTestIssue/runs/1742981935?check_suite_focus=true
It seems to be related to AES-GCM only. Other cryptography algorithms were tested successfully (AES, SHA256, RSA, ECDH).
AES-GCM trigger an error No usable version of libssl was found.
This error occured exactly at: actions/runner-images#2089

[vcsjones]

It seems to be related to AES-GCM only. Other cryptography algorithms were tested successfully (AES, SHA256, RSA, ECDH).

AesGcm and AesCcm rely on the presence of OpenSSL on macOS, which is not installed by default. Other algorithms like AES, SHA2, RSA, ECDSA, etc. all rely on SecurityTransforms / CommonCrypto, which is built in to macOS. That at least explains why AesGcm is singled out.

This looks similar to #46071.

[ycrumeyrolle]

According to https://github.com/actions/virtual-environments/blob/macOS-10.15/20201212.1/images/macos/macos-10.15-Readme.md, OpenSsl is installed by default on the GitHub MacOS image.
When running the command openssl version: OpenSSL 1.1.1i 8 Dec 2020

Maybe because of the openssl location ? ((/usr/local/opt/openssl -> /usr/local/Cellar/openssl@1.1/1.1.1i))

[ycrumeyrolle]

Just tried to add the environment variable for overriding the openssl path :
CLR_OPENSSL_VERSION_OVERRIDE: /usr/local/Cellar/openssl@1.1/1.1.1i
or
CLR_OPENSSL_VERSION_OVERRIDE: /usr/local/Cellar/openssl@1.1/1.1.1i/lib
Without success.

[bartonjs]

CLR_OPENSSL_VERSION_OVERRIDE only allows for controlling the (for macOS) version infix. e.g. CLR_OPENSSL_VERSION_OVERRIDE=1.1.1i would look for libssl.1.1.1i.dylib in the default library path.

Instead the problem is likely that the dylibs aren't in any default search path, so something like DYLD_FALLBACK_LIBRARY_PATH =$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/opt/openssl@1.1/lib would be what's required. (Using the path under opt means that you don't need to change the path every time a new revision is installed)

[ycrumeyrolle]

The variable DYLD_FALLBACK_LIBRARY_PATH is set to /usr/local/opt/openssl@1.1/lib (this folder contains the file libssl.1.1.dylib)
https://github.com/nohwnd/macOSTestIssue/runs/1758078565?check_suite_focus=true#step:9:5

The variable CLR_OPENSSL_VERSION_OVERRIDE is set to 1.1.1i (tested also without value and with value 1.1)
https://github.com/nohwnd/macOSTestIssue/runs/1758078565?check_suite_focus=true#step:9:6

But the test is still failing with the same error
https://github.com/nohwnd/macOSTestIssue/runs/1758078565?check_suite_focus=true#step:9:18

[carlossanlop]

@jeffschwMSFT is this a problem with the host not finding the DLLs? Any chance you can help route this?

[jeffschwMSFT]

@carlossanlop I would start with our crypto experts first.

cc @GrabYourPitchforks

[danmoseley]

Do any of the DYLD_PRINT environment variables help show what's happening?

https://stackoverflow.com/questions/51504439/what-environment-variables-control-dyld#51504440

[danmoseley]

Especially DYLD_PRINT_LIBRARIES

[MartyIX]

Does anybody know about a possible workaround until this is fixed?

[danmoseley]

@MartyIX I don't think we know enough about the problem. Did you try the workaround that @bartonjs suggested, which did not work for @ycrumeyrolle? Assuming so, could you please set the environment variables I mention above and get more info?

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Related to actions/runner-images#2089

I have an issue since the 14th December on my dotnet unit test workflow running on MacOS :
The active test run was aborted. Reason: Test host process crashed : No usable version of libssl was found
This occurs on netcoreapp3.1 with SDK 3.1.404.

Current runner version: '2.275.1' Operating System Mac OS X 10.15.7 19H15 Virtual Environment Environment: macos-10.15 Version: 20201212.1

Author:	ycrumeyrolle
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[MartyIX]

@danmoseley Yann mentioned a very simple-looking project https://github.com/nohwnd/macOSTestIssue which shows the issue. Isn't it enough to debug the issue? I'm not sure you are aware of it, so I'm just making it clear.

I can try to print DYLD_PRINT_LIBRARIES later today. I'm not sure how to do that at the moment.

Edit: https://github.com/nohwnd/macOSTestIssue/pull/1/files this is the repro.

[danmoseley]

@MartyIX ah yes. Still, we often ask folks to help with the diagnostic steps, since that can help get to a resolution more quickly; it can take some time for an issue to get up to the top of an engineer's list to look at in more detail.

[MartyIX]

@danmoseley I have just did this: https://github.com/nohwnd/macOSTestIssue/runs/2141171721?check_suite_focus=true (nohwnd/macOSTestIssue#3) where I print printenv. I believe that with this information it's much better but it's very easy to add further debug operations (I have added a single commit in a few minutes so ...)

[danmoseley]

OK, thanks. I don't have expertise to go further so I'll leave this up to owners of this area now.

[bartonjs]

The "Display OpenSsl lib 2" variant is using the patch-specific path (1.1.1i) which is failing because the current version is 1.1.1j. The first one worked, does everything pass if you get rid of the second version?

[MartyIX]

@bartonjs I can see The active test run was aborted. Reason: Test host process crashed : No usable version of libssl was found (https://github.com/nohwnd/macOSTestIssue/runs/2143008941?check_suite_focus=true) with commit nohwnd/macOSTestIssue@d610899. Is that what you have suggested to try?

[bartonjs]

To my understanding, things should be working. But, I note that DYLD_FALLBACK_LIBRARY_PATH doesn't appear in the printenv dump, which seems like https://apple.stackexchange.com/questions/212945/unable-to-set-dyld-fallback-library-path-in-shell-on-osx-10-11-1

I don't know if the run syntax allows for things like DYLD_FALLBACK_LIBRARY_PATH=/usr/local/opt/openssl@1.1/lib dotnet test --no-build --verbosity normal (or if that's subject to the same SIP problem); or if that type of invocation would work from a shell script. (Anyone who uses macs on a regular basis is invited to chime in)

The "easiest" approach is whoever owns the machines behind that pool adding symlinks for the /usr/local/opt/openssl@1.1/lib/lib{ssl,crypto}.1.1.dylib files to /usr/local/lib, or something else already in the path. But I have no idea who that is, or how hard it is to get them to make changes 😄.

[vcsjones]

adding symlinks for the

In the mean time, this is something that can be done in the actions to unblock anyone running in to this. Here is an example of a setting up the symlinks prior to running actions.

vcsjones/dotnet-openssl-actions@9ae6822

And I was able to get an action run to run successfully.

https://github.com/vcsjones/dotnet-openssl-actions/actions/runs/690449698

[MartyIX]

Kevin's workaround works great for me. Thanks!

Just for the sake of inspiration, this does the trick for me as I test on Ubuntu & macOS:

name: .NET Core
on: [push, pull_request]

jobs:
  build:
    strategy:
      matrix:
        operatingSystem: [ 'ubuntu-latest', 'macOS-latest' ]

    runs-on: ${{ matrix.operatingSystem }}
    timeout-minutes: 10

    steps:
    - name: Setup OpenSSL (macOS)
      if: matrix.operatingSystem == 'macOS-latest'
      run: |
        ln -s /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib /usr/local/lib/libssl.1.1.dylib
        ln -s /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib /usr/local/lib/libcrypto.1.1.dylib
    - uses: actions/checkout@v2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 5.0.x
    - name: Install dependencies
      run: dotnet restore
    - name: Build
      run: dotnet build --configuration Release --no-restore
    - name: Test
      run: dotnet test --no-restore --verbosity normal

[ycrumeyrolle]

@vcsjones the symlinks is working! 👍

@AlenaSviridenko @vcsjones @bartonjs
Adding the symlink working fix, but I would expect a more integreted solution.
Is it something that should be fixed in the dotnet SDK, or in the GitHub actions virtual environment? Or something that will not be fixed and should be the responsability of each user?

[bartonjs]

Given that the conversation has died down, and this is more about the GitHub Actions environment than .NET itself, I'm going to close the issue.

Hopefully all is quiet because things just work nicely in GitHub Actions. If not, at least we have a workaround (thanks, @vcsjones!)