C#: Running in Alpine Linux docker container fails with "Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "/app/libgrpc_csharp_ext.x64.so"

[georgeslegros]

What version of gRPC and what language are you using?

C# and grpc 2.23

What operating system (Linux, Windows,...) and version?

Docker Alpine Linux 3.10 (tag: 3.1.0-alpine3.10)

What runtime / compiler are you using (e.g. python version or version of gcc)

dotnet core 3.1

What did you do?

Here is my dockerfile:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1.0-alpine3.10

RUN apk update && apk add --no-cache libc6-compat && apk add bash

WORKDIR /app
COPY ./ .

ENTRYPOINT ["dotnet", "./myapp.dll"]

Instead of libc6-compat we also tried gcompat and musl but it does not work

What did you expect to see?

My app should not crash and should be able to call my grpc services

What did you see instead?

We the application should contact a grpc service, we get the following error:
D1211 15:15:26.577582 Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "/app/libgrpc_csharp_ext.x64.so"

And this:
Error loading native library "/app/libgrpc_csharp_ext.x64.so". Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/libgrpc_csharp_ext.x64.so)

Anything else we should know about your project / environment?

We just migrated from dotnet core 2.2 to 3.1.
When we were in dotnet core 2.2 we were using Alpine 3.8 but the Microsoft Docker image for dotnet core does not support Alpine 3.8 anymore so we are kind of stuck.

I should also mention that I saw and tried the suggestions from #15605

I'll be happy to share more information if need be.

Thanks.

[georgeslegros]

Hi there, small bits of information that I'd like to share.
I think there is a major difference between Alpine 3.8 and 3.10 or at least in the way Microsoft put it together.

Look at this Dockerfile

FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-alpine3.8

ENV ASPNETCORE_VERSION=3.1.0

RUN apk update && apk add libc6-compat && apk add bash && rmdir -r /usr/share/dotnet && wget -O aspnetcore.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/3.1.0/aspnetcore-runtime-3.1.0-linux-musl-x64.tar.gz && mkdir -p /usr/share/dotnet && tar -zxf aspnetcore.tar.gz -C /usr/share/dotnet && rm aspnetcore.tar.gz && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

So what I did is basically take the dotnet core 3 image from Microsoft that is based on Alpine 3.8 and I upgraded the dotnet core runtime myself. Doing so, it works like a charm.

I'm going to use this like that for now and see if someone find a fix for Alpine 3.10

[AugustoDeveloper]

Thanks @hazzik, that's worked for me!

[cidico]

Thank you very much @hazzik !
Worked like a charm!

[maljelihawi]

@hazzik, Thank you very much!

[sheeeng]

@hazzik, thank you for your workaround! It works.

[obawin]

Thank-you @hazzik! Workaround worked for dotnet core 3.1 and alpine 3.11.

However, I'm not sure if it's related to the workaround, but wondering if anyone knows why and/or has encontered problems with Google.Cloud.Storage.V1 versioning when using the workaround?

In my dotnet core 3.1 project, I am referencing and using Google.Cloud.Storage.V1 2.5. It builds fine and I can build a dotnet core 3.1 alpine 3.11 docker image successfully because of the workaround. Also, when the image is deployed in GCP, I don't get the native library load error or segmentation fault service crashes thanks to the workaround.

I do get another error when executing one of the calls in the Google.Cloud.Storage.V1 library indicating "System.Interactive.Async 3.2.0" wasn't found. I noticed that the older Google.Cloud.Storage.V1 2.3 version had a transitive dependency on "System.Interactive.Async 3.2.0", but in version 2.5, that dependency has been replaced with Microsoft.Bcl.Async.

When I open up the dotnet core 3.1 alpine 3.11 docker image that was built, I can confirm that the Google.Cloud.Storage.V1 library included is 2.5 and the dependency json for the dotnet core app references Google.Cloud.Storage.V1 version 2.5 only.

Given the error, I suspect that at runtime Google.Cloud.Storage.V1 2.3 version libraries are being used/called instead of 2.5. I'm not if this is due to using libc6-compat=1.1.19-r11 in the workaround and/or some other reason (the way the image is built in the docker file, other configuration, etc.)?

Has anyone experienced anything similar and/or know what is going on? @jskeet When I search for this type of problem (.net core 3.1, alpine 3.11, glibc, libc), I see that you've looked at it quite a bit and are quite knowledgeable. Do you happen to know?

[jskeet]

TL;DR: You should update to Google.Cloud.Storage.V1 version 3.0.0-beta03. The "beta-ness" is in terms of possible API changes coming up, but it should absolutely be as bug-free as 2.5.0.

Google.Cloud.Storage.V1 version 2.5.0 depends on Google.Api.Gax.Rest version 2.10.0. The GAX 2.x line used the "old" style of IAsyncEnumerable<T> (from System.Interactive.Async), which conflicts with the "new" style of IAsyncEnumerable<T> in .NET Core 3.x.
This has provoked a major version update across the Google Cloud libraries - see https://googleapis.github.io/google-cloud-dotnet/docs/major-version.html for more details.

We're hoping to do a GA release of Google.Cloud.Storage.V1 version 3.0.0 fairly soon - we're just finishing up one new feature.

[obawin]

Thank-you @jskeet! I updated to Google.Cloud.Storage.V1 3.0.0-beta03, and it worked like a charm! Will keep an eye out on the the GA release.

[jangolano]

We are seeing the same issue with Google Pub/Sub c# library. We are on the 2.0 version of that. Any suggestions?

[jskeet]

@jangolano: There are different fixes depending on which version of Alpine you're using.
There's a fix in googleapis/google-cloud-dotnet#4780 that works for Alpine 3.10 and 3.11.

[jangolano]

I need it to work with the SDK as well as the runtime .net core 3.1. We are building our applications with a docker container using concourse CI, therefore the sdk is required. When running the tests it fails.

[jskeet]

I'm afraid I have no experience with trying to do that. I suggest you experiment with the solutions on the other issue.

[jangolano]

We tried the other solutions. We will wait till the libraries are updated. The other solutions don't work for the SDK.

[jskeet]

When you say "the libraries are updated" do you mean the libc libraries on Alpine, or the gRPC libraries, or something else? The API client libraries can't fix this, if you were thinking that - basically this is something the gRPC team need to work out, I believe.

[jangolano]

Our code is interacting with the .net GCloud libraries, either those have to be changed to work with something other than gPRC or the gPRC has to be fixed to work with this version of alpine.

[jskeet]

@jangolano: Given that the other solutions don't work for the SDK, I suggest you file a new issue in this repo with a minimal example - Dockerfile, C# project, Program.cs. There's a fairly minimal example here although it would be ideal to remove the google-cloud-dotnet libraries from the repro entirely. If you can make it as easy as possible for someone on the gRPC team to reproduce, it's much more likely to be fixed soon. (If it only affects users who need the SDK rather than just the runtime, there may be very few users facing this - possibly even just you. So the more you can do to help reproduce it, the better. Currently we still don't even know which version of Alpine you're using. You've said that things have to be fixed to work with "this" version of Alpine, but we don't know which version "this" is.)

[francesconi]

Language: F#
gRPC: 2.27
OS: Docker Alpine Linux 3.11
Runtime: dotnet core 3.1

Got my docker container up and running using the proposed fix by @hazzik. Although it randomly restarts after getting the following debug message:

Container Sandbox: Unsupported syscall membarrier(0x0,0x0,0x6f717d3c4c4f4700,0x3d3d3d3d3d3d3d01,0xfefefefefefefeff,0x1). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/membarrier for more information.

I also interact with a few GCloud libraries such as:

• Google.Cloud.Diagnostics.AspNetCore
• Google.Cloud.SecretManager.V1
• Google.Cloud.Storage.V1
• Google.Cloud.Tasks.V2

but they seem to work as expected.

EDIT 1:
The problem seems to be related with Google.Cloud.SecretManager.V1 and Google.Cloud.Diagnostics.AspNetCore as I no longer need to downgrade libc6-compat by removing these two dependencies.

[StasPerekrestov]

There is a little issue with the workaround

RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.8/main' >> /etc/apk/repositories && 
    apk update --no-cache &&
    apk add --no-cache bash libc6-compat=1.1.19-r11

Basically, it makes the container exposed to CVE-2019-14697 with CVSS V3 9.8 CRITICAL, and CVSS V2 7.5 HIGH scores.

[obawin]

thanks for the instructions @elibendavid ! wondering if you could help clarify the file copy/replacement part? after the successful cmake, these files containing "libgrpc_csharp_ext*" are present in the grpc/ directory:

./cmake/build/libgrpc_csharp_ext.so.14.0.0
./cmake/build/libgrpc_csharp_ext.so.14
./cmake/build/libgrpc_csharp_ext.so
./src/csharp/unitypackage/unitypackage_skeleton/Plugins/Grpc.Core/runtimes/linux/x64/libgrpc_csharp_ext.so.meta
./src/csharp/unitypackage/unitypackage_skeleton/Plugins/Grpc.Core/runtimes/android/armeabi-v7a/libgrpc_csharp_ext.so.meta
./src/csharp/unitypackage/unitypackage_skeleton/Plugins/Grpc.Core/runtimes/android/x86/libgrpc_csharp_ext.so.meta
./src/csharp/unitypackage/unitypackage_skeleton/Plugins/Grpc.Core/runtimes/android/arm64-v8a/libgrpc_csharp_ext.so.meta
./src/csharp/unitypackage/unitypackage_skeleton/Plugins/Grpc.Core/runtimes/ios/libgrpc_csharp_ext.a.meta

do one of these files need to be copied/overwrite the .net core runtime version? or is it the other way around (copy the .net core runtime version and overwrite the version in the grpc/ directory)? i don't see "/src/Core/GrpcClientCommon/lib/"

[elibendavid]

@obawin This is highly dependant on your installation of .net runtime as well as your app installation.
In my case it was - copy it to the exact location reported by the original error.

Looks like your originals are here (sym-links + bin):

/cmake/build/libgrpc_csharp_ext.so.14.0.0
./cmake/build/libgrpc_csharp_ext.so.14
./cmake/build/libgrpc_csharp_ext.so

Assuming you built on x64, you will want to rename the above file ( go for the main one not sym links )
to libgrpc_csharp_ext.x64.so - this is your source

Copy it to the location reported by the original error

[obawin]

Got it! Big thank-you @elibendavid! I dug around our development/deployment setup, and found that since we build/deploy a self-contained app, the updated libgrpc_csharp_ext.so.14.0.0 had to be renamed and overwrite the file that gets automatically pulled down in the Docker publish folder.

Hopefully the issue gets an official fix in a future update as it's a bit clumsy, but having this solution means Alpine 3.10+ should work fine with Grpc now!

[elibendavid]

Happy to help :- )

[hazzik]

Our best current workaround (in collaboration with @Belovolov)

FROM alpine:3.13 AS grpc-build
WORKDIR /opt
RUN apk add --update alpine-sdk autoconf libtool linux-headers cmake git && \ 
    \
    git clone -b v1.36.4 https://github.com/grpc/grpc --depth 1 --shallow-submodules && \
    cd grpc && git submodule update --init --depth 1 && \
    \
    mkdir -p cmake/build && cd cmake/build && \
    \
    cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DgRPC_BACKWARDS_COMPATIBILITY_MODE=ON \
      -DgRPC_BUILD_TESTS=OFF \
      ../.. && \
    \
    make grpc_csharp_ext -j4 && \
    \
    mkdir -p /out && cp libgrpc_csharp_ext.* /out

Then in another stage:

COPY --from=grpc-build /out/libgrpc_csharp_ext.so /app/libgrpc_csharp_ext.x64.so

No libc6-compat or glibc necessary.

[StasPerekrestov]

Thank you so much @hazzik. It works!

Initially, I've packed the lib into a nuget package and deployed it to /app/runtimes/

├── linux-musl-x64
│   └── native
│       └── libgrpc_csharp_ext.x64.so

however, it didn't work.
It looks like, it requires to be deployed to app root to work properly.

[mathieu-benoit]

Is there any chance this issue got resolved/fixed?

As workaround, I'm leveraging this #21446 (comment), and will soon test this other approach: #21446 (comment). But it would be ideal to get this fixed upstream. Thanks!

[mathieu-benoit]

I don't know if it could help @jskeet but here is the minimal setup I'm able to reproduce that issue with (just 3 files and 2 docker commands):

cat <<EOF > Program.cs
using Grpc.Core;
using System;
using System.Text;
var marshaller = new Marshaller<string>(Encoding.UTF8.GetBytes, Encoding.UTF8.GetString);
var method = new Method<string, string>(MethodType.Unary, "test-service", "test-method", marshaller, marshaller);
var channel = new Channel("spanner.googleapis.com:443", ChannelCredentials.Insecure);
var callInvoker = channel.CreateCallInvoker();
var text = callInvoker.BlockingUnaryCall(method, "spanner.googleapis.com", new CallOptions(), "request");
Console.WriteLine(text);
EOF

cat <<EOF > issue.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Grpc.Core" Version="2.38.1" />
  </ItemGroup>
</Project>
EOF

cat <<EOF > Dockerfile
FROM mcr.microsoft.com/dotnet/sdk:5.0.301 AS build
WORKDIR /app
COPY issue.csproj ./
WORKDIR /app
RUN dotnet restore issue.csproj -r linux-musl-x64
COPY / .
FROM build AS publish
WORKDIR /app
RUN dotnet publish issue.csproj -r linux-musl-x64 --self-contained true -c release -o out --no-restore
FROM mcr.microsoft.com/dotnet/runtime-deps:5.0.7-alpine3.13-amd64
WORKDIR /app
COPY --from=publish /app/out ./
ENTRYPOINT ["/app/issue"]
EOF

docker build -t issue .
docker run -it issue

Error generated:

Unhandled exception. System.IO.IOException: Error loading native library "/app/libgrpc_csharp_ext.x64.so". Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/libgrpc_csharp_ext.x64.so)
   at Grpc.Core.Internal.UnmanagedLibrary..ctor(String[] libraryPathAlternatives)
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethodsUsingExplicitLoad()
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethods()
   at Grpc.Core.Internal.NativeExtension..ctor()
   at Grpc.Core.Internal.NativeExtension.Get()
   at Grpc.Core.Internal.NativeMethods.Get()
   at Grpc.Core.GrpcEnvironment.GrpcNativeInit()
   at Grpc.Core.GrpcEnvironment..ctor()
   at Grpc.Core.GrpcEnvironment.AddRef()
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials, IEnumerable`1 options)
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials)
   at <Program>$.<Main>$(String[] args) in /app/Program.cs:line 6

[jskeet]

@mathieu-benoit: It really isn't in my area - I don't work on gRPC directly; I just maintain a lot of libraries that depend on it.

[mathieu-benoit]

Thanks @jskeet, gotcha. JFYI: I was tagging you since you asked earlier on that thread about a minimal example to reproduce the issue, so was just sharing to the maintainers of this repo too in case that could help. Hope we'll get an answer/ETA from them. Thanks!

[rmclemente]

I've changed from alpine to bullseye-slim, .Net 5, and everything is working fine now: Firestore and Pub/Sub. No need to install any extra package.
If Alpine is not a prerequisite, this can be an easy "solution".

[obawin]

Our best current workaround (in collaboration with @Belovolov)

FROM alpine:3.13 AS grpc-build
WORKDIR /opt
RUN apk add --update alpine-sdk autoconf libtool linux-headers cmake git && \ 
    \
    git clone -b v1.36.4 https://github.com/grpc/grpc --depth 1 --shallow-submodules && \
    cd grpc && git submodule update --init --depth 1 && \
    \
    mkdir -p cmake/build && cd cmake/build && \
    \
    cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DgRPC_BACKWARDS_COMPATIBILITY_MODE=ON \
      -DgRPC_BUILD_TESTS=OFF \
      ../.. && \
    \
    make grpc_csharp_ext -j4 && \
    \
    mkdir -p /out && cp libgrpc_csharp_ext.* /out

Then in another stage:

COPY --from=grpc-build /out/libgrpc_csharp_ext.so /app/libgrpc_csharp_ext.x64.so

No libc6-compat or glibc necessary.

Hi @hazzik,

I tried creating a Alpine 3.13 x86 + Grpc 1.36.4 image (added the option: -DCMAKE_TOOLCHAIN_FILE=../util/32-bit-toolchain.cmake), but it gives the following error:

........
#6 465.4 [ 47%] Building CXX object CMakeFiles/gpr.dir/src/core/lib/gpr/time_posix.cc.o
#6 465.5 /opt/grpc/src/core/lib/gpr/time_posix.cc: In function 'gpr_timespec now_impl(gpr_clock_type)':
#6 465.5 /opt/grpc/src/core/lib/gpr/time_posix.cc:77:13: error: 'SYS_clock_gettime' was not declared in this scope; did you mean 'SYS_clock_gettime32'?
#6 465.5 77 | syscall(SYS_clock_gettime, clockid_for_gpr_clock[clock_type], &now);
#6 465.5 | ^~~~~~~~~~~~~~~~~
#6 465.5 | SYS_clock_gettime32
#6 465.5 make[3]: *** [CMakeFiles/gpr.dir/build.make:566: CMakeFiles/gpr.dir/src/core/lib/gpr/time_posix.cc.o] Error 1
#6 465.5 make[3]: *** Waiting for unfinished jobs....
#6 465.9 make[2]: *** [CMakeFiles/Makefile2:1166: CMakeFiles/gpr.dir/all] Error 2
#6 465.9 make[1]: *** [CMakeFiles/Makefile2:785: CMakeFiles/grpc_csharp_ext.dir/rule] Error 2
#6 465.9 make: *** [Makefile:169: grpc_csharp_ext] Error 2
------

Would you happen to know how to fix the problem above?

Thanks.

[peterhirn]

I've been using mcr.microsoft.com/dotnet/aspnet:6.0.1-bullseye-slim as suggested by @rmclemente but now found an even slimmer alternative: distroless CBL-Mariner.

The docker tag is not exposed on the official dockerhub page but is listed under all available tags.

FROM mcr.microsoft.com/dotnet/aspnet:6.0.1-cbl-mariner1.0-distroless-amd64 as runtime

[CharlieDigital]

This continues to be an issue in Alpine; any chance of a long term fix for this?

[jtattermusch]

To summarize the situation

• Grpc.Core currently doesn't have first class support for Alpine Linux (the level of support we currently have is "best effort" so basically it might be possible to use Grpc.Core on alpine when the right compatibility workarounds are in place, but we don't test that continously nor guarantee fixes).
• The proper way to add support for alpine linux would be to build the grpc_csharp_ext.so native library on alpine linux (which is entirely doable, it can be built with cmake) and use it instead of the grpc_csharp_ext.so that comes with the default distribution (and targets glibc based linux, not musl). See

  grpc/tools/run_tests/artifacts/build_artifact_csharp.sh

  Line 23 in d589f4e

  cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \

• Since Grpc.Core is now in maintenance mode and we plan to deprecate it in the future in favor of grpc-dotnet (see https://grpc.io/blog/grpc-csharp-future/), it does not make sense to invest work into full support of Alpine Linux (we'd need to add continuous tests, bloat the Grpc.Core nuget with one more version of grpc_csharp_ext etc.)
  Note the the grpc-dotnet implementation support Alpine Linux fully since it has not native components (unlike Grpc.Core).

With that I'm going to close this issue as "won't fix".

[billybraga]

@jtattermusch Will the Google Cloud dotnet libraries be updated to use grpc-dotnet?

[jtattermusch]

@jtattermusch Will the Google Cloud dotnet libraries be updated to use grpc-dotnet?

Yes, in fact you should be able to use the client libraries with grpc-dotnet even today (at the cost at needing a bit of glue code), by using https://www.nuget.org/packages/Google.Api.Gax.Grpc.GrpcNetClient/

A much better integration of client libraries and grpc-dotnet is planned and should land over the next few months (and on supported platforms, grpc-dotnet will actually be the default option).

[noam-bionicstork]

I still get this problem in alpine3.14 and grpc 2.41.0. switching to debian image fix it

[TCBroad]

• Since Grpc.Core is now in maintenance mode and we plan to deprecate it in the future in favor of grpc-dotnet (see https://grpc.io/blog/grpc-csharp-future/), it does not make sense to invest work into full support of Alpine Linux (we'd need to add continuous tests, bloat the Grpc.Core nuget with one more version of grpc_csharp_ext etc.)

What are we meant to do when the google API we depend on uses Grpc.Core, even when updated to the latest version? The package was released in December 2021 so well into the maintenance period. It also uses the Gax package so can the legacy part be removed somehow?

[jskeet]

@TCBroad: We're currently working (right now!) on a new version of GAX that will depend on Grpc.Net.Client by default, with an option of using Grpc.Core to continue supporting .NET Framework users.

I'm hoping that the GA version of GAX v4 (and a re-release of all client libraries) will happen some time in early/mid-May.

[TCBroad]

@jskeet Thanks for the update - I'll check back in a month or so and hopefully update them then. I've managed to get a build working using the bullseye-slim image mentioned in this thread, but alpine would be preferable.

[liiujinfu]

you can add RUN ln -s /lib/ld-musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2
ld-musl-x86_64.so.1 is compatible with ld-linux-x86-64.so.2.

[ChaosYoda]

I know this issue was closed, but there is still a Docker file, https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/tools/dockerfile/distribtest/csharp_alpine_x64/Dockerfile that has a very nasty "hack" to get glibc working. Can that perhaps be updated or removed? alpine does have a compiled grpc library against musl libc, can that not be used?