Service Fabric on Linux: libFabricResources.so is missed

[OlegKarasik]

Good day,

I was building docker container for build & test of C# application (.NET Standard 2.0) that uses Service Fabric packages and encountered the following error while tried to run dotnet test command:

[xUnit.net 00:00:02.00]     CoherentSolutions.Extensions.Hosting.ServiceFabric.Tests.Features.DependencyInjectionTests.Should_resolve_instance_From_autoregistred_types(case: CoherentSolutions.Extensions.Hosting.ServiceFabric.Tests.Theories.Items.TheoryItemPromise & IStatelessServicePartition) [FAIL]
[xUnit.net 00:00:02.01]       System.TypeInitializationException : The type initializer for 'System.Fabric.Common.AppTrace' threw an exception.
[xUnit.net 00:00:02.01]       ---- System.DllNotFoundException : Unable to load DLL 'libFabricCommon.so': The specified module or one of its dependencies could not be found.
[xUnit.net 00:00:02.01]        (Exception from HRESULT: 0x8007007E)
[xUnit.net 00:00:02.02]       Stack Trace:
[xUnit.net 00:00:02.02]            at Microsoft.ServiceFabric.Services.Remoting.V2.Messaging.BufferPoolManager..ctor(Int32 segmentSize, Int32 bufferLimit)
[xUnit.net 00:00:02.02]            at Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Runtime.FabricTransportServiceRemotingListener.InitializeSerializersManager(IServiceRemotingMessageSerializationProvider serializationProvider, FabricTransportRemotingListenerSettings listenerSettings)
[xUnit.net 00:00:02.02]            at Castle.Proxies.FabricTransportServiceRemotingListenerProxy..ctor(IInterceptor[] , ServiceContext serviceContext, IServiceRemotingMessageHandler serviceRemotingMessageHandler, FabricTransportRemotingListenerSettings remotingListenerSettings, IServiceRemotingMessageSerializationProvider serializationProvider)
[xUnit.net 00:00:02.02]         ----- Inner Stack Trace -----
[xUnit.net 00:00:02.02]            at System.Fabric.Interop.Utility.RunInMTA[TResult](Func`1 func)
[xUnit.net 00:00:02.02]            at System.Fabric.Common.Tracing.TraceConfig.InitializeFromConfigStore(Boolean forceUpdate)
[xUnit.net 00:00:02.02]            at System.Fabric.Common.AppTrace..cctor()

I have checked the /opt/microsoft/servicefabric/bin/Fabric/Fabric.Code/ path and found that libFabricCommon.so is there (below is the output of ls)

AzureCliProxy.sh
AzureFileUploader.dll
AzureFileUploader.pdb
AzureTableUploader.dll
AzureTableUploader.pdb
AzureUploaderCommon.dll
AzureUploaderCommon.pdb
ClusterVersion
Fabric
FabricDeployer.deps.json
FabricDeployer.dll
FabricDeployer.pdb
FabricDeployer.runtimeconfig.json
FabricDeployer.sh
FabricGateway.exe
FabricTypeHost.exe
FolderProducer.dll
FolderProducer.pdb
ImageBuilder.deps.json
ImageBuilder.dll
ImageBuilder.pdb
ImageBuilder.runtimeconfig.json
ImageBuilder.sh
LttProducer.dll
LttProducer.pdb
Microsoft.Data.Edm.dll
Microsoft.Data.OData.dll
Microsoft.Data.Services.Client.dll
Microsoft.WindowsAzure.Storage.dll
Newtonsoft.Json.dll
ServiceFabricServiceModel.xsd
System.Collections.Concurrent.dll
System.Collections.NonGeneric.dll
System.Collections.Specialized.dll
System.ComponentModel.Primitives.dll
System.ComponentModel.TypeConverter.dll
System.Diagnostics.DiagnosticSource.dll
System.Fabric.Dca.Validator.dll
System.Fabric.Dca.Validator.pdb
System.Fabric.Dca.dll
System.Fabric.Dca.pdb
System.Fabric.Management.ServiceModel.dll
System.Fabric.Management.ServiceModel.pdb
System.Fabric.Management.dll
System.Fabric.Management.pdb
System.Fabric.McgInterop.dll
System.Fabric.McgInterop.pdb
System.Fabric.deps.json
System.Fabric.dll
System.Fabric.pdb
System.IO.FileSystem.Primitives.dll
System.Linq.dll
System.Net.WebHeaderCollection.dll
System.Private.CompilerServices.ICastable.dll
System.Private.CoreLib.InteropServices.dll
System.Private.CoreLib.InteropServices.pdb
System.Private.DataContractSerialization.dll
System.Private.Interop.dll
System.Private.Interop.pdb
System.Runtime.Numerics.dll
System.Runtime.Serialization.Json.dll
System.Runtime.Serialization.Primitives.dll
System.Runtime.Serialization.Xml.dll
System.Security.Cryptography.OpenSsl.dll
System.Security.Cryptography.Primitives.dll
System.Spatial.dll
System.Threading.dll
System.Xml.XmlSerializer.dll
YamlDotNet.dll
__FabricSystem_App4294967295
azurecliproxy.deps.json
azurecliproxy.dll
azurecliproxy.pdb
azurecliproxy.runtimeconfig.json
bin
de
deployfabric.sh
es
fr
it
ja
ko
libFabricClient.so
**libFabricCommon.so**
libFabricImageStore.so
libFabricInfrastructureService.so
libFabricResources.so
libFabricRuntime.so
libFabricServiceCommunication.so
libFabricTokenValidationService.so
libboost_chrono-clang36-mt-d-1_61.so
libboost_chrono-clang36-mt-d-1_61.so.1.61.0
libboost_chrono.a
libboost_chrono.so
libboost_chrono.so.1.61.0
libboost_filesystem-clang36-mt-d-1_61.so
libboost_filesystem-clang36-mt-d-1_61.so.1.61.0
libboost_filesystem.a
libboost_filesystem.so
libboost_filesystem.so.1.61.0
libboost_prg_exec_monitor.a
libboost_prg_exec_monitor.so
libboost_prg_exec_monitor.so.1.61.0
libboost_random-clang36-mt-d-1_61.so
libboost_random-clang36-mt-d-1_61.so.1.61.0
libboost_random.a
libboost_random.so
libboost_random.so.1.61.0
libboost_regex-clang36-mt-d-1_61.so
libboost_regex-clang36-mt-d-1_61.so.1.61.0
libboost_regex.a
libboost_regex.so
libboost_regex.so.1.61.0
libboost_system-clang36-mt-d-1_61.so
libboost_system-clang36-mt-d-1_61.so.1.61.0
libboost_system.a
libboost_system.so
libboost_system.so.1.61.0
libboost_thread-clang36-mt-d-1_61.so
libboost_thread-clang36-mt-d-1_61.so.1.61.0
libboost_thread.a
libboost_thread.so
libboost_thread.so.1.61.0
libboost_timer-clang36-mt-d-1_61.so
libboost_timer-clang36-mt-d-1_61.so.1.61.0
libboost_timer.a
libboost_timer.so
libboost_timer.so.1.61.0
libboost_unit_test_framework.a
libboost_unit_test_framework.so
libboost_unit_test_framework.so.1.61.0
libc%2B%2B.so.1.1504
libc++.so.1
libcommon_utilities.so
libcoreclr.so
libcpprest.so
libcpprest.so.2.8
libhttptest_utilities.so
libicudata.so.52
libicui18n.so.52
libicuuc.so.52
libicuuc.so.52.1
pluginvalidators.dll
pluginvalidators.pdb
ru
runtimes
sfcli.sh
startservicefabricupdater.sh
system.fabric.strings.dll
system.fabric.strings.pdb
zh-Hans
zh-Hant

Then I traced the dependencies of libFabricCommon.so and found that libFabricResources.so isn't installed (below is the output of ldd)

linux-vdso.so.1 =>  (0x00007ffe56ffd000)
**libFabricResources.so => not found**
liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 (0x00007fb82b12d000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007fb82af04000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fb82ac9b000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fb82a857000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fb82a49c000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fb82a297000)
libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007fb82a07c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb829e74000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb829af2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb8297e9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb8295d3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb829209000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb829005000)
liblttng-ust-tracepoint.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007fb828dea000)
liburcu-bp.so.4 => /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4 (0x00007fb828be2000)
liburcu-cds.so.4 => /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4 (0x00007fb8289da000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb8287bd000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb82badc000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb8285a3000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fb8282c2000)
libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007fb827f2e000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fb827d0c000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fb827af8000)
libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007fb826041000)

Can you help me understand what is wrong here?

Here is the docker file I used to prepare build image:

FROM ubuntu:16.04

RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections && \
    echo "servicefabric servicefabric/accepted-eula-v1 select true" | debconf-set-selections && \
    echo "servicefabricsdkcommon servicefabricsdkcommon/accepted-eula-v1 select true" | debconf-set-selections && \
    echo "servicefabricsdkcommon servicefabricsdkcommon/accepted-all-eula select true" | debconf-set-selections

RUN apt-get update && \
    apt-get install --assume-yes apt-transport-https && \
    apt-get install --assume-yes apt-utils && \
    apt-get install --assume-yes sudo && \
    apt-get install --assume-yes software-properties-common
    
ADD https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb packages-microsoft-prod.deb
RUN dpkg -i packages-microsoft-prod.deb

RUN apt-get update && \
    apt-get install --assume-yes dotnet-sdk-2.1.4

RUN echo "deb [arch=amd64] http://apt-mo.trafficmanager.net/repos/servicefabric/ trusty main" > /etc/apt/sources.list.d/servicefabric.list && \
    echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet-release/ xenial main" > /etc/apt/sources.list.d/dotnetdev.list && \
    echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" > /etc/apt/sources.list.d/docker.list && \
    apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893 && \
    apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 417A0893 && \
    apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net --recv-keys 7EA0A9C3F273FCD8

RUN apt-get update && \
    apt-get install --assume-yes servicefabric servicefabricsdkcommon servicefabricsdkcsharp

[MedAnd]

@OlegKarasik - experienced something similar in issue: #1203

[ashishnegi]

@OlegKarasik libFabricResources.so is present in Fabric.Code which has libFabricCommon.so.
This is visible in ls output as shared by you.

To unblock youself, Can you set LD_LIBRARY_PATH to Fabric.Code in your script that starts the app ? like

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_fabric_code>

@chenxumsft / @maburlik What is recommendation for services running in the containers wrt LD_LIBRARY_PATH ?

[ashishnegi]

@OlegKarasik I now noticed that you are hitting the exception in dotnet test command.
This is expected as we need to set LD_LIBRARY_PATH in linux to access the native libraries.
So, have a setup script for your repository that adds <path/to/Fabric.Code> to LD_LIBRARY_PATH. Then dotnet test should succeed.

Closing this. Let us know if you face any problem with this.

[OlegKarasik]

@ashishnegi
I have added the LD_LIBRARY_PATH environment variable (and this fixed the problem with libFabricResources.so). Unfortunately now libFabricTransport.so isn't found. Should I open a new issue or we can reopen this one?

[ashishnegi]

@OlegKarasik Can you make sure that you have all these libraries in your Fabric.Code folder ?

Also, please double check this on a separate machine as these libs should be present.
If this is dev machine, may be this is some local development issue otherwise this will be our installer issue. Try reinstalling the SF deb installer.

[OlegKarasik]

@ashishnegi

Also, please double check this on a separate machine as these libs should be present.

Here is the list of libs missed:

1. libKPhysicalLog.so
2. libReliableCollectionRuntime.so
3. libFabricTransport.so
4. libFabricFaultAnalysisService.so

I am building docker image so I tried to reinstall everything from the scratch multiple times - the result is always the same.

[ashishnegi]

@OlegKarasik I will look into your dockerfile..
For unblocking you, can you build on top of service-fabric-onebox dockerfile ?

Here is some documentation on how to consume it.

[ashishnegi]

@OlegKarasik It seems that the problem is with trusty package : deb [arch=amd64] http://apt-mo.trafficmanager.net/repos/servicefabric/ trusty main .

If you follow documentation, you should be installing xenial packages.

You can try converting the script to Dockerfile or use above docker image i shared.

[OlegKarasik]

@ashishnegi

Thank you very much for all the help. I have worked through the documentation and script. According to them everything should work but it don't.

I have replaced trusty package with xenial: deb [arch=amd64] http://apt-mo.trafficmanager.net/repos/servicefabric/ xenial main and now it cannot be installed.

The error I am getting is:

Error: Failed to restart atop.service
dpkg: error processing package servicefabric (--configure):
 subprocess installed post-installation script returned error exit status 1

[ashishnegi]

@OlegKarasik
Did you just replaced trusty with xenail in your Dockerfile ? This did not work for me as well. You need to convert below script to Dockerfile.

Or Did you actually converted script to Dockerfile and it didn't work ?

Did you try service-fabric-onebox docker image ?

[OlegKarasik]

@ashishnegi

Did you just replaced trusty with xenail in your Dockerfile ? This did not work for me as well. You need to convert below script to Dockerfile. Or Did you actually converted script to Dockerfile and it didn't work ?

I have tried both. Here are the errors when using ubuntu:16.04 and service-fabric-onebox as base images:

In PostInst - configure
Creating sfuser
Creating sfappsuser
Enabling password authentication for FileStoreService accounts
Failed to connect to bus: No such file or directory
Copying servicefabric.service file to systemd
Copying servicefabricupdater.service file to systemd
Setting sfuser as owner of FabricBinRoot
Validate and fix any system service log files
getfacl: /dev/sr0: No such file or directory
getfacl: /dev/sda: No such file or directory
getfacl: /dev/sda1: No such file or directory
getfacl: /dev/sr1: No such file or directory
Failed to connect to bus: No such file or directory
Error: Failed to restart atop.service
dpkg: error processing package servicefabric (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of servicefabricsdkcommon:
 servicefabricsdkcommon depends on servicefabric (>= 6.3.119.1); however:
  Package servicefabric is not configured yet.

[ashishnegi]

@jeffreychenmsft Can you help @OlegKarasik get a working container image of service fabric ?

[ashishnegi]

@OlegKarasik Here is a setup script which you can use in Dockerfile before installing service fabric.

Also look at more files under src/prod/src/linux/dockers/

[OlegKarasik]

@ashishnegi

I hope I understand everything right.

I've tried to run the above setup script and after that execute:

apt-get update && \
apt-get install --assume-yes servicefabric servicefabricsdkcommon servicefabricsdkcsharp

This effectively results in the:

E: Unable to locate package servicefabric
E: Unable to locate package servicefabricsdkcommon
E: Unable to locate package servicefabricsdkcsharp

I have added:

sh -c 'echo "deb [arch=amd64] http://apt-mo.trafficmanager.net/repos/servicefabric/ xenial main" > /etc/apt/sources.list.d/servicefabric.list'
echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" > /etc/apt/sources.list.d/docker.list

And everything returned to the previous state with the error

Please correct me if I am doing something wrong but currently it is still not working.

[OlegKarasik]

In case if this would be helpful to anyone.

I have created a Docker Image that allows to run unit-tests that use Service Fabric classes.

[ashishnegi]

Thanks @OlegKarasik . Closing this issue.