FileNotFoundException thrown during Stateful reliable service startup PipelineFactory.CreatePipeline call

[SvenVanHauwe]

I have added Eventflow to a stateful service recently, works on my local development service fabric instance.
After I deployed it to our azure service fabric (test) environment the service failed to activate. The event log on the node had following entries. the file eventFlowConfig.json is present in the Config. directory on that node.

Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException
at Microsoft.Diagnostics.EventFlow.ServiceFabric.ServiceFabricDiagnosticPipelineFactory.CreatePipeline(System.String, System.String, System.String)
at LayerAggregatorServicesManager.Program.Main()

[karolz-ms]

This usually means that the deployment cluster does not have some dependencies that the application expects. For example, the application might be targeting a version of the .NET Framework that is not installed on cluster nodes. It just so happens that EventFlow is called early from service Main() method and triggers assembly resolution that fails, resulting in FileNotFound exception

[karolz-ms]

@SvenVanHauwe when you figure out what the problem was, please let us know--it will be helpful for people who run into the same problem. We had another issue like this one and it was not fully explained

[SvenVanHauwe]

I enabled logging bindings on one node, netstandard 2.0 failed to load.

LOG: DisplayName = netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
(Fully-specified)
LOG: Appbase = file:///D:/SvcFab/_App/svcFType_App2/LayerAggregatorServicePkg.Code.1.0.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = LayerAggregatorService.exe
Calling assembly : Microsoft.Extensions.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60.

[SvenVanHauwe]

I think my problem might have something to do with Issues with .NET Standard 2.0 with .NET Framework & NuGet .

The reliable services which fail are targeting framework 4.6.2, and use packages.config.

[karolz-ms]

@SvenVanHauwe @oliverbes so is the workaround described in dotnet/standard#481 (enable auto-generation of binding redirects and using PackageReference instead of packages.config) feasible for you? And if yes, does your application work when you apply it?

It would be great if we could reproduce the problem outside of Service Fabric. I suspect a machine with .NET Fx 4.6.2 but without .NET Core FX would do. Will try that in the next few days, but let me know if you find something interesting in the meantime.

[oliverbes]

I actually gave it a try this afternoon - i.e. converting my SF Service's project file for auto-generated binding re-directs and using PackageReference instead of packages.config - but it failed.
My SF Service is now completely broken, probably I screwed the project file somehow.

As of my understanding, only the top-level NuGet Packages would get a PackageReference setting in the project file, right? Would that be sufficient?
So in my case the list of PackageRefence settings would look like:

  <ItemGroup>
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.Core" version="1.1.8" />
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.EtwUtilities" version="1.1.1" />
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.Inputs.EventSource" version="1.2.0" />
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.Outputs.ElasticSearch" version="2.2.0" />
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.Outputs.StdOutput" version="1.1.0" />
    <PackageReference Include="Microsoft.Diagnostics.EventFlow.ServiceFabric" version="1.1.4" />
    <PackageReference Include="Microsoft.ServiceFabric" version="5.7.198" />
    <PackageReference Include="Microsoft.ServiceFabric.Data" version="2.7.198" />
    <PackageReference Include="Microsoft.ServiceFabric.Diagnostics.Internal" version="2.7.198" />
    <PackageReference Include="Microsoft.ServiceFabric.Services" version="2.7.198" />
    <PackageReference Include="Newtonsoft.Json" version="10.0.3" />
    <PackageReference Include="Elasticsearch.Net" version="5.5.0" />
    <PackageReference Include="NEST" version="5.5.0" />
    <PackageReference Include="Validation" version="2.4.18" />
    <PackageReference Include="NETStandard.Library" version="2.0.1" />
  </ItemGroup>  

Would that list be OK? Anything missing?

[SvenVanHauwe]

@karolz-ms I decided rebuild my solution from scratch, dividing the different microservices into different solutions was something on my todo list anyway.
This gives me the opportunity to use the new project templates and packagereferences. Will inform you when it's done.

[SvenVanHauwe]

@karolz-ms I changed the csproj file as indicated in dotnet/standard#481, but still get the same exception.
When I build locally using visual studio 2017, the netstandard dll is copied to the bin directory.
On my build server, that dll is not included.

[SvenVanHauwe]

@karolz-ms Finally got one of the services to work on an azure servicefabric cluster.
I changed the csproj file of that service to the new format. After removing some binding redirect leftovers in App.Config I was finally able to build and deploy both locally and on our build server.

[karolz-ms]

@SvenVanHauwe so is the application working as expected now? If so, can you describe what the fix/workaround was?

[SvenVanHauwe]

@karolz-ms yes the application is working.
I modified the csproj files of the reliable services to use the new format, see the blog from Nate McMaster.

Just removing package.config and using packagereferences in csproj was not enough in my case. The project build, deployed and worked on my development local cluster.
It also build on our build server, however without netstandard dll in the output directory, which in turn gave a FileNotFound runtime exception.
My build definition contains a nuget install task to install and use nuget 4.3, and a visual studio build task to build the solution.

[karolz-ms]

OK, sorry for being dense here, but for the sake of other users of the EventFlow libraries, when you say

Just removing package.config and using packagereferences in csproj was not enough in my case.

then do you remember what did make it work when deployed to remote cluster?

[SvenVanHauwe]

@karolz-ms
I changed the project so it uses the dotnet sdk.
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup Label="Globals"> <Configurations>Debug;Release;azureDev</Configurations> <SccProjectName>SAK</SccProjectName> <SccLocalPath>SAK</SccLocalPath> <SccAuxPath>SAK</SccAuxPath> <SccProvider>SAK</SccProvider> <Platforms>AnyCPU;x64</Platforms> </PropertyGroup> <PropertyGroup> <TargetFramework>net462</TargetFramework> <RuntimeIdentifier>win7-x64</RuntimeIdentifier> </PropertyGroup> <PropertyGroup> <IsServiceFabricServiceProject>true</IsServiceFabricServiceProject> <ApplicationIcon /> <OutputType>Exe</OutputType> <StartupObject /> </PropertyGroup> <PropertyGroup> <UpdateServiceFabricManifestEnabled>true</UpdateServiceFabricManifestEnabled> <ServicePackagePath>PackageRoot</ServicePackagePath> <ServicePackagePrefix>$(MSBuildProjectName)</ServicePackagePrefix> </PropertyGroup> <ItemGroup> <None Include="PackageRoot\Config\Settings.xml" /> <None Include="PackageRoot\ServiceManifest.xml" /> <None Include="App.config" /> </ItemGroup> <ItemGroup> <PackageReference Include="Microsoft.ServiceFabric.Services.Remoting" Version="2.8.232" /> </ItemGroup> <ItemGroup> <ProjectReference Include="..\SeedingConfiguration.Interfaces\SeedingConfiguration.Interfaces.csproj" /> </ItemGroup> </Project>

[karolz-ms]

Roger!

Since it is working, I am going to close this issue for now