C#: Grpc.Core native libs copying blocked by IIS locking the files

[vivainio]

What version of gRPC and what language are you using?

C#, Grpc.Core 2.26.0 , net472.

What operating system (Linux, Windows,...) and version?

Windows 10

What runtime / compiler are you using (e.g. python version or version of gcc)

.net 4.7.2, Visual Studio Build Tools 2017

What did you do?

I have a website using gRPC deplyed in IIS. While the application is running (i.e. has been accessed
at least once), try to rebuild the application.

What did you expect to see?

Application should be built succesfully, and IIS reloads the site.

What did you see instead?

Tyring to rebuild the application (by msbuild, or ctrl + shift + b in visual studio) gets interrupted by:

  C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(4582,5): 
error MSB3027: Could not copy "C:\Users\villevai\.nuget\packages\grpc.core\2.26.0\runtimes\win\native\grpc_csharp_ext.x64.dll" to "bin\grpc_csharp_ext.x64.dll". 
Exceeded retry count of 10. Failed. The file is locked by: "IIS Worker Process (22268)" [C:\.... \Startup.csproj]

This doesn't happen with managed libraries, as IIS doesn't lock them.

Anything else we should know about your project / environment?

I am using Paket, which automatically imports the targets file from Grpc.Core nuget package:
"packages\server\Grpc.Core\build\net45\Grpc.Core.targets".

This file contains:

    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\osx\native\libgrpc_csharp_ext.x86.dylib">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x86.dylib</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\osx\native\libgrpc_csharp_ext.x64.dylib">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x64.dylib</Link>
      <Visible>false</Visible>
    </Content>

Ignoring the error if it happens here (as it will probably always happen because file is locked), or making it conditional on target file not existing would help here.

Changing to:

 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>

doesn't seem to fix the problem, it still tries to copy the file.

I'm working around the issue by always recycling the IIS app pool before compilation, but that's not optimal.

[vivainio]

It would also be acceptable if there was a documented way to disable this copying altogether, e.g. with SkipGrpcNativeLibsCopy or something.

[lemkepf]

@vivainio I'm seeing the same thing. Version Grpc.Core 2.27.0 is also affected. The .net Core libraries don't seem to have this issue.

[vivainio]

Would this get more traction if I provided a PR implementing SkipGrpcNativeLibsCopy toggle?

[jtattermusch]

I think if you skipped copying the native libraries, gRPC C# would just stop working (because the necessary native libraries would be missing)

[vivainio]

I would copy the files to the target directories (of which there are several) in my own build rule, and only if the file doesn't exist. With this flag, users can exactly control when and how the copying happens.

This would not need documentation, as only the users googling for this problem would ever see it.

[jtattermusch]

@vivainio I think only copying a native library if the destination file doesn't exist is fragile because if you upgraded the Grpc.Core package, you could end up with a mutually incompatible copy of Grpc.Core.dll and the native libraries (which will very likely lead to crash or undefined behavior - so a pretty disastrous outcome). To make this safe, you would have to check if the source and destination files are identical, which would actually end up being pretty complicated.

My concern is that if we provide SkipGrpcNativeLibsCopy flag, it's still going to be hard for users to implement a reliable workaround to copy the native libs themselves and they are going to end up in weird hard-to-debug states (e.g. having incompatible native lib and Grpc.Core.dll).

[vivainio]

@jtattermusch yup, updating versions would probably mean a need to do clean build.

This should be considered an "i know what I'm doing" flag. Another thing that the custom build step could do is omitting mac and linux libs when only running on Windows. Or, the custom rule could try to copy over the file but fail immediately without retries if it's being locked.

Can the managed dll check that the native lib has a matching version number?

If this is not something seen as something that has a place in Grpc.Core package, is there an msbuild trick to "kill" the copying rule from outside? I can't avoid importing the .targets file since Paket forcefully does it anyway.

[jtattermusch]

@vivainio feel free to create a PR that adds <SkipGrpcNativeLibsCopy> flag so that we get a more concrete idea of how it would look like.
Please make sure to include a comment in the sense of "if you use this, you're taking full responsibility for making sure that your Grpc.Core assembly matches the native libraries and that the native libraries are copied to the location where they should."

[vivainio]

@jtattermusch PR here. It's a simple one as expected :). It only adds it to net45 targets, if this has wings I can add it to others as well #22894

[jtattermusch]

A workaround has been merged in #22894 There is not much we can do beyond that.

[rkalasky]

Seeing the same thing. Any idea when the flag will be released? Is there a workaround available in the meantime?

[jtattermusch]

This will be in the 2.30 release of gRPC C# (prerelease is coming within a few days and the stable release will follow soon afterwards).

[aburakab]

This will be in the 2.30 release of gRPC C# (prerelease is coming within a few days and the stable release will follow soon afterwards).

Here is the packages\Grpc.Core.2.30.0\build\net45\Grpc.Core.targets file for 2.30.0 Grp Core
Looks like the issue still exists.

Build operation will work only if I killed w3wp.exe process.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!-- 
    'Grpc_SkipNativeLibsCopy' should not be enabled in normal use.

    It only exists to support special scenarios where user wants to copy the native libraries
    to output directory themselves, in a separate build step or script.

    Only use this flag if you really know what you're doing. It's your responsibility to ensure that matching versions of 
    the Grpc.Core.dll assembly and the native libraries are used at all times.
  -->
  <ItemGroup Condition="'$(Grpc_SkipNativeLibsCopy)' != 'true'">
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\win\native\grpc_csharp_ext.x86.dll">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>grpc_csharp_ext.x86.dll</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\win\native\grpc_csharp_ext.x64.dll">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>grpc_csharp_ext.x64.dll</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\linux\native\libgrpc_csharp_ext.x86.so">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x86.so</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\linux\native\libgrpc_csharp_ext.x64.so">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x64.so</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\osx\native\libgrpc_csharp_ext.x86.dylib">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x86.dylib</Link>
      <Visible>false</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\..\runtimes\osx\native\libgrpc_csharp_ext.x64.dylib">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <Link>libgrpc_csharp_ext.x64.dylib</Link>
      <Visible>false</Visible>
    </Content>
  </ItemGroup>
</Project>

And Now I'm getting these two errors

Severity | Code | Description | Project | File | Line | Suppression State
-- | -- | -- | -- | -- | -- | --
Error |   | Could not copy "D:\.....\SourceCode\.nuget\packages\Grpc.Core.2.30.0\runtimes\win\native\grpc_csharp_ext.x64.dll" to "bin\grpc_csharp_ext.x64.dll". Exceeded retry count of 10. Failed. The file is locked by: "IIS Worker Process (22820)" | MyWebApp |   |   |  
Error |   | Unable to copy file "D:\......\SourceCode\.nuget\packages\Grpc.Core.2.30.0\runtimes\win\native\grpc_csharp_ext.x64.dll" to "bin\grpc_csharp_ext.x64.dll". The process cannot access the file 'bin\grpc_csharp_ext.x64.dll' because it is being used by another process. | MyWebApp |   |   |  

[jtattermusch]

@aburakab the workaround provided in #22894 is not supposed to be a full fix. It just provides the Grpc_SkipNativeLibsCopy property (which you need manually set yourself if you choose to and you know what you're doing - please doublecheck the comment) that allows you to skip copying the native libs altogether. #21867 (comment)

[vivainio]

@aburakab what @jtattermusch said. For illustrative purposes, I pushed an example msbuild file that "safely" copies the files to output dir: https://github.com/vivainio/GrpcNativeLibsCopier (using an own python script)

[Seabizkit]

how is this close when its still an issue am i missing something

[vivainio]

@Seabizkit it's closed because workaround for builds encountering this problem has been provided, and the underlying issue seems unfixable in grpc package side.

[Seabizkit]

@vivainio could a "better"* version of the fix be explained, in term of how to implement,

where does SkipGrpcNativeLibsCopy> go and what setting (true/false), and then what needs to happen after.

could a simpler version be explained.... which files and settings... its hard to follow as it is.
I have even downloaded the sample project, and trying to see how that applies.

where does SkipGrpcNativeLibsCopy> go... and then it needs to copy local files...?

need a little guide plz

[vivainio]

@Seabizkit I added Usage section to https://github.com/vivainio/GrpcNativeLibsCopier README. We should probably continue discussion there if you can't it working so I can improve it, as this ticket is closed