PublishSingleFile runtime exception when *.deps.json is missing

[marcbarry]

Referencing #3778 and dotnet/core#3885

Hi @swaroop-sridhar, I am also experiencing a problem with code targeting netcoreapp3.1 making use of the <PublishSingleFile> property.

In my specific case, I have as-yet unknown process(es) across multiple customer sites which appear to be purging non .dll files from the default DOTNET_BUNDLE_EXTRACT_BASE_DIR locations (C:\Windows\Temp and C:\Users\username\AppData\Local\Temp) - presumably in a half-baked effort to proactively save space.

When .deps.json is removed from the directory it causes frustrating runtime exceptions.

For example, the following program (combined with <PublishSingleFile>true</PublishSingleFile>) when extract dependencies, but throw a FileNotFoundException looking for ConsoleApp1.deps.json at DependencyContext.Load() if the .deps.json file is removed.

using System;
using System.Reflection;
using Microsoft.Extensions.DependencyModel;

namespace ConsoleApp1
{
    class Program
    {
        static void Main()
        {
            var dependencyContext = DependencyContext.Load(Assembly.GetEntryAssembly());

            foreach (var lib in dependencyContext.RuntimeLibraries)
            {
                Console.WriteLine(lib.Name);
            }
        }
    }
}

I notice that a missing .deps.json isn't always fatal and many applications may continue to execute without requiring its presence, but when missing it can lead to previously unanticipated failure modes. In my case the NancyFx library fails to bind when .deps.json is missing, breaking key functionality in our application.

Expected behaviour

During startup, if an extracted dependency file is missing (by name, not content) then re-extract it.

While it does indeed seems unreasonable for dotnet to guard against third party tampering of extracted files in the temp directory, it does seem like the proactive measure of enumerating the directory for missing dependencies at runtime and replacing them if necessary carries low over-heads and delivers significant wins.

If the SingleFile application's temp extraction directory is removed entirely it is recreated at the next time the program executes. I propose an extension of to this behaviour to check for the presence of all dependencies.

The previously suggested workaround of setting the DOTNET_BUNDLE_EXTRACT_BASE_DIR is not a viable option in our case as it affects other applications on a customer's system too. It also adds non-standard requirements for each endpoint and on-going management and documentation for each customer.

Does it make sense for the runtime to conduct basic a due diligence sanity check on the contents of the extraction directory and light-touch validate its dependency exceptions each time the application starts?

Please note that I'm aware of the redesign for single-file functionality in netcoreapp5.0 and very much looking forward to it, but in the mean time is it possible that the single-file behaviour in previous versions of dotnet core can become a little more resilient to the lawless and unstable nature of the temp directory?

[ghost]

Tagging subscribers to this area: @swaroop-sridhar
Notify danmosemsft if you want to be subscribed.

[marcbarry]

By way of a quick update, I believe what's happening here is much simpler than I initially anticipated.

The temp directory is being emptied, either using disk clean up Explorer UI, or some other tool.

The dotnet core application (which in this case is a long running Windows service) is running when while the temp directory purge happens.

All dll files required to run the application survive the purge as the running application holds open file handles to them. But any auxiliary files not held open (such as the *.json files) are removed which leads predictably to the described behaviour following when the application is next restarted, killing expected behaviour and any code dependent on the presence of .deps.json.

I now believe this represents a bug in the design of this feature as long running Windows applications written in dotnet core do not maintain dependency integrity following routine disk clean-up operations and have no capability for automatic recovery.

[marcbarry]

Steps to reproduce

• Publish a <PublishSingleFile> app which makes use of DependencyContext.Load().
• Run it.
• Empty temp directory (either manually delete, or use disk cleanup / similar tool).
• Restart application.

Expected behaviour

Application detects missing dependencies and re-extracts to the temp directory on startup.

Actual behaviour

Missing dependencies are not detected as absent and the application starts, but throws runtime exceptions presenting unexpected and hard to debug behaviour in which some features may either partially work, or not work at all.

[swaroop-sridhar]

This is a dup of #3778; this fix will be included in this month's servicing release.

This fix will help recover missing extracted files at each startup.
However, if an app is very long running (typically few months) then it doesn't recover missing files in the midst of execution. For this case, the best recommendation is to set DOTNET_BUNDLE_EXTRACT_BASE_DIR to a non-temp directory.

In .net 5, managed single-file apps will be executed directly from bundle, which alleviates this problem.

[marcbarry]

Hmm. Was the fix you mentioned definitely included in v3.1.300 released on 2020-05-19? I'm reading the latest comments in #3778 which suggests it was - but I'm still seeing the same behaviour described in this issue.

I believe I'm testing using the latest SDK version and the behaviour remains the same. If I remove the contents of the extracted temp directory, a second execution of the assembly does not restore the deleted files.

I'm testing with <TargetFramework>netcoreapp3.1</TargetFramework> and using <PublishReadyToRun>true</PublishReadyToRun>

Do I need to be using a particular build flag or env variable to take advantage of the fix?

Test environment

C:\>dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.1.300
 Commit:    b2475c1295

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.300\

Host (useful for support):
  Version: 3.1.4
  Commit:  0c2e69caa6

.NET Core SDKs installed:
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.300 [C:\Program Files\dotnet\sdk]

[marcbarry]

Just to say, the fix is working for me now - I've updated visual studio to the latest version and restarted. Not sure why that was required to take advantage of this fix, but that seemed to be the missing step.

It works as expected now, thank you @swaroop-sridhar