Avoid locking plugin DLLs when loading

[dylanrenwick]

Is your feature request related to a problem? Please describe.
When developing plugins, DesktopMagic needs to be fully closed to overwrite a plugin. This results in a frustrating loop of Make Change -> Close DM -> Update plugin file -> Reopen DM -> Repeat
Currently DesktopMagic uses Assembly.LoadFrom to load plugin DLLs. This results in a file lock on the plugin's main.dll, which prevents hot updates.

Describe the solution you'd like
A possible solution seems to be shadow copying, which copies the target DLL to a temp directory and loads the copy, but still uses the original path for binding context. This would allow DesktopMagic to load plugin assemblies without locking the original file, allowing the plugin to be hot-updated by overwriting the file and hitting "Reload" in DM.
A caveat to this is that the article linked in the above SO thread only applies to .NET Framework, and I'm struggling to find resources on shadow copying in general assembly loading on .NET 8+

Describe alternatives you've considered
It seems shadow copying could be implemented manually, though that could have unforeseen issues and inflates the scope of the change. I've also seen mention of 3rd party frameworks/libraries that can accomplish generic dotnet assembly "plugin" loading with hot reload support, such as MEF (Managed Extensibility Framework).
MEF is specific to .NET Framework 4, but modern alternatives may be available.

[Stone-Red-Code]

Hi there,

I'm actually aware of this, it's also a problem when you are trying to uninstall a loaded plugin because DM can't delete the files.

This is a problem since the last updated that fixed another issue causing plugin dependencies to not being loaded.
Before that update, the DLL files got loaded into memory, so the file wouldn't be locked. But since that prevented dependencies from loading correctly, I had to use another method that locks the files.

I also looked into shadow copying, but wasn't able to find a reasonable way to implement it.

As you mentioned, a third-party library is likely the best option. This probably would also simplify the whole plugin system, but would obviously require a rewrite of that system.

[Stone-Red-Code]

I will try to find a library that would work with the DM plugin system and try to implement it as soon as possible.

DotNetCorePlugins looks promising.

[dylanrenwick]

Loading the file into memory to avoid locking the file-on-disk was my first thought, so it's good to know that's been tried and ruled out. I hadn't considered plugin dependencies.
I'll do some research into and testing of shadow copy implementations and/or third-party libraries and see if I can find a good solution for DesktopMagic. DotNetCorePlugins does look very promising, hopefully it lives up to that promise!

I'm excited about the prospect of a C#/dotnet oriented replacement for Rainmeter (who thought object-oriented ini files were a good idea?) so I'm looking to contribute a bit 😄

[Stone-Red-Code]

It seems like the maintainer of DotNetCorePlugins doesn't want to maintain the library anymore and is considering archiving the repository. So that doesn't seem like a viable option.

The biggest problem with loading assemblies in .NET is unloading them.
The AssemblyLoadContext class should theoretically solve this problem, but I haven't played around with it yet.

Another option would be to load the plugins into a separate process that communicates with IPC with the main process.
This would make it very easy to unload the assembly because we can just stop/kill the process if required. And it would also solve some performance issues that are caused by WPF because it requires all UI interactions to run on the main UI thread. (The main window and other plugins can stutter when plugins are updating the Bitmap)

Both of these ideas would require some form of shadow copying because of the Assembly.Load method would still lock the files.

[dylanrenwick]

It's looking to me like we may need to implement shadow copying ourselves. I'm not finding many good options that are still maintained.

I'll take a stab over the weekend at implementing a standalone lib that can load an Assembly from a file without locking it, whilst still maintaining proper Load Context and dependencies

[dylanrenwick]

After some brief testing, it seems the solution may actually be as simple as copying the target file before loading it.
I created a simple proof-of-concept here to demonstrate this.
All it does is copy the .dll we want to load to a local _shadow directory before loading it. To my surprise this doesn't seem to break dependency resolution at all, and even appears to avoid locking loaded dependencies (Dependency.dll was not locked, even before the test project itself depended on it)

If you can think of a more complex real world use case/edge case that this PoC doesn't cover let me know and I'll add it to the test suite.

[dylanrenwick]

Additionally, it looks like if we load each plugin into its own AssemblyLoadContext and ensure they're set to collectible when created, we can unload plugins at runtime with AssemblyLoadContext.Unload().
Only whole LoadContexts can be unloaded though, not individual plugins, so each plugin would need its own LoadContext. I don't think this would cause problems, but it may warrant further investigation.

[Stone-Red-Code]

Your PoC looks good, but are the dependencies not copied into the _shadow directory?
When removing the Dependency project from the project references of the PluginDotNet.Test project, the DependentPlugin.dll fails to load because it can't find Dependency.dll

Additionally, it looks like if we load each plugin into its own AssemblyLoadContext and ensure they're set to collectible when created, we can unload plugins at runtime with AssemblyLoadContext.Unload(). Only whole LoadContexts can be unloaded though, not individual plugins, so each plugin would need its own LoadContext. I don't think this would cause problems, but it may warrant further investigation.

Yeah, that's how the AssemblyLoadContext is supposed to be used. Loading each plugin into its own AssemblyLoadContext shouldn't be a problem.

[dylanrenwick]

That's odd. I had run the DependencyPlugin test before adding the project reference and it had worked fine. I added the project reference solater that I could add the interface check.

I admittedly didn't check if the dependencies were also copied into _shadow, though I don't see that being an issue as we would probably need the dependencies shadow copied as well anyway, or at the very least not locked.

For that reason it may also make sense to give each loaded plugin its own subdirectory, to avoid file conflicts.

My one concern with this approach 8s shared dependencies, ie if multiple plugins reference the same dependency (is each plugin going to need its own PluginAPI assembly?)
I'll do some more testing and double check my results.

[dylanrenwick]

Running some further tests I was able to confirm your results; that the DependencyPlugin test fails if the test suite doesn't itself has a reference to the dependency.

Unfortunately I can't see a great way around this as there doesn't seem to be a good way to detect an assembly's dependencies before loading in a way that allows us to shadow copy and load them as well.

I'm going to keep investigating this, and look into how DotNetCorePlugins handled dependencies, but in the meantime an interrim solution may be to update the current plugin loading to use the AssemblyLoadContext approach, and provide an option on the UI to unload a plugin without having to restart DM itself.

This would still be an improvement over the current development cycle as you would just need to unload a single plugin, update the files, then reload

[Stone-Red-Code]

Thanks for looking into this.
Can't we just load the dependencies into the AssemblyLoadContext? If I understood that correctly AssemblyLoadContexts don't share dependencies by default.

[dylanrenwick]

We can. My concern is that we don't know what dependencies a plugin has without loading it, so when shadow copying the plugin assembly we don't know what dependencies also need to be shadow copied.

If we shadow copy the plugin assembly and not its dependencies, the library will fail to load with unresolved dependencies.