Make location native binary configurable

[embix]

When using libgit2sharp from within LinqPad via NuGet, loading the native binary does not work as intended. The executing assembly usually lies somewhere "AppData\Local\Temp\LINQPad\whatever" while all referenced libraries lie elsewhere. CodeBase seems unset for the executing assembly. Thus the type initializer of NativeMethods fails put the correct location into the path:

        static NativeMethods()
        {
            if (!IsRunningOnLinux())
            {
                string originalAssemblypath = new Uri(Assembly.GetExecutingAssembly().EscapedCodeBase).LocalPath;

                string currentArchSubPath = "NativeBinaries/" + ProcessorArchitecture;

                string path = Path.Combine(Path.GetDirectoryName(originalAssemblypath), currentArchSubPath);

                const string pathEnvVariable = "PATH";
                Environment.SetEnvironmentVariable(pathEnvVariable,
                                                   String.Format(CultureInfo.InvariantCulture, "{0}{1}{2}", path, Path.PathSeparator, Environment.GetEnvironmentVariable(pathEnvVariable)));
            }

            // See LibraryLifetimeObject description.
            lifetimeObject = new LibraryLifetimeObject();
        }

A workaround is to add the correct path with EnvironMent.SetEnvironmentVariable at application start/composition root which seems a bit odd as it may break when updateing via nuget once again (see: http://stackoverflow.com/a/18867945/303290 ).

[nulltoken]

CodeBase seems unset for the executing assembly

Well, that's unexpected. I recall some discussion regarding this topic with @paulcbetts ages ago (and issues about shadow copies performed by test runners). Relying on EscapedCodeBase fixed this.

I wonder, maybe should we rely on Location (see the documentation and this post)?

Maybe would that also fix the DllNotFoundExceptionissue in #153 by @chrisparnin...

@embix could you please help us and share with us the content of the properties of Assembly.GetExecutingAssembly() when used in a LinqPad context?

/cc @phkelley @jamill

[embix]

Properties at runtime:

InvocableAttributeCtorToken='100663296'
Flags='ASSEMBLY_FLAGS_INITIALIZED'
SyncRoot='System.Object'
CodeBase='file:///C:/Users/mbretsch/AppData/Local/Temp/LINQPad/xxtrqhmt/query_ljhrtk.dll'
FullName='query_ljhrtk, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
EntryPoint=''
DefinedTypes='System.RuntimeType[]'
Evidence='System.Security.Policy.Evidence'
EvidenceNoDemand='System.Security.Policy.Evidence'
PermissionSet=
    '<PermissionSet class="System.Security.PermissionSet"
    version="1"
    Unrestricted="true"/>
    '
SecurityRuleSet='Level2'
ManifestModule='query_ljhrtk.dll'
ReflectionOnly='False'
Location='C:\Users\mbretsch\AppData\Local\Temp\LINQPad\xxtrqhmt\query_ljhrtk.dll'
ImageRuntimeVersion='v4.0.30319'
GlobalAssemblyCache='False'
HostContext='0'
IsStrongNameVerified='True'
IsDynamic='False'
EscapedCodeBase='file:///C:/Users/mbretsch/AppData/Local/Temp/LINQPad/xxtrqhmt/query_ljhrtk.dll'
ExportedTypes='System.Type[]'
IsFullyTrusted='True'
CustomAttributes='System.Collections.ObjectModel.ReadOnlyCollection`1[System.Reflection.CustomAttributeData]'
Modules='System.Reflection.RuntimeModule[]'

The directory contains the query_$tmp.dll, questy_$tmp.pdb, an empty handle "file" and a shadow_$tmp2 directory containing the MyExtensions dll. Other referenced dlls aren't included, at least not for nuget.

[nulltoken]

@embix Thanks a lot for this.

Can you find the NativeBinaries directory anywhere under the C:\Users\mbretsch\AppData\Local\Temp\LINQPad\xxtrqhmt\ directory structure?

[embix]

No, as I said no other referenced dll's are in there.

[Aimeast]

Yes, I real want it is configurable. I don't like NativeBinaries but x86, x64.
In order to be compatible with current code, we can load dll by IntPtr LoadLibrary(string fileName) method and it is real configurable.

references, PreLoadSQLiteDll method.

[nulltoken]

I suspect this may be related to the way the native binaries are currently deployed through a post build event, installled by the install.ps1 script.

#425 changes the way the native binaries are deployed to get rid of the dependency with powershell, but I'm not sure this will fix this issue. This might be related to the way LinqPad copies the .Net assembly in the temp folder hierarchy.

I've sent a tweet to the author a LinqPad regarding this. Maybe will he be able to help us go forward with this.

[nulltoken]

@Aimeast Why do you want it configurable? What is your use case, please?

[Aimeast]

@nulltoken configurable is mean a flexible program, I don't want keep native binary file named NativeBinaries. Can be put native binary dll file in flexible place. Including current working dir, x86 x64 dirs, libgitsharp self, and so on.

And it can be both managed-only assembly and mixed-mode assembly.

I like this kind of Using Native Library Pre-Loading

[nulltoken]

I don't want keep native binary file named NativeBinaries. Can be put native binary dll file in flexible place. Including current working dir, x86 x64 dirs, libgitsharp self, and so on.

@Aimeast Ok, I understand what you do not want. But... why? What problem are you currently facing requires you to have the binaries in a different location?

I like this kind of Using Native Library Pre-Loading

Unless I'm wrong, it looks like that System.Data.SQLite also requires a fixed tree structure with x86 and x64 folders from which the native binaries are loaded.

Similarly the LibGit2Sharp assembly expects to find next to it a directory structure as follows

NativeBinaries+
              |___amd64+
                       |___git2.dll (or git2-{shortsha}.dll depending of your version)
                       |___git2.pdb (or git2-{shortsha}.pdb depending of your version)
              |_____x86+
                       |___git2.dll (or git2-{shortsha}.dll depending of your version)
                       |___git2.pdb (or git2-{shortsha}.pdb depending of your version)

[Aimeast]

What problem are you currently facing requires you to have the binaries in a different location?

Different people have different habits. No any mistake of current LibGit2Sharp directory structure, but I didn't like this kind of structure.

Now, I just modified dir structure for my local project. I hope LibGit2Sharp can search right dll and load it.

[nulltoken]

Well, I think the potential shadow copy issue I was thinking of earlier can be taken off of the table.

The following test loads the LibGit2Sharp assembly into a separate AppDomain configured to shadow copy assemblies... and it passes.

using System;
using System.IO;
using System.Reflection;
using System.Security;
using System.Security.Permissions;
using LibGit2Sharp.Tests.TestHelpers;
using Xunit;

namespace LibGit2Sharp.Tests
{
    public class ShadowCopyFixture : BaseFixture
    {
        [Fact]
        public void CanProbeForNativeBinariesFromAShadowCopiedAssembly()
        {
            Type type = typeof(Wrapper);
            Assembly assembly = type.Assembly;

            // Build a new domain which will shadow copy assemblies
            string cachePath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
            Directory.CreateDirectory(cachePath);

            var setup = new AppDomainSetup
            {
                ApplicationBase = Path.GetDirectoryName(new Uri(assembly.CodeBase).LocalPath),
                ApplicationName = "ShadowWalker",
                ShadowCopyFiles = "true",
                CachePath = cachePath
            };

            setup.ShadowCopyDirectories = setup.ApplicationBase;

            AppDomain domain = AppDomain.CreateDomain(
                setup.ApplicationName,
                null,
                setup, new PermissionSet(PermissionState.Unrestricted));

            // Instantiate from the remote domain
            var wrapper = (Wrapper)domain.CreateInstanceAndUnwrap(assembly.FullName, type.FullName);

            // Ensure that LibGit2Sharp correctly probes for the native binaries
            // from the other domain
            string repoPath = BuildSelfCleaningDirectory().DirectoryPath;
            wrapper.CanInitANewRepositoryFromAShadowCopiedAssembly(repoPath);

            Assembly sourceAssembly = typeof(Repository).Assembly;

            // Ensure both assemblies share the same escaped code base...
            string cachedAssemblyEscapedCodeBase = wrapper.AssemblyEscapedCodeBase;
            Assert.Equal(sourceAssembly.EscapedCodeBase, cachedAssemblyEscapedCodeBase);

            // ...but are currently loaded from different locations...
            string cachedAssemblyLocation = wrapper.AssemblyLocation;
            Assert.NotEqual(sourceAssembly.Location, cachedAssemblyLocation);

            // ...that the assembly in the other domain is stored in the shadow copy cache...
            string cachedAssembliesPath = Path.Combine(setup.CachePath, setup.ApplicationName);
            Assert.True(cachedAssemblyLocation.StartsWith(cachedAssembliesPath));

            // ...that this cache doesn't contain the `NativeBinaries` folder
            string cachedAssemblyParentPath = Path.GetDirectoryName(cachedAssemblyLocation);
            Assert.False(Directory.Exists(Path.Combine(cachedAssemblyParentPath, "NativeBinaries")));

            // ...whereas `NativeBinaries` of course exists next to the source assembly
            string sourceAssemblyParentPath = Path.GetDirectoryName(sourceAssembly.EscapedCodeBase);
            Assert.True(Directory.Exists(Path.Combine(new Uri(sourceAssemblyParentPath).LocalPath, "NativeBinaries")));

            AppDomain.Unload(domain);
        }

        public class Wrapper : MarshalByRefObject
        {
            private readonly Assembly assembly;
            public Wrapper()
            {
                assembly = typeof(Repository).Assembly;
            }

            public void CanInitANewRepositoryFromAShadowCopiedAssembly(string path)
            {
                var gitDirPath = Repository.Init(path);

                using (var repo = new Repository(gitDirPath))
                {
                    Assert.NotNull(repo);
                }
            }

            public string AssemblyLocation
            {
                get
                {
                    return assembly.Location;
                }
            }

            public string AssemblyEscapedCodeBase
            {
                get
                {
                    return assembly.EscapedCodeBase;
                }
            }
        }
    }
}

[nulltoken]

@embix Unfortunately, I'm not a LinqPad Premium user and it will be difficult for me to troubleshoot this issue. Do you think there may be a way for you to open a ticket/issue with LinqPad as well? This way, we may jointly work on the issue.

I don't know how LinqPad leverages the NuGet packages. I imagine that they're installed through something similar to nuget install awesomepackage -o c:\nuget-packages. Currently, LibGit2Sharp cannot work from within the unpacked NuGet directory structure as the binaries and managed assemblies are located in different directory structures.

@fearthecowboy @dotnetjunky @haacked can you think of some NuGet packages containing managed assemblies and native binaries and that can be leveraged directly from the unpacked package directory structure? Is there some standard regarding this?

[embix]

Unfortunately I'm couldn't connect to the forum via OpenID or Google Account so I had to register via email. Thus having currently no right to post anything waiting for the admin to grant those permissions :-/

After setting the path as described in the StackOverflow post I can grab the LibGit2Sharp assembly information at runtime via GetType().Assembly. Its CodeBase and Location properties are indeed pointing into the NuGet package folder:

Location='C:\Users\embix\AppData\Local\LINQPad\NuGet\LibGit2Sharp\LibGit2Sharp.0.14.0.0\lib\net35\LibGit2Sharp.dll'
EscapedCodeBase='file:///C:/Users/embix/AppData/Local/LINQPad/NuGet/LibGit2Sharp/LibGit2Sharp.0.14.0.0/lib/net35/LibGit2Sharp.dll'

From my observations LinqPad stores nuget packages in AppData\Local\LINQPad\NuGet\$packagename the output of nuget install $packagename -o AppData\Local\LINQPad\NuGet\$packagename plus an additional LINQPadPackageInfo.xml for the packages metadata:

<?xml version="1.0" encoding="utf-8"?>
<LINQPadPackageInfo>
  <MainPackageVersion>0.14.0.0</MainPackageVersion>
  <MainPackageTitle>LibGit2Sharp</MainPackageTitle>
  <MainPackageSummary>LibGit2Sharp brings all the might and speed of libgit2, a native Git implementation, to the managed world of .Net and Mono.</MainPackageSummary>
  <MainPackageIconUrl>https://github.com/libgit2/libgit2sharp/raw/master/square-logo.png</MainPackageIconUrl>
  <Packages>
    <Package ID="LibGit2Sharp" Version="0.14.0.0">
      <AssemblyReferences>
        <Reference>lib\net35\LibGit2Sharp.dll</Reference>
      </AssemblyReferences>
      <FrameworkReferences />
    </Package>
  </Packages>
</LINQPadPackageInfo>

[albahari]

LINQPad references LINQPad assemblies directly from the NuGet directory structure to avoid the overhead of copying them to a temporary location with each query. It handles the AppDomain's AssemblyResolve event to ensure dependencies are located correctly. Obviously this doesn't work for unmanaged DLLs, but then dealing with unmanaged DLLs is tricky no matter how you do it. If you currently rely on a post-build event to get them into the right place, then it will not work on LINQPad even if LINQPad were to copy the managed assemblies to another folder.

If you're following the runtime convention of assuming the native DLLs are locating in NativeBinaries, then a workaround at your end may be to extend the convention to also check ....\NativeBinaries. IOW, before modifying the PATH variable, first check to see whether the NativeBinaries folder actually exists, and if not, check again whether the ....\NativeBinaries folder exists, and if so, use that instead. The it could work directly against the NuGet folder structure.

[Aimeast]

If LibGit2Sharp can search any possible directories to load native binary dll, maybe this issue will be close.