Calling NUnitLite from LINQpad, can't parse assembly path

[astrohart]

I am a frequent user of LINQPad and I am trying to run some really quick unit-tests in LINQPad and I have done so by adding the NUnitLite NuGet package to my LINQPad query.

I went on this blog and they advised utiling NUnitLite and using the following code in a C# Program LINQPad query's Main() method:

void Main()
{
  new TextRunner().Execute(new[]{"--noheader"});
}

// Define other methods and classes here
[Test]
public void SomeTest()
{
  Assert.Pass();
}

Obviously, we're assuming that the NuGet has been added to the query and that all the proper namespaces are brought in.

Well, this does not work with the latest NUnitLite. The results I get are;

No test assembly was specified.

Usage: NUNITLITE-RUNNER assembly [options]
       USER-EXECUTABLE [options]

Runs a set of NUnitLite tests from the console.

Assembly:
      File name or path of the assembly from which to execute tests. Required
      when using the nunitlite-runner executable to run the tests. Not allowed
      when running a self-executing user test assembly.

Options:
      --test=NAMES           Comma-separated list of NAMES of tests to run or
                               explore. This option may be repeated.
      --testlist=PATH        File PATH containing a list of tests to run, one
                               per line. This option may be repeated.
      --prefilter=NAMES      Comma-separated list of NAMES of test classes or
                               namespaces to be loaded. This option may be
                               repeated.
      --where=EXPRESSION     Test selection EXPRESSION indicating what tests
                               will be run. See description below.
      --params, -p=VALUE     Define a test parameter.
      --timeout=MILLISECONDS Set timeout for each test case in MILLISECONDS.
      --seed=SEED            Set the random SEED used to generate test cases.
      --workers=NUMBER       Specify the NUMBER of worker threads to be used
                               in running tests. If not specified, defaults to
                               2 or the number of processors, whichever is
                               greater.
      --stoponerror          Stop run immediately upon any test failure or
                               error.
      --wait                 Wait for input before closing console window.
      --work=PATH            PATH of the directory to use for output files. If
                               not specified, defaults to the current
                               directory.
      --output, --out=PATH   File PATH to contain text output from the tests.
      --err=PATH             File PATH to contain error output from the tests.
      --result=SPEC          An output SPEC for saving the test results. This
                               option may be repeated.
      --explore[=SPEC]       Display or save test info rather than running
                               tests. Optionally provide an output SPEC for
                               saving the test info. This option may be
                               repeated.
      --noresult             Don't save any test results.
      --labels=VALUE         Specify whether to write test case names to the
                               output. Values: Off, On, All
      --test-name-format=VALUE
                             Non-standard naming pattern to use in generating
                               test names.
      --teamcity             Turns on use of TeamCity service messages.
      --trace=LEVEL          Set internal trace LEVEL.
                               Values: Off, Error, Warning, Info, Verbose (
                               Debug)
      --noheader, --noh      Suppress display of program information at start
                               of run.
      --nocolor, --noc       Displays console output without color.
      --help, -h             Display this message and exit.
      --version, -V          Display the header and exit.
Notes:
    * File names may be listed by themselves, with a relative path or 
      using an absolute path. Any relative path is based on the current 
      directory.

    * On Windows, options may be prefixed by a '/' character if desired

    * Options that take values may use an equal sign or a colon
      to separate the option from its value.

    * Several options that specify processing of XML output take
      an output specification as a value. A SPEC may take one of
      the following forms:
          --OPTION:filename
          --OPTION:filename;format=formatname

      The --result option may use any of the following formats:
          nunit3 - the native XML format for NUnit 3
          nunit2 - legacy XML format used by earlier releases of NUnit

      The --explore option may use any of the following formats:
          nunit3 - the native XML format for NUnit 3
          cases  - a text file listing the full names of all test cases.
      If --explore is used without any specification following, a list of
      test cases is output to the console.

And then a NullReferenceException is caught.

So, I tried to give it the assembly name as follows:

void Main()
{
    new TextRunner().Execute(new[] { $@"assembly {Assembly.GetAssembly(typeof(UserQuery)).Location}"});
}

/* ... tests ... */

I get:

NUnitLite 3.13.0 (.NET Framework 4.5)
Copyright (c) 2021 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.19041.0 
  CLR Version: 4.0.30319.42000 

Test Files
    assembly D:\Users\Foo\AppData\Local\Temp\LINQPad5\_lahhlytm\query_qwchmc.dll

System.NotSupportedException: The given path's format is not supported.
   at System.Security.Util.StringExpressionSet.CanonicalizePath(String path, Boolean needFullPath)
   at System.Security.Util.StringExpressionSet.CreateListFromExpressions(String[] str, Boolean needFullPath)
   at System.Security.Permissions.FileIOPermission.AddPathList(FileIOPermissionAccess access, AccessControlActions control, String[] pathListOrig, Boolean checkForDuplicates, Boolean needFullPath, Boolean copyPathList)
   at System.Security.Permissions.FileIOPermission..ctor(FileIOPermissionAccess access, String path)
   at System.Reflection.AssemblyName.GetAssemblyName(String assemblyFile)
   at NUnit.Framework.Internal.AssemblyHelper.Load(String nameOrPath)
   at NUnitLite.TextRunner.Execute()

I am not 100% sure why I am getting a NotSupportedException here. There are no spaces in the path and it's a perfectly valid file name. I can highlight the path, copy it and paste it into File Explorer and there's a file that certainly exists at that path.

Suspected Issue: NUnitLite, instead of simply calling System.IO.File.Exists() on whatever value it's given for the assembly parameter (and being done with it), instead is laboriously trying to validate the path and stuff. I think this is hogwash.

Suggested Fix: Parse out whatever value is passed to assembly and simply call System.IO.File.Exists(path) on it.

[mikkelbu]

I don't use NUnitLite or LINQpad, so this is just testing locally. From the log it seems that it tries to run the assembly file called assembly D:\Users\Foo\AppData\Local\Temp\LINQPad5\_lahhlytm\query_qwchmc.dll (notice the assembly in the beginning of the filename). If I change it to new TextRunner().Execute(new[] { $@"{Assembly.GetAssembly(typeof(UserQuery)).Location}" }); then it work locally on my machine. Alternatively, you can try new AutoRun().Execute(new string[] { });.

Ps. I don't think we do a lot of validation of the filename - the stacktrace above is mostly .NET code.

[stevenaw]

@astrohart Could you please confirm which version of Linqpad you see this on?

[astrohart]

Hi @stevenaw I am using LINQPad 5.43.00(AnyCPU).

[astrohart]

Just now tried a run as:

LINQPad Query:

void Main()
{
	new TextRunner().Execute(new[] { $@"{Assembly.GetAssembly(typeof(UserQuery)).Location}" });
}

// Define other methods and classes here
[Test]
public void TestMethd()
{
	Assert.IsTrue(true);
}

The following output was displayed:

See the circled NullReferenceException. When I expand the stack trace, I get:

   at LINQPad.ObjectGraph.Formatters.FastChannelResultsWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at LINQPad.ObjectGraph.Formatters.OutputWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at System.IO.TextWriter.SyncTextWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at NUnit.Common.ExtendedTextWrapper.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at NUnitLite.TextRunner.Execute(String[] args)
   at UserQuery.Main() in D:\Users\NCWDG_Developer\AppData\Local\Temp\LINQPad5\_xyymcaak\query_xkzakn.cs:line 46
   at LINQPad.ExecutionModel.ClrQueryRunner.Run()
   at LINQPad.ExecutionModel.Server.RunQuery(QueryRunner runner)
   at LINQPad.ExecutionModel.Server.StartQuery(QueryRunner runner)
   at LINQPad.ExecutionModel.Server.<>c__DisplayClass153_0.<ExecuteClrQuery>b__0()
   at LINQPad.ExecutionModel.Server.SingleThreadExecuter.Work()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

It seems to me as if there is a thread somewhere that has not called Dispose() on a TextWriter, but I am not 100% sure. When I squint a little bit at the stack trace, it almost looks like maybe the exception is LINQPad's fault.

[stevenaw]

Thanks @astrohart
I'm not a big Linqpad user either, but coincidentally that's the exact same version I have installed. And I'm able to reproduce too (both with NUnitlite 3.13 + 3.12)

[astrohart]

Thanks @astrohart
I'm not a big Linqpad user either, but coincidentally that's the exact same version I have installed. And I'm able to reproduce too (both with NUnitlite 3.13 + 3.12)

I am confused what is it you are able to reproduce? The NullReferenceException at the end? Could you please be just a little bit more specific?

[stevenaw]

Sorry, I've been juggling commenting from my phone while testing on a desktop. Let me elaborate:

When running this:

<Query Kind="Program">
  <NuGetReference>NUnitLite</NuGetReference>
  <Namespace>NUnit.Framework</Namespace>
  <Namespace>NUnitLite</Namespace>
</Query>

void Main()
{
	//new AutoRun().Execute(new string[] { });
	//var codeBase = $@"{Assembly.GetAssembly(typeof(UserQuery)).Location}";
	//var v = AssemblyName.GetAssemblyName(codeBase);
	
	new TextRunner().Execute(new[] { $@"{Assembly.GetAssembly(typeof(UserQuery)).Location}" });
}

// Define other methods and classes here
[Test]
public void SomeTest()
{
	Assert.Pass();
}

From a file saved on my desktop I can reproduce the NullReferenceException on Linqpad 5.43.00

at System.IO.TextWriter.Dispose()
at System.IO.TextWriter.Dispose()
at System.IO.TextWriter.SyncTextWriter.Dispose(Boolean disposing)
at System.IO.TextWriter.Dispose()
at NUnit.Common.ExtendedTextWrapper.Dispose(Boolean disposing)
at System.IO.TextWriter.Dispose()
at NUnitLite.TextRunner.Execute(String[] args)
at UserQuery.Main()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

[astrohart]

From looking at the stack trace, I would not be suprised but it seems as if LINQPad itself is not disposing of an internally-used TextWriter, from what I can tell. What do you think, @stevenaw?

[astrohart]

Perhaps should the NUnitLite.TextRunner.Execute method swallow such an exception?

[stevenaw]

I haven't had the chance to dig too much into why this is happening, but when testing locally with Linqpad 6.11.11(X64), NET50, and NUnitLite 3.13 I'm seeing slightly different results on my local.

This invocation line:

new TextRunner().Execute(new[] { $@"{Assembly.GetAssembly(typeof(UserQuery)).Location}" });

Now throws this:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.ArgumentException: Delegates must be of the same type.
   at System.MulticastDelegate.CombineImpl(Delegate follow)
   at System.Runtime.Loader.AssemblyLoadContext.add__resolving(Func`3 value)
   at System.Runtime.Loader.AssemblyLoadContext.add_Resolving(Func`3 value)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.EventInfo.AddEventHandler(Object target, Delegate handler)
   at NUnit.Framework.Internal.AssemblyHelper.ReflectionAssemblyLoader.TryInitialize() in D:\a\1\s\src\NUnitFramework\framework\Internal\AssemblyHelper.cs:line 124
   at NUnit.Framework.Internal.AssemblyHelper.ReflectionAssemblyLoader.TryGet() in D:\a\1\s\src\NUnitFramework\framework\Internal\AssemblyHelper.cs:line 108
   at NUnit.Framework.Internal.AssemblyHelper.Load(String name) in D:\a\1\s\src\NUnitFramework\framework\Internal\AssemblyHelper.cs:line 149
   at NUnitLite.TextRunner.Execute() in D:\a\1\s\src\NUnitFramework\nunitlite\TextRunner.cs:line 210

While using @mikkelbu 's snippet:

new AutoRun().Execute(new string[] { });

works successfully for me now on Linqpad 6. On Linqpad 5 however, I see the same TextWriter.Dispose() error.

It's tough for me to tell whether this is because of a difference in Linqpad or NUnitlite, since the different versions of Linqpad also consume different versions of NUnitlite (full framework vs core/NET5). I'm going to label this as a bug for further investigation

[mcintyre321]

There's a use SO answer on this https://stackoverflow.com/a/53753215/2086

[stevenaw]

This may have been fixed via #4325 which is scheduled for the next release