Guidance for testing CLI tools in .NET realm

[SidShetye]

We've had a tool (using your CLI framework) that started simple but is gaining usage. We've had a simple script to test the the linux-x64 and win-x64 release flavors but want to have some formal tests for it. I know we can do class level xunit tests but that's the recommended route when one want to test cli tools as if running within a terminal environment?

I suspect this is out of scope from your current focus on CommandLineUtils (and congrats on being a dad earlier this year, just learnt that!) but if you have pointers, those would be appreciated by my team. We'd be very curious how complex CLIs like dotnet handle testing scenarios.

[natemcmaster]

I'm sure there are a dozen ways to do testing for CLI tools. Here are some suggestions that have worked well for the .NET CLI tool projects I've worked on.

• As you mentioned, start with unit tests at a class level. To facilitate this:

  • Split the command-line argument parser and application execution into separate class structures. For example, see https://github.com/natemcmaster/dotnet-serve/. I have an options class which is parsed by a fairly simple entry point, and then passed in to the "real" application code. In testing, it makes it easy to test various options programmatically. It also makes it easier to test the string[] args parsing, which is good if you want to test more advanced parsing scenarios.
  • If your tests invoke CommandLineApplication, implement and pass in your own CommandLineContext...especially if you are doing file I/O and are taking advantage of the CommandLineApplication.WorkingDirectory property.
  • Go ahead and copy this helper into your project: makes debugging console output much friendlier. https://github.com/natemcmaster/CommandLineUtils/blob/v1.4.0/test/CommandLineUtils.Tests/Utilities/TestConsole.cs

• To better simulate a terminal environment, you'll need to use Process.Start to launch your app.

  • I wrote a helper class to wrap dotnet-serve for integration testing. You might be able to learn from its approach. https://github.com/natemcmaster/dotnet-serve/blob/v1.4.0/test/dotnet-serve.Tests/DotNetServe.cs
  • Using a little MSBuild glue code, you can make integration testing in your IDE work pretty well. Publish your console tool project into a test folder. Example: https://github.com/natemcmaster/dotnet-serve/blob/v1.4.0/test/dotnet-serve.Tests/dotnet-serve.Tests.csproj#L33-L37. Then use AppContext.BaseDirectory in test code to find your console tool executable. https://github.com/natemcmaster/dotnet-serve/blob/v1.4.0/test/dotnet-serve.Tests/DotNetServe.cs#L17
  • If you need to invoke "dotnet " during your tests, I would recommend using DotNetExe.FullPathOrDefault() so the "right" dotnet.exe is used and found (important for some CI environments)
  • Instead of string.Join(" ", testArgs), use ProcessStartInfo.ArgumentList (in the box in newer versions of .NET Core) or https://github.com/natemcmaster/CommandLineUtils/blob/master/src/CommandLineUtils/Utilities/ArgumentEscaper.cs. This will save you when you inevitably need to pass a space or quote into your test app.

Hope that helps!

[SidShetye]

Thanks Nate, appreciate the guidance. Solid stuff.