.NET 8: CommandLineApplication.Execute with unknown command throws System.InvalidOperationException: Enumeration already finished

[craigktreasure]

Describe the bug

Using McMaster.Extensions.CommandLineUtils 4.1.0.

Upgrading my application from .NET 7 to .NET 8 and my tests and application started failing with an unexpected error when parsing unexpected commands.

Unhandled exception. System.InvalidOperationException: Enumeration already finished.
   at System.SZGenericArrayEnumerator`1.get_Current()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.CommandArgumentEnumerator.get_Current()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.CommandArgumentEnumerator.MoveNext()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessCommandOrParameter(CommandOrParameterArgument arg)
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessNext()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.Process()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Parse(String[] args)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Program.<Main>$(String[] args) in C:\Users\craig\Desktop\MyApp\MyApp\Program.cs:line 10

To Reproduce

Create a simple sample app with the following:

MyApp.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net8.0;net7.0</TargetFrameworks>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="McMaster.Extensions.CommandLineUtils" Version="4.1.0" />
  </ItemGroup>

</Project>

Program.cs

using McMaster.Extensions.CommandLineUtils;

CommandLineApplication app = new();
app.HelpOption();
app.OnExecute(() =>
{
    app.ShowHelp();
    return 1;
});
return app.Execute(args);

Now run the app with an invalid command for each of the target frameworks:

~\Desktop\MyApp\MyApp
> dotnet run -f net7.0 -- invalid
Specify --help for a list of available options and commands.
Unhandled exception. McMaster.Extensions.CommandLineUtils.UnrecognizedCommandParsingException: Unrecognized command or argument 'invalid'
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessUnexpectedArg(String argTypeName, String argValue)
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessCommandOrParameter(CommandOrParameterArgument arg)
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessNext()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.Process()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Parse(String[] args)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Program.<Main>$(String[] args) in C:\Users\craig\Desktop\MyApp\MyApp\Program.cs:line 10
~\Desktop\MyApp\MyApp
> dotnet run -f net8.0 -- invalid
Unhandled exception. System.InvalidOperationException: Enumeration already finished.
   at System.SZGenericArrayEnumerator`1.get_Current()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.CommandArgumentEnumerator.get_Current()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.CommandArgumentEnumerator.MoveNext()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessCommandOrParameter(CommandOrParameterArgument arg)
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.ProcessNext()
   at McMaster.Extensions.CommandLineUtils.CommandLineProcessor.Process()
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Parse(String[] args)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Program.<Main>$(String[] args) in C:\Users\craig\Desktop\MyApp\MyApp\Program.cs:line 10

Expected behavior

The result from the net7.0 invocation is what I would expect. It correctly indicates that the parameter wasn't recognized.

Additional context

I'm also seeing the issue with valid commands, but I haven't nailed down a minimal repro for that. I have a feeling the fix for the unexpected commands will either fix the same issue or shine light on a deeper issue.

Runtime information:

> dotnet --list-sdks
6.0.417 [C:\Program Files\dotnet\sdk]
7.0.404 [C:\Program Files\dotnet\sdk]
8.0.100 [C:\Program Files\dotnet\sdk]

> dotnet --list-runtimes
Microsoft.AspNetCore.App 6.0.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.25 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[sallerga]

There has been a change in the undefined behaviour for calling enumerator.Current before MoveNext has been called in net8: dotnet/runtime#94256

[thomaslevesque]

Is there a workaround for this? It basically makes the library unusable on .NET 8...
@natemcmaster are you aware of this?

[thomaslevesque]

The problem is here:

CommandLineUtils/src/CommandLineUtils/Internal/CommandLineProcessor.cs

Lines 534 to 539 in e0b6a2c

	public bool MoveNext()
	{
	if (Current == null || !Current.MultipleValues)
	{
	return _enumerator.MoveNext();
	}

MoveNext calls Current, assuming it will be null if the enumeration hasn't started. However, the behavior for calling Current after MoveNext() returned false is undefined; it used to return the last value, but now it throws.

[craigktreasure]

@thomaslevesque Correct. It's not usable on .NET 8. There is no workaround. I just spent a few days porting an app to System.CommandLine.

If you take a look above, there was also a fix submitted by @sallerga (#542).

[thomaslevesque]

If you take a look above, there was also a fix submitted by @sallerga (#542).

Oh, I missed it. Thanks, I'll take a look at it.

[prodigy]

We also just ran into this issue with pipelines failing after upgrading to .NET 8. Googling the error led me here. Just for my information. Do you by chance have a rough estimate when this will be available as nuget? 😊

Thank you for your great work!

[thomaslevesque]

Do you by chance have a rough estimate when this will be available as nuget?

Given that @natemcmaster has never replied to this issue, he's probably unavailable for a prolonged period of time. I wouldn't expect a new release soon.

I tried to switch to System.CommandLine, but it was hell. Since my use case was pretty simple, I eventually just wrote a bespoke parser.

[connorstorer-kbxcom]

I've got a simple workaround that's working for my case and I'm able to replicate issue consistently with existing commands. Executing a command with a following empty argument would consistently throw the Enumeration already finished exception.

tool "" --help -> Enumeration already finished

args.Where(a => !string.IsNullOrEmpty(a)).ToArray()

[natemcmaster]

Fixed in #542 by @sallerga. Thanks for the fix!

This will be released as version 4.1.1