Copilot seems to break my tests execution in VS 2022

[vincent1405]

Select Topic Area

Bug

Body

Hi guys!

I am using the copilot extensions for VS 2022 from my GH Enteprise subscription.

As I ran my tests using NUnit and its VS Tests Adapter, I got this exception in the Tests ouput:

NUnit Adapter 4.5.0.0: Test execution complete
========== Test run finished: 48 Tests (48 Passed, 0 Failed, 0 Skipped) run in 471 ms ==========
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.<UpdateAskCopilotButtonVisibilityAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.<>c__DisplayClass18_0.<<AppendContent>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Threading.JoinableTask.CompleteOnCurrentThread()
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.AppendContent(ISummaryGenerator summaryGenerator, String indent, CancellationToken cancellation)
   at Microsoft.VisualStudio.TestWindow.Logging.ILoggerExtensions.CallInternalWithLogging(IInternalLogger log, Action action, Boolean shouldThrow)
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.<UpdateAskCopilotButtonVisibilityAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.<>c__DisplayClass18_0.<<AppendContent>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Threading.JoinableTask.CompleteOnCurrentThread()
   at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.AppendContent(ISummaryGenerator summaryGenerator, String indent, CancellationToken cancellation)
   at Microsoft.VisualStudio.TestWindow.Logging.ILoggerExtensions.CallInternalWithLogging(IInternalLogger log, Action action, Boolean shouldThrow)

As the exception comes from Microsoft.VisualStudio.TestWindow.Copilot.Summarize... I guess the NullReferenceException is thrown by the Copilot extension.

Does anyone encounter this issue ?

Thank you very much in advance for any help ;)

Technical info:

Microsoft Visual Studio Professional 2022
Version 17.9.6
VisualStudio.17.Release/17.9.6+34728.123
Microsoft .NET Framework
Version 4.8.09037

Installed Version: Professional

[...]

GitHub Copilot 1.181.0.1 (v1.181.0.1@b03e96737)
GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.

GitHub Copilot Agent 1.181.0
[...]
Visual Studio IntelliCode 2.2
AI-assisted development for Visual Studio.

[MatthewSteeples]

I'm seeing the same issue

Same versions of everything as @vincent1405 but we're using MSTest v3.3.1 instead of NUnit

[brianrizzo-deltek]

I am also seeing the same issue (same versions) as @vincent1405 using MSTest: v3.3.1

[MarkKingston]

I am also seeing the same issue.

[RWernerCo]

I am also seeing the issue

[jlhuang0821]

I am also seeing the issue

[tymesink]

Same issue with VS2022. I'm using XUnit. All my tests pass however, Visual Studio is reporting an error:

System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.<>c__DisplayClass18_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Threading.JoinableTask.CompleteOnCurrentThread()
at Microsoft.VisualStudio.TestWindow.Copilot.Summary.SummaryExtender.AppendContent(ISummaryGenerator summaryGenerator, String indent, CancellationToken cancellation)
at Microsoft.VisualStudio.TestWindow.Logging.ILoggerExtensions.CallInternalWithLogging(IInternalLogger log, Action action, Boolean shouldThrow)

[mrkoggles]

Same here. I get the exact same error and stack trace.
Using XUnit.

[ayim]

Tracking this issue here:
https://developercommunity.visualstudio.com/t/GitHub-Copilot-Chat-breaks-MSTestNUnit/10674159?

--
Aaron Yim
Product Manager for GitHub Copilot in Visual Studio

[MatthewSteeples]

That link gives me a 403 (both logged in and logged out)

[ayim]

Try this link: https://developercommunity.visualstudio.com/t/GitHub-Copilot-Chat-breaks-MSTestNUnit/10674159?

[vincent1405]

Hello,

Thank you for the link, but I have same issue as @MatthewSteeples (403, either if I am authenticated).

[Edelbitter]

I am also seeing the issue

The exceptions and the 403