Discussion: IDE Test Plan for New Language Features

[dpoeschl]

This issue is now closed. The IDE test plan is now checked in.

The purpose of this issue is to come to a consensus on what the general IDE Test Plan should be for testing new language features. This post is a work in progress - things may be added/removed/updated as we work through testing a few language features or as suggested changes are incorporated.

Test plan

• Read the feature specification and related conversations
• Think about how the new language feature interacts with each IDE feature
  • Are new keywords visualized appropriately?
  • Do refactorings and fixes produce correct code utilizing the new language feature?
  • How do IDE error recovery scenarios work?
  • Etc.
• Brainstorm ideas for new features to enhance the language feature
  • Should existing code be refactored to some new pattern?
  • Should users be able to easily switch between two equivalent forms?
  • Do we need to make the flow of this feature easier to visualize?
  • Etc.
• Open questions
  • IDE test coverage -- how much / which parts / added by whom?

Test Outputs

• Merge Signoff: Can the language feature be merged, with all Internal Dogfooding Requirements met and all IDE Features free of crashes and asserts?
• Feature Signoff: Per-feature signoff with links to non-blocking issues.
• Feature Suggestions: What can we add to the IDE to make the new language feature delightful to use?

Testing tags

✅ = Tested & Approved (possibly with linked bugs)
❌ = Merge blocking
🚧 = Non-blocking bugs
🆗 = Issue has been fixed
🆓 = No expected impact
❓ = Open question
❔ = Answered question

¹ Feature updated by compiler team, with appropriate unit tests.
² Feature requires more complete testing by IDE team. (?)

Internal Dogfooding Requirements (Merge Signoff)

Critical IDE features in the areas of Typing, Navigating, and Viewing must function as expected. Any non-trivial issues in these areas blocking language feature merging.
All feature descriptions and testing suggestions are merely examples. Each language feature should be carefully considered independently against each IDE feature to find interesting intersections

Category	Feature/Description	C# Signoff/Notes	VB Signoff/Notes
Enable/Disable	Feature Flag Completely enables/disables the feature in the compiler & IDE
Typing	General Typing - Type and paste new constructs - Nothing interferes with verbatim typing
	Completion - Typing new keyword/construct names - Dotting off of new constructs
	Formatting - Spacing in and around new constructs - Spacing options - Format Document command
	Automatic Brace Completion (C# only) - Auto-insert close brace - Shift+Enter commit of pending brace completion
	Indentation - Typing Enter in an unfinished statement indents the next line
Navigating	Go To Definition - F12 from callsites to definition
	Go To Implementation - Jump from virtual members to their implementations - Jump from inheritable types to their implementations
	Find All References - Lists references to a symbol in "Find Symbol Results" window - Shows results in hierarchy grouped by definition
Viewing	Colorization - Keywords, literals, and identifiers colored appropriately in code
	Error Squiggles - Squiggles appear as expected on reasonable spans

IDE Features (Feature Signoff)

For the remaining set of IDE features, only crashes and asserts are considered blocking. However, all issues discovered must be filed and linked here.
All feature descriptions and testing suggestions are merely examples. Each language feature should be carefully considered independently against each IDE feature to find interesting intersections

Code Transformations

Category	Feature/Description	C# Signoff/Notes	VB Signoff/Notes
Refactoring with UI	Inline Rename (with UI) - Dashboard shows correct information - Highlighted spans are updated appropriately - Rename operation updates the correct set of symbols
	Change Signature (with UI) - Updates all direct & cascaded definitions/callsites - Shows appropriate signature & parameter previews in UI
	Extract Interface (with UI) - Generated Interface has expected shape - UI shows appropriate method previews
	Generate Type (with UI) - Dialog gives all valid options
Refactorings	Rename Tracking - Tracking span tracks & dismisses as expected
	Extract Method - Extracted method has the expected signature - All arguments/return values handled correctly - Extracted code block is reasonable
	Introduce Variable - Introduced variable has the expected signature and initializer expression - "Introduce for All" correctly finds dupes
	Inline Temporary Variable - Inlined values are appropriately expanded/reduced
	Organize Usings - Honors "Place 'System' namespace first" option
	Convert "Get" Methods to Properties* Add Description
	Encapsulate Field Add Description
Fixes	Add Using - Triggers on appropriate constructs
	Generate Local Add Description
	Generate Field Add Description
	Generate Method/Constructor - Generated method has the expected signature and accessibility
	Generate Constructor from members - Select fields/properties to generate a constructor accepting corresponding arguments - Generated constructor has the expected signature and accessibility
	Implement Interface - Only missing methods added - All added methods have the expected signature and accessibility
	Implement IDisposable Add Description
	Implement Abstract Class Add Description
Code Gen	Snippets Add Description
	Event Hookup on Tab (C# only) - Type "+=" after an event name and QuickInfo shows - Invoking should pick good name & launch Inline Rename		N/A
	End Construct Generation (VB only) Add Description	N/A
	Automatic End Construct Update (VB only) Add Description	N/A

IDE Features

Feature/Description	C# Signoff/Notes	VB Signoff/Notes
Signature Help - Overloads shown with appropriate, colorized signature
Quick Info - Hover on identifiers - On completion list items
Outlining Add Description
Brace Matching (C# only) Add Description		N/A
Highlight References Add Description
Peek Add Description
Navigation Bars Add Description
Metadata As Source Add Description
Navigate To Add Description
Go to Next/Previous Method Add Description
Solution Explorer Pivots Add Description
Call Hierarchy Add Description
Code Lens Add Description
Project System Add Description
Debugger IntelliSense Add Description
Breakpoint Spans Add Description
Code Model / Class Designer Add Description
Object Browser / Class View Add Description
Lightbulb Add Description
Line Separators Add Description
Options Add Description

Project System & External Integration

(Any changes to the runtime will require much more testing in this category)
All feature descriptions and testing suggestions are merely examples. Each language feature should be carefully considered independently against each IDE feature to find interesting intersections

Category	Integration	Signoff/Notes
Projection Buffers	Razor - Verify expression and block contexts - Test on projection boundaries - Emphasis on rename and formatting
	Venus - Verify expression and block contexts - Test on projection boundaries - Emphasis on rename and formatting
Designers	WPF - Event generation from designer to code - Designer consumption of new types of members - Cross language features (GTD, FAR)
	WinForms Add Description
Project System Interactions	.NET Core "Console Application (Package)" Add Description
	Linked Files (all flavors) Add Description

Interaction with other new language features

Verify IDE handling of the new language feature in conjunction with other new/unreleased language features

Feature	C# Signoff/Notes	VB Signoff/Notes
<New Language Feature 1>
<New Language Feature 2>

New Feature Suggestions

What refactorings, fixes, code transformations, or other in-editor experiences would enhance this language feature?

Feature Name	Description
?	?

[dpoeschl]

/cc: @balajikris @jasonmalinowski @Pilchie @DustinCampbell

Are we headed in the right direction for a generalized IDE Test Plan for new language features here? I'll continue filling in feature descriptions as I move through them, and I hope that as we test a few different features, we'll come up with good guidance for testing specific features.

[DustinCampbell]

Sorry for not commenting sooner.

Yes, this is definitely the right direction. When we showed it to the DevLT meeting yesterday there was audible drooling. 👍

[DustinCampbell]

I suggest adding "WinForms" to the list of "Project System & External Integration"

[DustinCampbell]

"Inline Temporary Variable" is missing from refactorings

[DustinCampbell]

"Convert 'Get_' Methods to Properties" and "Encapsulate Field" are listed under Fixes but I _think* they're technically refactorings.

[alrz]

Implement IEquatable<T>, ==, !=, GetHashCode and Equals, that is really painful currently.

[DustinCampbell]

@alrz -- this isn't about feature requests. This is about creating a template for testing existing IDE features against new language features to ensure everything works properly. If you have a feature request, please feel free to file an issue.

[alrz]

@DustinCampbell Oh right, thanks.

[dpoeschl]

@DustinCampbell Thanks! Added Inline Temp, moved Methods->Properties & Encapsulate Field. WinFroms added to the list as well, but we haven't expanded it into suggested test areas yet.

[jcouv]

This issue is now closed. The IDE test plan is now checked in.