Skip to content

Add refactoring-to-async skill#269

Open
mrsharm wants to merge 16 commits intodotnet:mainfrom
mrsharm:musharm/refactoring-to-async-skill
Open

Add refactoring-to-async skill#269
mrsharm wants to merge 16 commits intodotnet:mainfrom
mrsharm:musharm/refactoring-to-async-skill

Conversation

@mrsharm
Copy link
Member

@mrsharm mrsharm commented Mar 6, 2026

Adds the refactoring-to-async skill for bottom-up async conversion patterns in .NET.

Note: Replaces #79 (migrated from skills-old repo to new plugins/ structure).

What the Skill Teaches

  • Bottom-up async conversion order — start from leaf I/O methods, propagate upward
  • Correct sync-to-async API mappings (stream.Read → ReadAsync, connection.Open → OpenAsync, etc.)
  • CancellationToken propagation through the entire call chain
  • Identifying sync-over-async anti-patterns (.Result, .Wait(), .GetAwaiter().GetResult())
  • ConfigureAwait(false) usage in library code vs ASP.NET Core apps
  • Recognizing when async does not apply (CPU-bound work)Adds the refactoring-to-async skill for bottom-up async conversion patterns in .NET.

mrsharm added 2 commits March 6, 2026 09:22
Teaches bottom-up async conversion patterns: identifying sync-over-async,
CancellationToken propagation, ConfigureAwait usage, and ValueTask selection.

Eval results: +46.9% improvement over baseline (threshold: 10%)
Includes eval.yaml with positive scenario + negative test and fixture files.
@mrsharm
Copy link
Member Author

mrsharm commented Mar 6, 2026

Migration Note

This PR replaces #79 which was opened from mrsharm/skills-old. The skill and eval files have been migrated to the new plugins/ directory structure:

  • src/dotnet/skills/refactoring-to-async/plugins/dotnet/skills/refactoring-to-async/
  • src/dotnet/tests/refactoring-to-async/tests/dotnet/refactoring-to-async/

All prior review feedback from #79 still applies — please see that PR for the full discussion history.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new refactoring-to-async skill to the dotnet plugin, along with an evaluation scenario and fixture project intended to test bottom-up async conversion guidance.

Changes:

  • Added refactoring-to-async skill documentation under plugins/dotnet/skills/.
  • Added a new eval scenario plus C# fixture project under tests/dotnet/refactoring-to-async/.
  • Updated .github/CODEOWNERS to assign ownership for the new skill and test directory.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
plugins/dotnet/skills/refactoring-to-async/SKILL.md New skill content describing async refactoring workflow and pitfalls.
tests/dotnet/refactoring-to-async/eval.yaml New eval scenarios/assertions for async refactoring and CPU-bound non-async guidance.
tests/dotnet/refactoring-to-async/UserService.cs Fixture code containing sync I/O and blocking patterns to be refactored by the model.
tests/dotnet/refactoring-to-async/SyncService.csproj Fixture project enabling dotnet build-style validation.
.github/CODEOWNERS Adds code owners for the new skill and tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@mrsharm
Copy link
Member Author

mrsharm commented Mar 6, 2026

Feedback carried over from #79 (refactoring-to-async)

Code Review Comments (10 threads)

@copiloteval.yaml L12

The scenario requires identifying sync-over-async patterns like .Result/.Wait() (rubric line 14), but the output_not_matches assertion fails the run if the agent output contains .Result or .Wait() anywhere (including when calling them out as anti-patterns). This creates false negatives and makes the rubric effectively incompatible with the assertions. Consider tightening the regex to ...


@copilotSKILL.md L35

The grep pattern uses \b for a word-boundary, but grep's default regex syntax does not treat \b as a word boundary (itΓÇÖs typically interpreted as a backspace). Also, \.Read() only matches parameterless Read() calls and will miss the common Read(...) overloads. Consider switching to rg (ripgrep) or grep -P/grep -E with a corrected pattern (e.g., matching \.Read\( and `.W...


@copilotSKILL.md L155

Same issue as Step 1: grep without -P won't treat \b as a word boundary, so this command may not reliably find .Result usages. Adjust the command to use a regex flavor that supports word boundaries (or use \</\>), so the "should return zero results" guidance is accurate.

grep -rnP '\.Result\b|\.Wait\(\)|\.GetAwaiter\(\)\.GetResult\(\)' --include="*.cs" .

@danmoseleySKILL.md L8

move any part of use/not use into decription similar to
https://github.com/dotnet/runtime/pull/125005/changes#diff-ded9450821c1df27638def6250c00784c7f795e3a9c56ad13d09a34853a0d09bR3-R4
if it can avoid unnecessaryt reloading.

Possibly not all can go there. For example: user wants to parallelize work is something that can be known before loading the skill, and avoid load. Whereas possibly "code ...


@danmoseleySKILL.md L167

this is missing from all the examples. maybe indicate those are app code?

It might be worth mentioning as instructions, not just in anti-patterns -- if code is library add .ConfigureAwait(false) to every await. It should be used consistently or not at all.


@danmoseleySKILL.md L185

this may be a big high level/conceptual for this skill?


@danmoseleySKILL.md L53

| `reader.Read()` (DbDataReader) | `await reader.ReadAsync()` |
| `command.ExecuteNonQuery()` (DbCommand) | `await command.ExecuteNonQueryAsync()` |

maybe


@danmoseleyeval.yaml L25

is this really something the user would write? seems like a gimme as written. more likely the user would say "make DoIt() async so it's faster" and the AI would have to figure out that it's CPU bound?


@danmoseleyeval.yaml L33

do you expect ConfigureAwait(false) in this UserService case? either way, should be a test for the opposite case (eg winforms) and verify in each case it's present or not as expected


@danmoseleySKILL.md L148

| Error | Fix |
|---|---|
 Error | Fix |
   |---|---|
   | `CS4032`: `await` in non-async method | Add `async` to the method signature and return `Task` or `Task<T>` |
   | `CS0029`: Cannot convert `Task<T>` to `T` | Add `await` before the call |
   | `CS0127`: Method returns `Task` but body returns value | Change return type to `Task<T>` |
   | `CS1983`: Return type of async meth...

Discussion Comments (5)

@ViktorHofer:

Please share the results as a comment similar to #75 (comment)
Also make sure that you test at least 3 runs (--runs 3 for local validation): https://github.com/dotnet/skills/pull/82/changes


@mrsharm:

Skill Validation Results ΓÇö refactoring-to-async

Skill Test Baseline With Skill Δ Verdict
refactoring-to-async Refactor synchronous service to async 3.3/5 5.0/5 +1.7 ✅
refactoring-to-async Async refactoring should not apply to CPU-bound code 1.0/5 1.0/5 0.0 ✅

**Overall improvement: +18.8%...


@mrsharm:

Skill Validation Results ΓÇö refactoring-to-async

Skill Test Baseline With Skill Δ Verdict
refactoring-to-async Refactor synchronous service to async 3.3/5 5.0/5 +1.7 ✅
refactoring-to-async Async refactoring should not apply to CPU-bound code 1.0/5 1.0/5 0.0 ✅

**Overall improvement: +18.8%...


@mrsharm:

Please share the results as a comment similar to #75 (comment) Also make sure that you test at least 3 runs (--runs 3 for local validation): https://github.com/dotnet/skills/pull/82/changes

Done.


@danmoseley:

add codeowners entry, move files around to match new pattern in main.


mrsharm and others added 2 commits March 6, 2026 10:27
The 'should not apply to CPU-bound code' scenario was dropping agents into
an empty workspace, causing them to give up instead of engaging with the
problem. Now provides MatrixMultiplier.cs and .csproj so the agent has
actual code to optimize.
@ViktorHofer
Copy link
Member

/evaluate

@github-actions
Copy link
Contributor

github-actions bot commented Mar 6, 2026

Skill Validation Results

Skill Scenario Baseline With Skill Δ Skills Loaded Overfit Verdict
refactoring-to-async Refactor synchronous service to async 4.0/5 5.0/5 +1.0 ✅ refactoring-to-async; tools: skill, grep ✅ 0.10
refactoring-to-async Async refactoring should not apply to CPU-bound code 2.7/5 5.0/5 +2.3 ℹ️ not activated (expected) ✅ 0.10
refactoring-to-async Library code should use ConfigureAwait(false) 4.3/5 5.0/5 +0.7 ✅ refactoring-to-async; tools: skill, bash ✅ 0.10
refactoring-to-async Partial async conversion in large codebase 3.3/5 4.0/5 +0.7 ✅ refactoring-to-async; tools: skill ✅ 0.10

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results

SKILL.md changes:
- Move when-to-use/not-use into description for lazy-loading activation
- Clarify ConfigureAwait(false) guidance: app code vs library code
- Streamline decision tree section

eval.yaml changes:
- Fix CPU-bound negative scenario: inline MatrixMultiplier.cs fixture
- Make negative test prompt realistic (user doesn't state 'CPU-bound')
- Add ConfigureAwait(false) library code scenario
- Add partial async conversion scenario

Eval results (3 runs, 4 scenarios):
  Refactor sync to async:       3.0 -> 5.0 (+2.0) PASS
  CPU-bound negative:           2.0 -> 5.0 (+3.0) PASS
  ConfigureAwait library code:  4.0 -> 4.0 ( 0.0) FAIL
  Partial async conversion:     3.3 -> 3.7 (+0.4) PASS
  Overall: +33.8% (3/4 passed)
@dotnet dotnet deleted a comment from github-actions bot Mar 6, 2026
@dotnet dotnet deleted a comment from github-actions bot Mar 6, 2026
@mrsharm
Copy link
Member Author

mrsharm commented Mar 6, 2026

/evaluate

@dotnet dotnet deleted a comment from github-actions bot Mar 6, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 6, 2026

✅ Evaluation completed. View results | View workflow run

@mrsharm
Copy link
Member Author

mrsharm commented Mar 6, 2026

@danmoseley - could you please take another look? I believe I have addressed your feedback.

@ViktorHofer
Copy link
Member

@mrsharm check the one scenario in which the skill doesn't get activated.

@mrsharm
Copy link
Member Author

mrsharm commented Mar 6, 2026

@mrsharm check the one scenario in which the skill doesn't get activated.

@ViktorHofer: Intentional - is a negative test and is the correct outcome. The eval suggests optimizing matrix multiplication (CPU-bound), not about converting I/O to async. The skill's description targets I/O-bound async refactoring, so the agent should recognize this isn't an async problem and not load the skill. That's exactly what happened in all 3 runs.

@ViktorHofer
Copy link
Member

@mrsharm such tests must be annotated with expect_activation: false:

expect_activation: false

Otherwise we have no way to distinguish between intentionally not activated and unintentionally not activated.

@mrsharm
Copy link
Member Author

mrsharm commented Mar 8, 2026

/evaluate

@github-actions
Copy link
Contributor

github-actions bot commented Mar 8, 2026

✅ Evaluation completed. View results | View workflow run

@mrsharm
Copy link
Member Author

mrsharm commented Mar 8, 2026

@ViktorHofer - thanks! Been implemented for this and the other PRs.

@mrsharm mrsharm requested a review from Copilot March 10, 2026 19:16
@mrsharm mrsharm enabled auto-merge (squash) March 10, 2026 19:16
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@mrsharm
Copy link
Member Author

mrsharm commented Mar 10, 2026

/evaluate

@github-actions
Copy link
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
refactoring-to-async Refactor synchronous service to async 3.7/5 → 5.0/5 🟢 ✅ refactoring-to-async; tools: skill, grep ✅ 0.08
refactoring-to-async Async refactoring should not apply to CPU-bound code 2.3/5 → 5.0/5 🟢 ℹ️ not activated (expected) ✅ 0.08
refactoring-to-async Refactor sync-over-async in ASP.NET Core controller 3.7/5 → 4.3/5 🟢 ✅ refactoring-to-async; tools: skill, grep ✅ 0.08
refactoring-to-async Convert file I/O operations to async 3.0/5 → 5.0/5 🟢 ✅ refactoring-to-async; tools: skill ✅ 0.08
refactoring-to-async Partial async conversion in large codebase 3.7/5 → 4.0/5 🟢 ✅ refactoring-to-async; tools: skill ✅ 0.08

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results

Copilot AI review requested due to automatic review settings March 10, 2026 21:00
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +30 to +31
/plugins/dotnet-upgrade/skills/migrate-nullable-references/ @danmoseley
/tests/dotnet-upgrade/migrate-nullable-references/ @danmoseley
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are now duplicate CODEOWNERS entries for the exact same migrate-nullable-references paths (lines 30-33). Since CODEOWNERS applies the last matching rule, the earlier two lines are redundant and can confuse future edits—please remove the duplicates or consolidate into a single pair of entries.

Suggested change
/plugins/dotnet-upgrade/skills/migrate-nullable-references/ @danmoseley
/tests/dotnet-upgrade/migrate-nullable-references/ @danmoseley

Copilot uses AI. Check for mistakes.
Removed duplicate CODEOWNERS entries for nullable references.
fixing thread pool starvation from .Result/.Wait()/.GetAwaiter().GetResult(),
modernizing sync-over-async code, adding CancellationToken support.
DO NOT USE FOR: CPU-bound computation (use Parallel.For or Task.Run instead),
code with no I/O operations, parallelizing work rather than making it async.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The budget for skills descriptions is limited which is why they are intended to be brief. This seems longer than I would expect for a skill.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only validate the hard limit of 1024 chars. Beyond that, it's hard to score whether there are too many (waste) or appropriate (prevents loading the skill inappropriately). If anything evaluation may drive to more tokens (in order to force skill load when it should). I don't know whether human intuition can help much here or not.


- name: "Refactor sync-over-async in ASP.NET Core controller"
prompt: |
This ASP.NET Core controller has performance issues under load. Some methods use .GetAwaiter().GetResult() and others mix sync and async patterns. Can you fix the async anti-patterns and convert it properly?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry that the prompt here is too strongly connected to the prompts in the skill. Basically the skill says "for ASP.NET do X" and the eval say "this is ASP.NET". It feels like the test is being fitted for the skill vs. the user experience. My intuition is that customers would more naturally write

This controller has performance issues under load ...

The model would then need to infer what type of code this was to decide the right action from the skill.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I agree this is arguably "overfitting". Current overfitting judge doesn't catch it. @JanKrivanek I wonder whether this suggests we should tune prompt given to the overfitting judge.


public User GetById(int id)
{
using var connection = new SqlConnection(_connectionString);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should the skill recommend using IAsyncDisposable (ie await using) ? and eval should check that all these using var get converted.

| `task.Result` or `task.Wait()` | Blocks thread, risks deadlock | `await task` |
| `async void` methods | Exceptions crash the process | `async Task` (except event handlers) |
| `Task.Run` wrapping async I/O | Wastes a thread pool thread | Call async method directly |
| Missing `ConfigureAwait(false)` in libraries | Can deadlock in UI/ASP.NET sync contexts | Add `.ConfigureAwait(false)` to every `await` in library code; omit in ASP.NET Core app code (no SynchronizationContext) |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Missing `ConfigureAwait(false)` in libraries | Can deadlock in UI/ASP.NET sync contexts | Add `.ConfigureAwait(false)` to every `await` in library code; omit in ASP.NET Core app code (no SynchronizationContext) |
| Missing `ConfigureAwait(false)` in libraries | Can deadlock in Winforms/WPF/ASP.NET sync contexts | Add `.ConfigureAwait(false)` to every `await` in library code; omit in ASP.NET Core app code (no SynchronizationContext) |

- "Converted controller action methods to async Task<ActionResult<T>>"
- "Replaced .GetAwaiter().GetResult(), .Wait(), and .Result with await"
- "Added CancellationToken parameter to async actions to support request cancellation"
- "Used Task.WhenAll or sequential awaits instead of blocking .Result in the loop"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI says, "Task.WhenAll fires all HTTP requests concurrently, which could overwhelm a downstream service. Sequential await should be the preferred default; WhenAll should only be suggested with a concurrency caveat."

| Performance worse after conversion | Async adds overhead for CPU-bound work; only use for I/O |
| Forgetting to update tests | Test methods must return `Task` and use `await` |
| Breaking interface consumers | Consider keeping sync wrappers temporarily during staged migration |
| `ValueTask` vs `Task` confusion | Use `Task` by default; `ValueTask` only for hot-path methods that frequently return synchronously |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a bit more mention of ValueTask in the guidance higher up and its limitations over Task. Eg., cannot be awaited multiple times, cannot be stored.

```

This should return zero results in the refactored code paths.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is IAsyncEnumerable / await foreach in scope? Probably should be. If not, it should say it's out of scope. Similar for IAsyncDisposable

Search for synchronous I/O patterns in the codebase:

```bash
grep -rnE '\.Result\b|\.Wait\(\)|\.GetAwaiter\(\)\.GetResult\(\)|ReadToEnd\(\)|\.Read\(|\.Write\(' --include='*.cs' .
Copy link
Member

@danmoseley danmoseley Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest to provide a list of example method names, rather than write the grep for it. It's great at writing grep, and giving it the command suggests that this is a complete list. Plus if this hits something inappropriate eg a random property named "Result" it can adjust itself.

Copilot AI review requested due to automatic review settings March 11, 2026 09:13
@ViktorHofer
Copy link
Member

/evaluate

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- type: "output_matches"
pattern: "(async Task<ActionResult|async Task<IActionResult)"
- type: "output_matches"
pattern: "(CancellationToken|cancellation)"
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The controller scenario rubric requires removing sync-over-async anti-patterns (.Result/.Wait()/.GetAwaiter().GetResult()), but the assertions only check for an async signature + CancellationToken. Add an output_not_matches assertion (similar to scenario 1) to prevent false passes where the response keeps/mentions these blocking calls.

Suggested change
pattern: "(CancellationToken|cancellation)"
pattern: "(CancellationToken|cancellation)"
- type: "output_not_matches"
pattern: "\\.Result\\b(?=\\s*[);])|\\.Wait\\(\\)(?=\\s*[);])|\\.GetAwaiter\\(\\)\\.GetResult\\(\\)"

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +10
---
name: refactoring-to-async
description: >
Convert synchronous .NET code to async/await, including proper Task propagation,
cancellation support, and avoiding common async anti-patterns.
USE FOR: converting blocking I/O calls (database, HTTP, file, stream) to async,
fixing thread pool starvation from .Result/.Wait()/.GetAwaiter().GetResult(),
modernizing sync-over-async code, adding CancellationToken support.
DO NOT USE FOR: CPU-bound computation (use Parallel.For or Task.Run instead),
code with no I/O operations, parallelizing work rather than making it async.
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description text appears to contain a duplicated sentence at the end ("Adds the refactoring-to-async skill..."). Consider cleaning that up so the description reads cleanly for reviewers and release notes.

Copilot uses AI. Check for mistakes.
@github-actions
Copy link
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
refactoring-to-async Refactor synchronous service to async 3.7/5 → 5.0/5 🟢 ✅ refactoring-to-async; tools: skill ✅ 0.10
refactoring-to-async Async refactoring should not apply to CPU-bound code 2.0/5 → 5.0/5 🟢 ℹ️ not activated (expected) ✅ 0.10
refactoring-to-async Refactor sync-over-async in ASP.NET Core controller 3.3/5 → 5.0/5 🟢 ✅ refactoring-to-async; tools: skill ✅ 0.10
refactoring-to-async Convert file I/O operations to async 3.0/5 → 5.0/5 🟢 ✅ refactoring-to-async; tools: skill ✅ 0.10
refactoring-to-async Partial async conversion in large codebase 4.0/5 → 4.0/5 ✅ refactoring-to-async; tools: skill ✅ 0.10

Model: claude-opus-4.6 | Judge: claude-opus-4.6

Full results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants