Skip to content

Make verify tool accessible to the coding agent #9753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 8, 2025
Merged

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jun 7, 2025

This PR adds the verify tool to the copilot setup workflow and provides documentation for snapshot testing usage.

Changes Made

  1. Added verify tool installation to copilot setup workflow (.github/workflows/copilot-setup-steps.yml):

    • Added a step to install verify.tool globally before the build step
    • This ensures the verify tool is available to coding agents during their sessions
  2. Updated copilot instructions (.github/copilot-instructions.md):

    • Added a new "Snapshot Testing with Verify" section
    • Documented that the repository uses Verify.XunitV3 for snapshot testing
    • Explained where snapshot files are stored (Snapshots directories)
    • Provided instructions to use dotnet verify accept -y to accept pending snapshot changes
    • Noted that the verify tool is available globally as part of the setup

Background

The repository already uses the Verify library (Verify.XunitV3 version 30.3.0) extensively for snapshot testing, with verified snapshot files in multiple test projects including:

  • tests/Aspire.Hosting.Azure.Tests/Snapshots/
  • tests/Aspire.Hosting.Docker.Tests/Snapshots/
  • tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/

When coding agents modify tests that use snapshot testing, they need a way to accept the new snapshots. The verify tool provides this capability through the dotnet verify accept -y command.

Fixes #9621.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Make verify tool accessible to the coding agent Make verify tool accessible to the coding agent Jun 7, 2025
@Copilot Copilot AI requested a review from davidfowl June 7, 2025 07:38
Copilot finished work on behalf of davidfowl June 7, 2025 07:38
@davidfowl davidfowl marked this pull request as ready for review June 7, 2025 07:46
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR integrates the Verify snapshot tool into the Copilot setup workflow and documents how to use it for accepting snapshot updates.

  • Installs the verify.tool global .NET tool in the CI setup.
  • Adds a “Snapshot Testing with Verify” section to the Copilot instructions.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/copilot-setup-steps.yml Added a step to install the Verify global tool
.github/copilot-instructions.md Documented snapshot testing usage with dotnet verify

# Install verify tool for snapshot testing
- name: Install verify tool
run: dotnet tool install -g verify.tool

# Include PrepareForHelix to maximise what is downloaded here
Copy link
Preview

Copilot AI Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After installing a global .NET tool, you need to add its install directory ($HOME/.dotnet/tools) to the PATH. Add a step like:

- name: Add .NET tools to PATH
  run: echo "$HOME/.dotnet/tools" >> $GITHUB_PATH
Suggested change
# Include PrepareForHelix to maximise what is downloaded here
- name: Add .NET tools to PATH
run: echo "$HOME/.dotnet/tools" >> $GITHUB_PATH
# Include PrepareForHelix to maximise what is downloaded here

Copilot uses AI. Check for mistakes.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@danmoseley danmoseley merged commit 16e3a4a into main Jun 8, 2025
493 of 496 checks passed
@danmoseley danmoseley deleted the copilot/fix-9621 branch June 8, 2025 02:51
@radical
Copy link
Member

radical commented Jun 9, 2025

Run dotnet tool install --global Verify.Tool --version 0.6.0
The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application 'tool' does not exist.
  * You intended to execute a .NET SDK command:
      A compatible .NET SDK was not found.

Requested SDK version: 9.0.200
global.json file: /home/runner/work/aspire/aspire/global.json

Installed SDKs:

Install the [9.0.200] .NET SDK or update [/home/runner/work/aspire/aspire/global.json] to match an installed SDK.

Learn about SDK resolution:
https://aka.ms/dotnet/sdk-not-found
8.0.116 [/usr/lib/dotnet/sdk]

@danmoseley this is failing copilot setup - https://github.com/dotnet/aspire/actions/runs/15542423162/job/43756124976

@davidfowl
Copy link
Member

hmmmm

@radical
Copy link
Member

radical commented Jun 9, 2025

We can run this after the build step using dotnet.sh? Where would that install the tool though, and would the future dotnet invocation be able to access it?

davidfowl added a commit that referenced this pull request Jun 9, 2025
davidfowl added a commit that referenced this pull request Jun 9, 2025
radical added a commit that referenced this pull request Jun 9, 2025
@danmoseley
Copy link
Member

I should have done a test copilot setup before merging this. What's more, copilot review suggested the problem..

@github-actions github-actions bot locked and limited conversation to collaborators Jul 11, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Error loading sessions

Retrying..

Successfully merging this pull request may close these issues.

Make verify tool accessible to the coding agent
4 participants