Adding in the ability to add a git worktree into the created workspace for a test

[richardpark-msft]

Adding in the ability to add a git worktree into a workspace created for a copilot run. This is most useful for people that are developing skills within a repo they're also using the skills - this lets them get a clean copy of the repo for each test run. See 'internal/models/testdata/git-resources-task-example.yaml' for an idea of what the YAML looks like to use it.

Future plans are to add in more git clone methods, but this is a nice cheap way of boostrapping a clean copy of a repo, when working locally.

We also now validate that the task configuration is valid before starting - prior to this we were apparently just throwing out log messages but no errors, so a user could start with a partially invalid configuration and not realize that critical checks, etc... weren't even being loaded.

Fixes #121

[richardpark-msft]

(just a note, the coding is done, we have tests in place, but I want to see if I can add some more E2E tests before moving this into final review)

[Copilot]

Pull request overview

This PR adds first-class “git resources” to task inputs so a test run can materialize a clean copy of a local repository (via git worktree) inside the per-run workspace, and tightens up task input validation so invalid file/resource config fails fast instead of only warning.

Changes:

• Extend the task schema and models to support inputs.repos (git worktree resources) and stricter inputs.files validation.
• Introduce ExecutableTestCase to pre-validate/materialize file and git resources before execution, and plumb git resources through to the execution engine(s).
• Add execution-layer git worktree support plus tests and docs updates.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
schemas/task.schema.json	Adds inputs.repos and tightens resourceRef validation (needs workdir added + stricter gitResource def).
internal/orchestration/runner.go	Adds ExecutableTestCase, preloads resources, and passes git resources/workdir into execution requests.
internal/orchestration/runner_test.go	Updates runner unit tests to use ExecutableTestCase.
internal/orchestration/runner_orchestration_test.go	Updates resource-loading tests for new (resources, gitResources, err) signature.
internal/orchestration/filter.go	Updates filtering helpers to operate on ExecutableTestCase.
internal/orchestration/filter_test.go	Updates filter tests for ExecutableTestCase.
internal/models/testcase.go	Adds git resource types and ResourceRef.Validate (missing WorkDir field currently blocks compilation).
internal/models/testcase_test.go	Adds tests for git resources, workdir, and ResourceRef validation; refactors trigger YAMLs into files.
internal/models/testdata/*	Adds example task YAMLs for git resources, file resources, triggers, and workdir.
internal/execution/engine.go	Adds GitResources + WorkDir to ExecutionRequest and introduces ResolveWorkDir.
internal/execution/copilot.go	Plumbs git resources into workspace setup and adds cleanup tracking (but still doesn’t apply WorkDir).
internal/execution/mock.go	Adds git resource materialization/cleanup in mock engine (but changes WorkspaceDir semantics incorrectly).
internal/execution/git.go	Implements git worktree materialization and cleanup helpers.
internal/execution/git_test.go	Adds tests for git worktree behavior and workdir resolution (needs git author config for reliability).
internal/execution/generate.go	Adds mock generation for GitResource.
internal/execution/execution_mocks.go	Generated GoMock for GitResource.
internal/execution/engine_shutdown_test.go	Extends shutdown tests to ensure git resources are cleaned up.
README.md	Documents git resources + workdir usage (cleanup timing is currently misstated).

Comments suppressed due to low confidence (2)

schemas/task.schema.json:97

• Task YAML/examples now use inputs.workdir, but the task schema doesn’t define a workdir property under inputs. Add it (as a string) so schema validation and editor tooling match the documented behavior.

      "properties": {
        "prompt": {
          "type": "string",
          "description": "The prompt message sent to the agent."
        },
        "context": {
          "type": "object",
          "description": "Arbitrary key-value metadata passed to the agent."
        },
        "files": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/resourceRef"
          },
          "description": "File references or inline content provided to the agent."
        },
        "repos": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/gitResource"
          },
          "description": "Git repositories provided to the agent workspace."
        },
        "environment": {
          "type": "object",
          "additionalProperties": {
            "type": "string"
          },
          "description": "Environment variables set during task execution."
        }

internal/models/testcase.go:33

• TestStimulus is missing a WorkDir field, but the new YAML examples/tests (and runner.go via tc.Stimulus.WorkDir) rely on it. This will not compile and workdir: in task YAML will be ignored until a WorkDir string yaml:"workdir,omitempty"`` field is added to TestStimulus.

// TestStimulus defines the input for a test
type TestStimulus struct {
	Message     string            `yaml:"prompt" json:"message"`
	Metadata    map[string]any    `yaml:"context,omitempty" json:"metadata,omitempty"`
	Resources   []ResourceRef     `yaml:"files,omitempty" json:"resources,omitempty"`
	Repos       []GitResource     `yaml:"repos,omitempty" json:"repos,omitempty"`
	Environment map[string]string `yaml:"environment,omitempty" json:"environment,omitempty"`
}

[Copilot]

$defs.gitResource currently allows unknown keys because it lacks "additionalProperties": false (many other object defs in this schema are strict). Consider adding additionalProperties:false and (optionally) minLength: 1 for required strings like source to catch typos early.

[Copilot]

doShutdown cleans up git resources using the caller-provided ctx. If Shutdown is invoked with a canceled/deadline-exceeded context, git cleanup will be skipped/aborted, leaving stale worktrees behind. Consider using a background/best-effort context (or a fresh timeout) for cleanup operations.

[Copilot]

If CloneGitResources fails, setupWorkspace returns without cleaning up the newly created temp workspace (already appended to e.workspaces) and without cleaning any partially-created git resources. Consider cleaning up immediately on error (and/or have CloneGitResources rollback created resources) to avoid leaking temp dirs/worktrees when Execute fails early.

[Copilot]

m.gitResources is appended to on each Execute, but prior worktrees are never cleaned up when a new workspace replaces the old one (the old workspace dir is deleted earlier, but the source repo will still track the worktree). Consider cleaning up any existing m.gitResources before creating/materializing new git resources, and reset the slice per-execute.

[Copilot]

MockEngine sets ExecutionResponse.WorkspaceDir to the resolved workdir (agentCwd). WorkspaceDir is used by graders/tests as the workspace root; changing it breaks existing expectations (e.g., tests that read resp.WorkspaceDir/fixtures/...). Keep WorkspaceDir as the workspace root and, if needed, introduce a separate field for the agent CWD.

[Copilot]

CloneGitResource joins workspaceDir with gitRes.Dest without validating that Dest stays within the workspace (e.g., dest: "../../.." would write outside the sandbox). Please validate Dest is relative/non-traversing (similar to ResolveWorkDir) before constructing targetDir.

Suggested change

	targetDir = filepath.Join(workspaceDir, gitRes.Dest)
	cleanDest := filepath.Clean(gitRes.Dest)
	// Ensure destination is a relative, non-traversing path within the workspace.
	if filepath.IsAbs(cleanDest) ||
	cleanDest == ".." ||
	strings.HasPrefix(cleanDest, ".."+string(os.PathSeparator)) {
	return nil, fmt.Errorf("invalid git destination path %q", gitRes.Dest)
	}
	targetDir = filepath.Join(workspaceDir, cleanDest)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 67.35751% with 63 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@52e123f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
internal/orchestration/runner.go	55.42%	31 Missing and 6 partials ⚠️
internal/execution/copilot.go	54.54%	6 Missing and 4 partials ⚠️
internal/execution/git.go	82.60%	6 Missing and 2 partials ⚠️
internal/execution/mock.go	37.50%	3 Missing and 2 partials ⚠️
internal/models/testcase.go	62.50%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #154   +/-   ##
=======================================
  Coverage        ?   73.42%           
=======================================
  Files           ?      143           
  Lines           ?    15999           
  Branches        ?        0           
=======================================
  Hits            ?    11747           
  Misses          ?     3394           
  Partials        ?      858           

Flag	Coverage Δ
go-implementation	73.42% <67.35%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 11 comments.

Comments suppressed due to low confidence (1)

internal/execution/git.go:78

• git worktree add <path> <commit> will check out a branch if commit is a branch name (e.g. main). That commonly fails because the branch is already checked out in the source repo (“already checked out at ...”). Since the schema/docs allow branch/tag names, consider always using --detach when a commit is provided (i.e., git worktree add --detach <path> <commit-ish>) so branch names work reliably without conflicts.

// gitWorkTreeAdd runs 'git worktree add', creating a git worktree (an incredibly cheap copy) of a local repo to
// another local path on disk. Note, this requires a local clone of a git repo to work.
func gitWorkTreeAdd(ctx context.Context, commit string, repoDir string, targetDir string) (*GitWorkTree, error) {
	args := []string{"worktree", "add"}

	if commit != "" {
		// git worktree add <path> <commit>
		args = append(args, targetDir, commit)
	} else {
		// git worktree add --detach <path>  (uses HEAD)
		args = append(args, "--detach", targetDir)
	}

	if _, err := runGitCommand(ctx, repoDir, args...); err != nil {
		return nil, fmt.Errorf("git worktree add failed: %w", err)
	}

[Copilot]

When gitRes.Dest is empty, targetDir is set to workspaceDir, but setupWorkspace() creates that directory ahead of time. git worktree add <path> fails if the target directory already exists, so the documented “omit dest to use workspace root” behavior won’t work. Consider creating the workspace as a parent dir and adding the worktree into a non-existent child dir, or removing/renaming the pre-created workspace dir before running git worktree add (and updating the returned workspace root accordingly).

This issue also appears on line 63 of the same file.

[Copilot]

sourceDir is computed/validated, but the original repo (with the original repo.Source) is what gets appended to gitResources. If repo.Source is relative, it will later be interpreted relative to the process working directory in execution.CloneGitResource, not relative to the task file path used for validation. Either normalize repo.Source (e.g., assign repo.Source = sourceDir) before appending, or ensure CloneGitResource resolves relative sources consistently.

Suggested change

	} else {
	} else {
	// Normalize the repo source to the validated, task-relative directory
	repo.Source = sourceDir

[Copilot]

On CloneGitResources error, this returns immediately without cleaning up the temp workspace directory and without cleaning up any worktrees created before the failure. This can leak both the workspace dir and git worktrees until engine shutdown (and in the worktree case, may persist in the source repo if the process exits). Consider doing best-effort cleanup (worktrees + workspace) before returning the error, and only appending the workspace to e.workspaces after setup succeeds.

See below for a potential fix:

func (e *CopilotEngine) setupWorkspace(ctx context.Context, resources []ResourceFile, gitResources []models.GitResource) (workspaceDir string, err error) {
	workspaceDir, err = os.MkdirTemp("", "waza-*")
	if err != nil {
		return "", fmt.Errorf("failed to create temp workspace: %w", err)
	}

	// Track any git resources created during setup so we can best-effort
	// clean them up if a subsequent step fails.
	var wts []GitResource

	// Best-effort cleanup on failure: remove any git resources created so far
	// and delete the temporary workspace directory.
	defer func() {
		if err != nil {
			for _, wt := range wts {
				if cerr := wt.Cleanup(ctx); cerr != nil {
					slog.Warn("failed to cleanup git resource after setup failure", "error", cerr)
				}
			}
			if rmErr := os.RemoveAll(workspaceDir); rmErr != nil {
				slog.Warn("failed to remove workspace after setup failure", "dir", workspaceDir, "error", rmErr)
			}
		}
	}()

	// Write resource files to workspace
	if err = setupWorkspaceResources(workspaceDir, resources); err != nil {
		err = fmt.Errorf("failed to setup resources at workspace %s: %w", workspaceDir, err)
		return "", err
	}

	wts, err = CloneGitResources(ctx, gitResources, workspaceDir)
	if err != nil {
		return "", err
	}

	if len(wts) > 0 {
		e.resourcesMu.Lock()
		e.gitResources = append(e.gitResources, wts...)
		e.resourcesMu.Unlock()
	}

	e.resourcesMu.Lock()
	e.workspaces = append(e.workspaces, workspaceDir)
	e.resourcesMu.Unlock()

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.

[Copilot]

When commit is non-empty, gitWorkTreeAdd runs git worktree add <path> <commit>, which will fail if <commit> is a branch currently checked out in the main worktree (common case when users specify commit: main). To support branch/tag names as documented, consider using --detach even when a commit-ish is provided (e.g., git worktree add --detach <path> <commit>), or otherwise ensure the branch-checkout conflict is handled.

[Copilot]

TestResolveWorkDir hardcodes POSIX-style expected paths (e.g. /workspace/my-repo). Since CI runs tests on Windows too, filepath.Join will produce Windows separators and these assertions will fail. Build want values using filepath.Join (or normalize path separators before comparing) to keep the test cross-platform.

[Copilot]

README documents an inputs.workdir field, but there is no corresponding field in the task schema (schemas/task.schema.json) or model (internal/models/testcase.go), and it isn’t wired through ExecutionRequest. As written, users can’t actually configure workdir despite the docs. Either remove/adjust this section, or implement and document the end-to-end support for workdir consistently (schema + models + runner/engine).

[Copilot]

ResourceRef.Validate allows content without a path, but setupWorkspaceResources ignores resource files with an empty Path, so this input will be silently dropped. If inline content is meant to create a file in the workspace, validation (and the JSON schema) should require path when content is provided, or alternatively treat a missing path as an error during resource loading.

Suggested change

	// Validate checks that at least one file reference field is specified.
	func (r *ResourceRef) Validate() error {
	if r.Location == "" && r.Body == "" {
	return fmt.Errorf("resource must specify one of: path or content")
	}
	// Validate checks that a valid file reference is specified.
	func (r *ResourceRef) Validate() error {
	if r.Location == "" && r.Body == "" {
	return fmt.Errorf("resource must specify one of: path or content")
	}
	if r.Location == "" && r.Body != "" {
	return fmt.Errorf("resource must specify path when content is provided")
	}

[Copilot]

The comment on GitResource.Source says that if it’s empty it "uses the current directory", but the task schema requires source for worktree repos and the orchestration code resolves/validates it differently. Please align the comment with actual supported behavior (either make source truly optional, or update the comment/docs/schema to reflect it’s required).

Suggested change

	// - For 'worktree', Source is the folder where the git repository resides. If empty, uses the current directory.
	// - For 'worktree', Source is the folder where the git repository resides. This is required and must not be empty.

[Copilot]

The comment on mustCreateRepo says it creates test.txt, but the helper actually writes hello.txt. This can confuse future maintenance/debugging of these tests; please update the comment to match the file that’s created.

[richardpark-msft]

Going to take another look at something, I think this PR might be too general and it could be more targeted.