Invoke‑ShpEval

SYNOPSIS

Runs deterministic agent-behavior evaluation cases and reports their pass@1 and pass^k reliability.

SYNTAX

Invoke-ShpEval [-Case] <Hashtable[]> [-Trial <Int32>] [-Name <String[]>] [-Tag <String[]>] [-IncludeLiveCanary]
 [<CommonParameters>]

DESCRIPTION

Grades agent behavior the way the rest of this module grades anything else: as data, in the ordinary test gate, without a model.

A case supplies its own body. This cmdlet never calls a model, reads a credential, or sends a request - it runs what the case gives it, records what happened, and grades it. In practice a deterministic case drives the real Invoke-Shp Tool-calling loop through a scripted -RequestTransport, so the same inputs produce the same outputs on every run and the whole surface can live beside the unit suite instead of in a separate credentialed job.

Two things are graded, and they are not the same claim. The observable OUTCOME is what the caller got back. The Tool-call TRAJECTORY is what the agent did to get there. An agent that reaches the right answer by running a command it was forbidden has failed, and only the trajectory grade says so.

RELIABILITY IS REPORTED TWICE, on purpose. -Trial runs each case k times. PassAt1 is the fraction of trials that passed, which is what a single attempt is worth. PassPowK is 1 only when EVERY trial passed, which is what "this behavior is reliable" means. A case at 0.8 pass@1 and 0 pass^k is a case that mostly works, and reporting only the average would hide the word "mostly".

A case fails, it does not throw. The run grades every case and reports them together, because the second failure is as interesting as the first. A case this cmdlet cannot understand - no name, no body, no expectation, an unknown member - IS an error, because a case that silently does nothing is worse than no case at all.

LIVE CANARIES ARE NOT PART OF THIS. A case marked Mode = 'LiveCanary' is skipped unless -IncludeLiveCanary is passed, so a credentialed run against a real provider stays out of the deterministic gate by default and is listed as skipped rather than quietly absent.

EXAMPLES

EXAMPLE 1

Invoke-ShpEval -Case $cases -Trial 5

Runs every deterministic case five times and reports pass@1 and pass^k per case and for the run.

EXAMPLE 2

(Invoke-ShpEval -Case $cases -Tag 'prompt-injection').FailedCaseCount

Runs only the prompt-injection cases and reports how many failed.

PARAMETERS

-Case

The evaluation cases. Each is a hashtable:

Name              Required, unique within the run.
Tag               Optional labels for -Tag selection.
Mode              'Deterministic' (default) or 'LiveCanary'.
Setup             Optional scriptblock run before each trial.
Invoke            Required scriptblock; receives the trial number
                  and returns the run to grade.
Teardown          Optional scriptblock run after each trial, even
                  when the trial failed.
ExpectOutcome     Outcome graders (see Test-ShpEvalExpectation).
ExpectTrajectory  Trajectory graders.
MaxCostUSD        Spend ceiling for the run.

At least one of ExpectOutcome, ExpectTrajectory or MaxCostUSD is required: a case that grades nothing cannot fail.

Type: Hashtable[]
Parameter Sets: (All)
Aliases:

Required: True
Position: 1
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-IncludeLiveCanary

Also run cases marked Mode = 'LiveCanary'. They are skipped otherwise.

Type: SwitchParameter
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: False
Accept pipeline input: False
Accept wildcard characters: False

-Name

Run only the cases with these exact names.

Type: String[]
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-Tag

Run only the cases carrying at least one of these tags.

Type: String[]
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-Trial

How many times to run each case. Default 1.

Type: Int32
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: 1
Accept pipeline input: False
Accept wildcard characters: False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

INPUTS

OUTPUTS

System.Management.Automation.PSCustomObject

A ShellPilot.EvalReport with SchemaVersion, TrialCount, CaseCount,

PassedCaseCount, FailedCaseCount, SkippedCaseCount, SkippedCase,

PassAt1, PassPowK and the per-case results.

NOTES

RELATED LINKS

Invoke-Shp

Set-ShpToolPolicy