New-Package metadata review with GitHub Copilot

[Demitrius Nelon (denelon)]

New-Package metadata review with GitHub Copilot

This discussion collects feedback and evaluation results for automatically requesting GitHub Copilot code review on deterministically classified New-Package pull requests.

The proposed design intentionally avoids another agent-authored bot comment.
Current investigation favors a two-stage flow:

1. Collect deterministic evidence when the PR is submitted or classified as a
   New-Package submission.
2. Wait for validation labels before deciding whether a semantic Copilot review
   is still useful.

For clean New-Package PRs, the likely trigger is the combination of
New-Package and Validation-Completed. Failure labels should route to their
specific diagnostic workflows instead of also triggering a broad metadata
review.

What the review checks

Only non-omitted metadata in the changed default-locale manifest:

• ShortDescription and Description accuracy
• License claims against explicit publisher evidence
• ReleaseNotes accuracy for the submitted version
• ReleaseNotesUrl and Documentation product/version alignment

What it must not report

• Missing optional fields
• Accurate paraphrases
• Secondary features omitted from a truthful short description
• SPDX or stylistic preferences
• Claims that cannot be verified from publisher-controlled sources
• Generic release-index URLs that remain relevant to the package

Pilot lesson

Ten merged New-Package PRs produced mostly correct metadata, two unverifiable cases, and one likely actionable contradiction. Several plausible findings disappeared when the analysis was corrected to use the exact PR head rather than the repository's later state. Precision and source provenance therefore matter more than review volume.

Label timing provides a natural efficiency experiment: New-Package
classification usually appeared around 15 minutes after submission, while a
clean or failed validation outcome generally appeared around 50-150 minutes
later. PPE should compare an early-review cohort with a deferred-review cohort
and measure how often early reviews become stale after another author push.

Waiver metadata can be inspected during preflight, but .validation entries are
version- and commit-scoped and normally arrive through a separate wingetbot PR.
Their presence is evidence, not an automatic waiver for the submitted changes.

Tracking issue: #411358

Feedback requested:

1. Should clean metadata review wait for Validation-Completed, or is the
   roughly 35-135 minute earlier start worth the risk of a stale review?
2. Which submission-time signals should be retained for later failure-label
   workflows?
3. Should a new push request a fresh review only when locale metadata changed?
4. Are there additional publisher-controlled sources that should be trusted?
5. What false-positive rate is acceptable before production rollout?

Created with GitHub Copilot assistance.

[Demitrius Nelon (denelon)]

PPE A/B experiment: interim results

The experiment was designed to compare:

1. Copilot review requested immediately for an early cohort.
2. Copilot review requested only after classification plus Validation-Completed for a deferred cohort.

Four exact production PR snapshots were submitted to PPE. After draft-to-ready PRs did not enter validation, they were replaced with PRs opened ready:

Cohort	Package	Outcome
Early	MeldStudio.MeldStudio	Copilot reviewed in about 3.5 minutes; one installer-modeling finding, but no assessment of the suspected License contradiction
Early	EmanueleMicheletti.Proxelar	Copilot reviewed in about 2.5 minutes; three repetitive schema/checklist findings unrelated to publisher metadata
Deferred	Code-iX.QTop	Review not requested because validation labels never appeared
Deferred	SIMSDEV.AndroidAppsManager	Review not requested because validation labels never appeared

None of the replacement PRs registered with wingetvalidator-ppe or received New-Package, New-Manifest, failure, or Validation-Completed labels after more than three hours. The timing comparison is therefore incomplete.

What we learned

Generic Copilot review is too broad. The early reviews focused on installer structure and schema/checklist consistency. They did not perform the requested comparison of supplied metadata against authoritative publisher sources.

Fast review is not the same as useful review. Both early reviews arrived within minutes, but their findings would add noise to a metadata-specific workflow. The Proxelar comments were also duplicative across all three manifest files.

The deferred design remains preferable. Nothing in this run justifies spending a review before deterministic validation completes. Waiting for Validation-Completed avoids reviewing PRs already headed into a failure-specific workflow and provides time to collect exact-head evidence silently.

The experiment itself must avoid contaminating the result. The copied PR template asserted schema 1.12 conformance while one control used 1.9 manifests, creating an unrelated review target.

Proposed next iteration

1. Use a PPE submission path known to register with the validator, preferably the same internal-copy mechanism used by prior PPE samples.
2. Supply explicit metadata-review instructions that limit findings to ShortDescription, Description, License, ReleaseNotes, ReleaseNotesUrl, and Documentations.
3. Require authoritative publisher evidence and no finding when evidence is unavailable or ambiguous.
4. Request review at most once per head commit, only after Validation-Completed plus New-Package or New-Manifest.
5. Route failure labels to their specific diagnostic workflows instead of also requesting broad metadata review.