wfctl plugin conformance: artifact mode binary discovery#650
Merged
Conversation
⏱ Benchmark Results✅ No significant performance regressions detected. benchstat comparison (baseline → PR)
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Agent-Logs-Url: https://github.com/GoCodeAlone/workflow/sessions/166c7b9e-b235-4c39-9e1d-e731b68072f7 Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Agent-Logs-Url: https://github.com/GoCodeAlone/workflow/sessions/166c7b9e-b235-4c39-9e1d-e731b68072f7 Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix wfctl plugin conformance artifact mode to discover release binaries
wfctl plugin conformance: artifact mode binary discovery
May 13, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Improves wfctl plugin conformance artifact-mode usability by discovering and validating plugin binaries under multiple candidate names (install name vs manifest/project name), then emitting clearer evidence/diagnostics when selection or handshake fails—aimed at supporting common GoReleaser archive naming.
Changes:
- Add multi-candidate artifact binary discovery + per-candidate handshake attempts, with diagnostics included in evidence output.
- Expand conformance tests with helpers for building fixture binaries and constructing binary-only tar.gz archives; add coverage for discovery ordering and failure-evidence behavior.
- Document the new artifact binary discovery/handshake behavior in
docs/WFCTL.md.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
cmd/wfctl/plugin_conformance.go |
Implements multi-candidate artifact binary discovery and enhanced evidence diagnostics for artifact conformance. |
cmd/wfctl/plugin_conformance_test.go |
Adds helpers and new unit/integration tests validating candidate ordering, selection by manifest name, and failure evidence. |
docs/WFCTL.md |
Documents artifact-mode multi-candidate discovery and handshake verification behavior. |
plugin/external/sdk/iacserver_test.go |
Minor formatting change for an embedded stub type in tests. |
cmd/wfctl/deploy_providers.go |
Minor struct field alignment/formatting change. |
Comment on lines
+407
to
+435
| // Scan the archive root for any additional executables not already considered. | ||
| entries, err := os.ReadDir(sourceDir) | ||
| if err != nil { | ||
| return out | ||
| } | ||
| for _, entry := range entries { | ||
| if entry.IsDir() { | ||
| continue | ||
| } | ||
| name := entry.Name() | ||
| if seen[name] { | ||
| continue | ||
| } | ||
| seen[name] = true | ||
| info, statErr := entry.Info() | ||
| if statErr != nil { | ||
| continue | ||
| } | ||
| if info.Mode()&0o111 == 0 { | ||
| continue | ||
| } | ||
| // Skip files with well-known non-binary extensions. | ||
| switch strings.ToLower(filepath.Ext(name)) { | ||
| case ".json", ".yaml", ".yml", ".md", ".txt", ".sh", ".bat", ".ps1", | ||
| ".go", ".sum", ".mod", ".toml", ".lock", ".html", ".xml": | ||
| continue | ||
| } | ||
| out = append(out, name) | ||
| } |
Comment on lines
+232
to
+307
| if opts.ArtifactPath != "" { | ||
| candidates := discoverArtifactBinaryCandidates(sourceDir, manifest.Name, installName) | ||
| if len(candidates) > 0 { | ||
| var diagLines []string | ||
| diagLines = append(diagLines, fmt.Sprintf("artifact binary discovery: install=%q manifest=%q candidates=[%s]", | ||
| installName, manifest.Name, strings.Join(candidates, ", "))) | ||
|
|
||
| var lastCheckErr error | ||
| for _, cand := range candidates { | ||
| srcPath := filepath.Join(sourceDir, cand) | ||
| srcInfo, statErr := os.Stat(srcPath) | ||
| if statErr != nil { | ||
| diagLines = append(diagLines, fmt.Sprintf(" [skip] %q: %v", cand, statErr)) | ||
| continue | ||
| } | ||
| if copyErr := copyFile(srcPath, binaryPath, srcInfo.Mode()); copyErr != nil { | ||
| diagLines = append(diagLines, fmt.Sprintf(" [fail] %q: copy error: %v", cand, copyErr)) | ||
| lastCheckErr = copyErr | ||
| continue | ||
| } | ||
| cstdout, cstderr, checkErr := checkTypedIaCPlugin(opts.Timeout, filepath.Join(tmp, "plugins"), installName) | ||
| conformanceStdout = cstdout | ||
| conformanceStderr = cstderr | ||
| if checkErr == nil { | ||
| diagLines = append(diagLines, fmt.Sprintf(" [pass] %q selected", cand)) | ||
| conformanceChecked = true | ||
| lastCheckErr = nil | ||
| break | ||
| } | ||
| lastCheckErr = checkErr | ||
| diagLines = append(diagLines, fmt.Sprintf(" [fail] %q: %v", cand, checkErr)) | ||
| } | ||
|
|
||
| diagMsg := strings.Join(diagLines, "\n") | ||
| if conformanceStderr != "" { | ||
| conformanceStderr = diagMsg + "\n" + conformanceStderr | ||
| } else { | ||
| conformanceStderr = diagMsg | ||
| } | ||
|
|
||
| if !conformanceChecked { | ||
| // All candidates failed; build and return fail evidence with diagnostics. | ||
| if lastCheckErr == nil { | ||
| lastCheckErr = fmt.Errorf("no executable artifact candidate could be staged from archive (candidates: %s)", strings.Join(candidates, ", ")) | ||
| } | ||
| manifestSHA, _ := hashFileSHA256(filepath.Join(installDir, "plugin.json")) | ||
| binarySHA := "" | ||
| if _, statErr := os.Stat(binaryPath); statErr == nil { | ||
| binarySHA, _ = hashFileSHA256(binaryPath) | ||
| } | ||
| ev := PluginCompatibilityEvidence{ | ||
| Plugin: manifest.Name, | ||
| Version: manifest.Version, | ||
| EngineVersion: opts.EngineVersion, | ||
| WfctlVersion: buildVersion(), | ||
| Mode: opts.Mode, | ||
| Status: PluginCompatibilityStatusFail, | ||
| OS: runtime.GOOS, | ||
| Arch: runtime.GOARCH, | ||
| ArchiveSHA256: archiveSHA, | ||
| BinarySHA256: binarySHA, | ||
| PluginManifestSHA256: manifestSHA, | ||
| GeneratedBy: "wfctl plugin conformance", | ||
| StdoutTail: conformanceStdout, | ||
| StderrTail: conformanceStderr, | ||
| } | ||
| if normalized, normErr := ValidateCompatibilityEvidence(ev); normErr == nil { | ||
| ev = normalized | ||
| } | ||
| return ev, lastCheckErr | ||
| } | ||
| // conformanceChecked=true: a candidate passed; binary is at binaryPath. | ||
| } | ||
| } else { | ||
| // len(candidates)==0: no executables found in archive root; fall through to go build | ||
| // below (supports source-in-archive tarballs that include Go source). | ||
| } |
Contributor
|
@copilot apply changes based on the comments in this thread and fix coverage: #650 (comment) |
…for source tarballs Agent-Logs-Url: https://github.com/GoCodeAlone/workflow/sessions/bb2b2d56-8dd7-4665-b9bf-35b09f4ddb47 Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Contributor
Author
Applied in commit
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Artifact mode only looked for a binary named
normalizePluginName(manifest.Name)(e.g.digitalocean). GoReleaser archives ship binaries named after the full project (e.g.workflow-plugin-digitalocean), causing artifact conformance to fall through togo build, fail withexit status 1, and write no evidence.Changes
plugin_conformance.godiscoverArtifactBinaryCandidates(sourceDir, manifestName, installName)— new function returning candidates in priority order:installName(normalized, e.g.digitalocean)manifestNamewhen it differs (full name, e.g.workflow-plugin-digitalocean).exevariants of the aboveinstallNameormanifestName(case-insensitive), e.g.digitalocean_linux_amd64. This covers platform-suffixed GoReleaser binaries while preventing execution of arbitrary unrelated executables bundled in the archive.runPluginConformanceCheck— artifact path now:discoverArtifactBinaryCandidates, then runs the go-plugin handshake on each candidate in orderstderrTailin evidencego.modis absent (binary-only artifact) → returnsstatus: failevidence with per-candidate failure reasons instead of a bareexit status 1go.modexists (source-in-archive tarball) → falls back togo build, preserving existing behaviour for source tarballs that happen to include a pre-built or unrelated executablego builddirectlyplugin_conformance_test.gowriteTarGzFileshelper: creates binary-only archives frommap[name→srcPath]buildFixtureBinaryhelper: compiles a fixture to a named binary for use in archive testsTestDiscoverArtifactBinaryCandidates— verifies priority ordering; platform-suffixed names (e.g.digitalocean_linux_amd64) are included; unrelated executables (e.g.some-helper) are excludedTestDiscoverArtifactBinaryCandidatesSameNames— deduplication when installName and manifestName are identicalTestPluginConformanceArtifactDiscoversByManifestName— archive containing onlyworkflow-plugin-iac-pass(noiac-pass) → passTestPluginConformanceArtifactWrongBinaryEmitsEvidence— non-plugin executable →status: failevidence with diagnostic lines naming the candidateTestPluginConformanceArtifactGoModFallback— source archive with a non-plugin executable matching the name prefix → handshake fails →go.modpresent →go buildfallback → passTestPluginConformanceArtifactNoGoModFailsCleanly— binary-only archive (nogo.mod) with a non-plugin executable →status: failevidence with diagnosticsdocs/WFCTL.mdDocuments the multi-candidate discovery, handshake-verification, and
go.mod-based fallback behaviour underplugin conformance.