Add the a8c-integration CLI (init + validate) by aswasif007 · Pull Request #1 · Automattic/integration

aswasif007 · 2026-07-20T05:49:19Z

Description

a8c-integration is a standalone CLI for building WordPress VIP Integration Center add-ons. It exists so partners outside Automattic can scaffold an integration and check it for conformance locally and in CI. It ships two commands: init (scaffold a new integration) and validate (conformance-check one).

$ a8c-integration --help
Usage: a8c-integration [options] [command]

Scaffold and validate WordPress VIP Integration Center add-ons.

Options:
  -v, --version              output the version number
  -h, --help                 display help for command

Commands:
  init [options]             Start a new integration by scaffolding from the VIP
                             Integrations Starter Kit.
  validate [options] [path]  Check a VIP integration for conformance before
                             submitting it.

What this PR adds

init command — scaffolds from the VIP Integrations Starter Kit: clones it (no git history), rewrites the example prefix set to the partner's vendor/name, renames the entry file, starts a fresh repo.
validate command — the conformance checker with human and --format json output; exits 1 when non-conformant.
Docs — README.md, SETUP.md, ARCHITECTURE.md, TESTING.md, RELEASING.md.
CI — lint.yml (for eslint, prettier, types, build) and test.yml (jest on Node 20 & 22).
Release — npm-publish.yml for the Github Actions that does the npm publishing.

Tech stack

TypeScript compiled to CommonJS with tsc; bin/a8c-integration is a thin launcher.
commander for arg parsing (only runtime dependency; colors are a small ANSI helper).
Jest + ts-jest for tests.
pnpm, ESLint (@automattic/eslint-plugin-wpvip), and wp-prettier.

How to test

Install and run the suite (28 unit tests) plus the static checks:

pnpm install
pnpm lint && pnpm format:check && pnpm check-types && pnpm test

Build the CLI:

pnpm build && pnpm link

init — scaffold a new integration:

$ a8c-integration init
Vendor name (e.g. "Wordpress"): Wordpress
Integration name (e.g. "Content Sync"): Content Sync
Laying down the VIP Starter Kit into /Users/aswasif007/automattic/content-sync …

✓ Created ContentSync integration at /Users/aswasif007/automattic/content-sync
  slug=content-sync, namespace=Wordpress\ContentSync, config=VIP_CONTENT_SYNC_CONFIG, entry=content-sync.php

Next steps:
  → cd /Users/aswasif007/automattic/content-sync
  → composer install && npm install
  → Run it locally: vip dev-env create && vip dev-env start
  → Edit the integration, then run: a8c-integration validate

validate — check the scaffold (exit code 0):

$ cd content-sync
$ a8c-integration validate

Integration conformance check: /Users/aswasif007/automattic/content-sync

  PASS  Rule 1: Loads through the Starter Kit workflow
        Plugin entry file and Composer wordpress-plugin package are present.
        - Entry file: content-sync.php
        - composer.json type is "wordpress-plugin" with an autoload section.
  PASS  Rule 2: `composer test` runs PHPUnit and e2e tests
        composer test declares a PHPUnit run and an e2e runner (Playwright/Cypress).
        - Resolved commands: phpunit • npm test • playwright test
        - Static check: it verifies the test commands are wired, not that the tests pass.
  PASS  Rule 3: `composer run validate-integration` exists
        composer.json defines a "validate-integration" script.
  PASS  Rule 4: Config constant is documented and referenced in code
        Config constant VIP_CONTENT_SYNC_CONFIG is referenced in code and documented.
  PASS  Rule 5: Missing/invalid config is handled without fataling
        Config access is guarded against a missing or invalid constant.
        - Guards found: is_ready(), missing_fields(), is_available(), defined() guard, is_array() guard
        - Static signal only — behavioral proof comes from the integration's own tests (rule 2).
  PASS  Rule 6: Docs include valid and incomplete config examples
        Docs include both a valid and an incomplete config example.
  PASS  Rule 7: Compatibility evidence covers WP 6.9/7.0 and PHP 8.2-8.5
        CI matrix covers WP 6.9 + 7.0 and PHP 8.2-8.5.
  PASS  Rule 8: Build and test commands are documented
        Docs document both build/install and test commands.
  PASS  Rule 9: Telemetry uses the Starter Kit pattern (Tracks only, no secrets)
        Telemetry uses the guarded VIP Tracks helper with no obvious secrets in properties.

Human review required (not automated):
  • Plugin - platform config-schema match
    Whether the plugin's expected config matches the platform schema is not fully deterministic and is confirmed in human review, not by this checker.
  • Security review
    Security posture (input handling, secret storage, capability checks) is assessed in human review, not by this checker.

Summary: 9 passed, 0 failed, 0 warnings, 0 n/a.
✓ Conformant — no automated checks failed.

Rule 2: drop no-op commands before following npm delegations, so an `echo npm test` can no longer expand into a real e2e run and smuggle a passing verdict past the filter. Rule 8: match the same e2e runner set as Rule 2 so docs using Cypress/Codeception/Puppeteer are not dinged.

Rule 3 only checked that the composer "validate-integration" key exists, so a stub like "echo ok" passed clean. Resolve the script body and drop no-op commands, the same guard Rule 2 uses, so an empty stub can no longer satisfy the gate. Asserting the real VIP validator runs is left until VIP publishes it (today it is a documented placeholder).

realCommands treated an entry starting with echo/:/true/# as a pure no-op, so a banner like `echo "Running" && phpunit` discarded the whole chain and Rule 2/3 saw no real command — a false FAIL that blocks partners who banner their test runs. Split each entry on shell separators first and drop only the segments that are themselves no-ops.

pandah3

Thank you for the huge effort porting this over!!

I got an error running the init command in the testing steps and needed to build it first:

a8c-integration init
zsh: command not found: a8c-integration

pnpm build
node bin/a8c-integration init
node bin/a8c-integration validate ./test-integration

Worked as expected!! But I haven't tweaked the integration to try to have it fail yet.

pandah3 · 2026-07-21T05:58:54Z

+	program
+		.command( 'init' )
+		.description( 'Start a new integration by scaffolding from the VIP Integrations Starter Kit.' )
+		.option( '--vendor <vendor>', 'Vendor name (e.g. "Wordpress").' )


Suggested change

.option( '--vendor <vendor>', 'Vendor name (e.g. "Wordpress").' )

.option( '--vendor <vendor>', 'Vendor name (e.g. "WordPress").' )

pandah3 · 2026-07-21T05:59:12Z

+Interactive — it asks for your **vendor name** and **integration name**, always builds from the canonical [VIP Integrations Starter Kit](https://github.com/Automattic/vip-integrations-starter-kit) (its default branch, no git history pulled), rewrites the example prefix set to your names, renames the entry file, and starts a fresh git history. You can also pass the answers as flags:
+
+```bash
+a8c-integration init --vendor "Wordpress" --name "Content Sync"


Suggested change

a8c-integration init --vendor "Wordpress" --name "Content Sync"

a8c-integration init --vendor "WordPress" --name "Content Sync"

pandah3 · 2026-07-21T06:21:04Z

+	// changelog or prose. Scanning arbitrary docs lets coincidental substrings
+	// pass, so only workflows and an explicit exception note count here.
+	if (
+		/compatibility exception|approved exception/i.test(


Looks like this will match any record of compatibility exception or approved exception anywhere in docs/workflows, so if there's a line in the README that says, "There is no approved exception on file," it'll pass.

Should we gate this on something more explicit? Suggestions:

Option A: An HTML comment in the docs

 An HTML comment is handy here because markdown doesn't render it — it's invisible in the displayed README but sits in the source, so it reads as a deliberate machine marker, not prose. The check becomes anchored and specific: const EXCEPTION_MARKER = //i; if (EXCEPTION_MARKER.test(ctx.docsText)) { return { ...base, status: 'pass', message: '…' }; }

Option B: A structured field in composer.json (preferred)

{ "extra": { "vip": { "compatibility-exception": "approved" } } } const flag = ctx.composer?.extra?.vip?.['compatibility-exception']; if (flag === 'approved') { … pass … }

Also, instead of auto-passing for this rule, should we require it to still route to the human-review section for sign-off or maybe downgrade to a warning that says "exception claimed, needs reviewer confirmation?"

pandah3 · 2026-07-21T06:25:11Z

+        run: |
+          set -euo pipefail
+          tag_name="v$PACKAGE_VERSION"
+          gh release create "$tag_name" --target "$GITHUB_SHA" --title "$tag_name" --generate-notes


This checks out ref: trunk up top, so it always builds and publishes trunk's code — but the release is created with --target "$GITHUB_SHA". On a workflow_dispatch, GITHUB_SHA is the commit of whatever ref you triggered from, which the UI lets you pick. So if someone dispatches from a feature branch, we publish trunk to npm but tag vX.Y.Z on the feature branch commit. The tag ends up pointing at code that was never shipped.

Should we tag the tree we actually built? --target "$(git rev-parse HEAD)" after checkout would keep the tag and the published code in sync.

pandah3 · 2026-07-21T06:29:15Z

+
+function checkGracefulConfigHandling( ctx: Context ): CheckResult {
+	const base = {
+		id: 'graceful-config-handling',


The config guard (rule 5) passes if is_array( / is_ready( / is_available( appears anywhere in the concatenated PHP source, and rule 9's telemetry guard does the same with class_exists(. Those are generic functions used all over a normal plugin, so an unrelated is_array() in some helper makes it print "Config access is guarded against a missing or invalid constant" even when the actual config read has no guard at all. It's checking "does this word exist in the codebase," not "is the risky call guarded."

Could we scope the search to a window around the config constant (and around record_event( for rule 9)? Same approach the secret-key scan just below already uses.

pandah3 · 2026-07-21T06:31:57Z

+	if ( blocks.length < 2 ) {
+		return {
+			...base,
+			status: 'fail',


Rule 6 tells "valid" from "incomplete" by counting keys. If one doc example has fewer keys than another, it calls that pair valid+incomplete. But fewer keys usually just means fewer optional settings, not a missing required field, so two perfectly valid examples of different lengths pass as if one were the broken case the rule is trying to require.

Detecting "missing a required field" statically is genuinely hard, so maybe we lean on the explicit signal instead. The mentionsIncomplete regex is already there. Require the docs to actually label the incomplete example rather than guessing from key count?

pandah3 · 2026-07-21T06:32:57Z

+	// cloned plainly (git ignores --depth for local paths and warns).
+	const depth = isRemote ? [ '--depth', '1' ] : [];
+	// `--` ends option parsing so the source can never be read as a git flag.
+	execFileSync( 'git', [ 'clone', ...depth, '--quiet', '--', source, target ], {


If the clone dies partway (network drop) or scaffoldTree throws, we leave a half-populated target on disk. The next init then hits the "Target directory is not empty" guard and the user is stuck cleaning up by hand and the error doesn't say why. The comment at line 86 says we fail fast so we don't leave a half-laid-down directory behind, but that only covers the name validation, not these two spots.

Can we wrap the clone + scaffold in a try/catch and rmSync(target, { recursive: true, force: true }) on failure, so a failed run leaves nothing behind?

pandah3 · 2026-07-21T06:33:12Z

+	console.log( gray( `Laying down the VIP Starter Kit into ${ target }` ) );
+	laySkeleton( target );
+
+	const { entryFile, prefix } = scaffoldTree( target, vendor, name );


Same here: https://github.com/Automattic/integration/pull/1/changes#r3619936703

pandah3 · 2026-07-21T06:33:49Z

+/** Paths (relative, forward-slash) that must not be rewritten. Mirrors the
+ * skip set in bin/setup.php: dependencies, lockfiles, the scaffolder itself,
+ * and binary assets. */
+const SKIP_PATTERN =


The skip set only excludes .png/.jpg/.jpeg/.gif/.webp, but we read every other file as utf8 and write it back. If the kit ships any other binary (.ico, .woff2, .ttf, .pdf) that happens to contain a token's ASCII bytes, reading it as utf8 swaps the invalid bytes for U+FFFD and we write the corrupted version back. The updated !== contents check saves token-free binaries, so it only bites a binary that also contains a token — but the comment says we skip "binary assets" and we only skip some of them.

Could we either broaden the skip list to the other binary types, or sniff for a NUL byte and skip anything that looks binary before rewriting?

pandah3 · 2026-07-21T06:37:16Z

+		missing.push( 'WordPress 7.0' );
+	}
+	for ( const php of [ '8.2', '8.3', '8.4', '8.5' ] ) {
+		if ( ! ctx.workflowsText.includes( php ) ) {


The PHP versions are checked with .includes('8.4'), so it matches that string anywhere in the workflow YAML, not just on a PHP-version line. CI files are full of other version numbers — node-version: 18.4, mysql:8.4, a pinned action tag — any of those makes it think PHP 8.4 is covered. So an integration that only tests 8.2 but has a Node matrix of 18.3/18.4/18.5 reports full 8.2–8.5 coverage and passes.

The WP checks right above already dodge this with \b6.9\b. A word boundary here helps (\b8.4\b won't match inside 18.4), but it still matches mysql:8.4, so ideally we scope the match to a php-version key the way the wpLatest pattern scopes to a WP key. Can we tie it to the php line instead of a loose substring?

aswasif007 added 6 commits July 20, 2026 18:04

Set up project tooling and build config

05ef755

Add the integration conformance validator

4d29a5d

Add the Starter Kit init scaffolder

2557d17

Wire up the a8c-integration CLI entry point

a3f07d3

Add CI workflows and npm release tooling

b83e1d4

Add README and documentation

402b034

aswasif007 changed the title ~~Create/a8c integration cli~~ Add the a8c-integration CLI (init + validate) Jul 20, 2026

aswasif007 force-pushed the create/a8c-integration-cli branch from 3060680 to 402b034 Compare July 20, 2026 12:09

Use "Wordpress" instead of "Acme" as example vendor name

5466067

aswasif007 mentioned this pull request Jul 20, 2026

Add vip integration validate conformance checker Automattic/vip-cli#2978

Closed

7 tasks

aswasif007 added 3 commits July 20, 2026 19:41

Frame init output as a fresh scaffold, not a rewrite

a59ebd9

Remove the built-in help subcommand

df3d7a8

Add vip dev-env step to init next steps

b3c61b7

aswasif007 marked this pull request as ready for review July 20, 2026 13:57

aswasif007 added 5 commits July 20, 2026 20:03

Remove ... from desc

d60d4db

Skip symlinks in the scaffolder file walk

668f8e2

pandah3 reviewed Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add the a8c-integration CLI (init + validate)#1

Add the a8c-integration CLI (init + validate)#1
aswasif007 wants to merge 15 commits into
trunkfrom
create/a8c-integration-cli

aswasif007 commented Jul 20, 2026 •

edited

Loading

Uh oh!

pandah3 left a comment

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

pandah3 Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	.option( '--vendor <vendor>', 'Vendor name (e.g. "Wordpress").' )
	.option( '--vendor <vendor>', 'Vendor name (e.g. "WordPress").' )

	a8c-integration init --vendor "Wordpress" --name "Content Sync"
	a8c-integration init --vendor "WordPress" --name "Content Sync"

Uh oh!

Conversation

aswasif007 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What this PR adds

Tech stack

How to test

Uh oh!

pandah3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aswasif007 commented Jul 20, 2026 •

edited

Loading