Skip to content

feat: Add static plugin catalog sync#11

Merged
johnnygreco merged 4 commits intomainfrom
johnny/implement-new-catlog-sync
May 6, 2026
Merged

feat: Add static plugin catalog sync#11
johnnygreco merged 4 commits intomainfrom
johnny/implement-new-catlog-sync

Conversation

@johnnygreco
Copy link
Copy Markdown
Contributor

What

Adds a static plugin catalog tap at catalog/plugins.json and a ddp sync catalog command to regenerate it from the plugins currently in the repository.

The catalog includes package metadata, runtime plugin metadata, entry point information, and DataDesigner compatibility metadata derived from each plugin package\s direct versioned data-designer dependency.

Why

Other CLIs, libraries, and applications need a machine-consumable source of plugin metadata that is not tied to the documentation site. This gives those consumers a stable JSON file they can fetch directly and use to answer questions such as whether a plugin is compatible with a particular DataDesigner version.

Usage

For maintainers:

make catalog
uv run ddp sync catalog --check

For external consumers, fetch catalog/plugins.json and inspect fields such as:

{
  "name": "text-transform",
  "entry_point": {
    "group": "data_designer.plugins",
    "name": "text-transform",
    "value": "data_designer_template.plugin:plugin"
  },
  "compatibility": {
    "data_designer": {
      "requirement": "data-designer>=0.5.7",
      "specifier": ">=0.5.7"
    }
  }
}

Consumers can evaluate compatibility.data_designer.specifier with standard Python packaging tools such as packaging.specifiers.SpecifierSet.

How

  • Adds ddp.catalog to discover local plugin packages, load installed data_designer.plugins entry points, and render deterministic JSON.
  • Parses direct data-designer dependencies with PyPA packaging and rejects missing, malformed, duplicate, or unversioned compatibility metadata.
  • Adds ddp sync catalog and ddp sync catalog --check.
  • Wires catalog drift checking into make check while preserving the existing plugin docs workflow.
  • Updates maintainer docs and PR checklist language for the new generated catalog artifact.

Validation

uv run pytest devtools/ddp/tests/test_catalog.py devtools/ddp/tests/test_cli.py -v
make lint
make check
make all

Observed results:

  • Catalog and CLI focused tests passed: 24 tests.
  • make check verified plugin docs, catalog freshness, CODEOWNERS, and SPDX license headers.
  • make all passed, including isolated plugin tests, plugin validation, generated checks, and strict Zensical docs build.

@johnnygreco johnnygreco changed the title Add static plugin catalog sync feat: Add static plugin catalog sync May 6, 2026
@johnnygreco johnnygreco marked this pull request as ready for review May 6, 2026 16:57
@johnnygreco johnnygreco requested a review from a team as a code owner May 6, 2026 16:57
@johnnygreco johnnygreco requested a review from eric-tramel May 6, 2026 17:06
@eric-tramel eric-tramel added the enhancement New feature or request label May 6, 2026
Copy link
Copy Markdown
Contributor

@eric-tramel eric-tramel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review setup note: GitHub rejected several batch inline-review submissions, so the actionable review was posted below as a changes-requested review with file threads where the API allowed them.

Copy link
Copy Markdown
Contributor

@eric-tramel eric-tramel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Catalog blocker: missing entry-point metadata can silently drop plugin packages from the generated registry. See inline thread.

Comment thread devtools/ddp/src/ddp/catalog.py Outdated
Copy link
Copy Markdown
Contributor

@eric-tramel eric-tramel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the registry-contract issues called out inline. One more blocker that GitHub would not let me attach to .github/workflows/ci.yml: PR #11 adds catalog/CLI tests under devtools/ddp/tests, but CI only calls make test, and that target only iterates plugin packages under plugins/*. Please add a devtools pytest target to make test or call uv run pytest devtools/ddp/tests in CI so these tests are enforced.

Comment thread devtools/ddp/src/ddp/catalog.py
Comment thread docs/authoring.md
@johnnygreco
Copy link
Copy Markdown
Contributor Author

Addressing the CI blocker from the changes-requested review body: e723425 adds a test-devtools target, makes make test run devtools pytest before the isolated plugin tests, and updates the CI test job to run make sync followed by make test. Locally validated with make test, make lint, make check, and make all.

Copy link
Copy Markdown
Contributor

@eric-tramel eric-tramel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this addresses my review points. I re-checked the latest head: devtools tests are now wired into make test/CI, catalog generation now validates entry-point metadata and version/Python compatibility, stale installed entry points are guarded, docs include make sync, local checks passed, and GitHub CI is green.

@johnnygreco johnnygreco merged commit 26b774b into main May 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants