Make DataDesignerPlugins an install-ready plugin tap

[eric-tramel]

What

Makes this repository consumable as a Data Designer plugin tap by adding a schema v2 catalog contract, source metadata, install-target derivation, scaffold support for multiple plugin types, release validation, contract fixtures, and end-user/maintainer documentation.

The schema v2 contract is now package-first and generic: top-level packages[] entries describe installable Python distributions, and each package lists the runtime Data Designer plugins exposed after installation. Catalog JSON can be hosted from any raw JSON endpoint, while packages can come from PyPI, Git, direct HTTP(S) wheel/sdist URLs, or local paths for authoring.

The epic merge train includes:

• Define tap catalog schema v2 contract #26: define the schema v2 tap catalog contract.
• Add repo-level tap metadata config #27: add repo-level tap metadata configuration.
• Add schema v2 catalog source metadata #28: emit schema v2 catalog source metadata.
• Add scaffold plugin type support #29: support ddp new --type column-generator|seed-reader|processor.
• Add tap source install target derivation #30: derive installer-ready targets for PyPI, Git, direct URL, and local path catalog sources.
• Document default tap catalog URL #31: document the default tap catalog URL.
• Document plugin tap governance and trust #32: document plugin tap governance and trust boundaries.
• Add tap catalog contract fixtures #33: add schema v2 catalog contract fixtures and fixture validation.
• Strengthen release validation #34: strengthen release validation for catalog and CODEOWNERS readiness.
• Document plugin tap workflows #35: document the complete plugin tap workflow.

Closes #15.

Why

Data Designer needs a predictable plugin discovery surface outside the core package. This gives NDD-style clients a stable catalog to read, a contract to validate, and enough trusted metadata to decide what can be installed without importing arbitrary plugin code first.

The package-first shape avoids baking this repository layout into the public contract. A tap host only needs to serve valid JSON, and each catalog package can point to whichever install source is appropriate for that package.

Usage

Scaffold a plugin by type:

uv run ddp new github-repo-reader --type seed-reader
uv run ddp new normalize-text --type column-generator
uv run ddp new redact-records --type processor

Check generated tap metadata and release readiness:

uv run ddp sync catalog --check
uv run ddp check-release data-designer-template 0.1.0
make validate-release PLUGIN=data-designer-template

A client can read catalog/plugins.json, validate schema v2, list package-level sources, flatten runtime plugins for search/info views, filter by Data Designer compatibility, and derive install targets such as:

data-designer-template==0.1.0
data-designer-git-seed-reader @ git+https://github.com/NVIDIA-NeMo/DataDesignerPlugins.git@data-designer-git-seed-reader/v0.2.0#subdirectory=plugins/data-designer-git-seed-reader
data-designer-url-processor @ https://packages.example.test/data_designer_url_processor-0.2.1-py3-none-any.whl
uv pip install -e /path/to/DataDesignerPlugins/plugins/data-designer-template

How

The implementation centralizes package-first catalog parsing, schema v2 validation, source metadata normalization, compatibility filtering, and install-target derivation in ddp.catalog. Repository tap defaults are loaded through ddp.tap_config, and generated catalog output now groups runtime plugin entries under each installable package.

The scaffold code maps the requested plugin type into the correct Data Designer plugin contract and generated package shape. Release checks verify checked-in schema v2 package entries, source consistency, release-eligible CODEOWNERS, and generated metadata expectations.

Docs cover the catalog schema, default tap URL, plugin tap workflow, governance/trust model, authoring flow, and release workflow.

Validation

Subissue PRs were merged into the epic branch only after targeted checks. The final integrated branch was validated with:

uv run pytest devtools/ddp/tests/test_catalog.py devtools/ddp/tests/test_catalog_fixtures.py devtools/ddp/tests/test_tap_config.py devtools/ddp/tests/test_validate_release.py -q
make docs
make check
make all
git diff --check

The final make all run passed lint, format check, devtools tests, isolated plugin tests, plugin validation, generated artifact checks, and strict docs validation.

I also ran an isolated scratch-space mock NDD CLI QA in /tmp/ddp-epic15-package-first-mock-ndd-qa. That mock client validated the real package-first catalog, listed packages and flattened runtime plugins, exercised search/info/install planning, validated schema fixtures, checked compatibility filtering, verified Git subdirectory and Git root install-target derivation, verified direct URL install-target derivation, rejected invalid fixtures, rewrote the template package to a local path source, installed it editable into a scratch venv, and verified the data_designer.plugins entry point was discoverable.