docs: graduate plugins out of experimental mode

[johnnygreco]

Summary

Updates the docs around two related areas: plugin authoring now that the extension points are no longer experimental, and the code reference section so APIs are grouped by package/layer and have enough context to be useful from the docs site.

Changes

Added

• docs/plugins/build_your_own.md as the consolidated guide for building column generator, seed reader, and processor plugins.
• docs/plugins/models.md for model-backed plugin patterns and model registry usage.
• Package/layer-oriented code reference pages under docs/code_reference/config/, docs/code_reference/engine/, and docs/code_reference/interface/, with overview pages for each group.
• __init__.py files in engine resources and processing subpackages so mkdocstrings/griffe can discover seed reader and processor classes.

Changed

• Reworked the Plugins nav and overview around Overview, Build Your Own, Using Models, and Available Plugins.
• Embedded the NVIDIA-maintained plugin catalog table from the DataDesignerPlugins repo in docs/plugins/available.md.
• Reorganized the Code Reference nav in mkdocs.yml by Config, Engine, and Interface, with updated cross-links from concepts and recipes.
• Expanded and corrected docstrings for plugin extension points, config objects, generators, seed readers, processors, interface classes, and analysis/config references so generated docs render with useful field and method descriptions.
• Improved code reference table styling so wide generated tables remain readable on narrower viewports.

Removed

• Replaced the older plugin example pages (example.md, filesystem_seed_reader.md, processor.md) with the consolidated Build Your Own guide and targeted reference pages.
• Replaced the older flat code reference pages with package-grouped code reference pages.

Attention Areas

Reviewers: Please pay special attention to the following:

• mkdocs.yml - navigation moved from flat reference pages to package groups, and remote markdown snippets are now enabled for the plugin catalog table.
• docs/plugins/available.md - the NVIDIA plugin table is pulled from the raw DataDesignerPlugins catalog during docs builds.
• docs/code_reference/ - page paths and anchors changed as part of the Config / Engine / Interface split.
• packages/data-designer-*/src/data_designer/ docstrings and discovery shims - these are mostly documentation-facing changes, but they affect what mkdocstrings exposes in the generated docs.

Validation

• uv run --group docs mkdocs build

---

Description updated with AI

[github-actions]

Docs preview: https://cac89582.dd-docs-preview.pages.dev

Notebook tutorials are placeholder-only in previews.

[github-actions]

Review: PR #603 — docs: graduate plugins out of experimental mode

Summary

This PR reorganizes plugin documentation now that the three plugin extension points (column generator, seed reader, processor) are stable. It removes the "Experimental Feature" banners, replaces the single plugins/example.md walkthrough with per-type implementation guides (implement.md, processor.md, expanded filesystem_seed_reader.md), and adds proper API reference pages (code_reference/plugins.md, code_reference/generators.md). The plugins/available.md page becomes a real "Catalog" that points to the NVIDIA-maintained DataDesignerPlugins repo.

Code changes are minimal and non-functional:

• Docstrings added to Plugin, PluginType, SingleColumnConfig.allow_resize, ProcessorConfig.processor_type, SeedSource, FileSystemSeedSource, ColumnGeneratorCellByCell, and ColumnGeneratorFullColumn.
• A typo fix in Plugin.config_qualified_name's description ("name o the" → "name of the").
• Three empty __init__.py files under data_designer.engine so griffe/mkdocstrings can resolve SeedReader, FileSystemSeedReader, and Processor for the new reference pages.
• mkdocs.yml nav reshuffle + alphabetizing of Code Reference pages.

Scope matches what the PR description claims.

Findings

Correctness

• __init__.py additions respect the namespace-package invariant. AGENTS.md pins the PEP 420 rule at the top-level data_designer namespace. The three new files live under data_designer.engine.resources, data_designer.engine.processing, and data_designer.engine.processing.processors, all inside a single distribution package — this is the normal way to expose subpackages for griffe and does not break the cross-distribution namespace merge. No concern.
• Nav + anchors cross-link correctly. Spot-checked docs/code_reference/plugins.md {#data_designer.plugins.plugin.Plugin} anchor against the references in docs/plugins/overview.md, docs/plugins/implement.md, docs/code_reference/generators.md, and the column_configs.md addition for SingleColumnConfig. All the page-relative links (../plugins/implement.md, ../code_reference/generators.md, etc.) match the new filenames in this PR.
• Plugin description typo fix matches the docstring fix. Both say "name of the …" now. Good.
• Env-var documentation. overview.md states DISABLE_DATA_DESIGNER_PLUGINS=true disables entry point discovery. Verify the name matches the actual variable in the discovery code — docs of this kind rot silently if the flag is renamed.

Example code quality

• Processor example uses astype(str).apply(lambda …) in both implement.md and processor.md. Idiomatic pandas would be data[self.config.column].astype(str).str.contains(self.config.pattern, regex=True) (optionally pre-compiled is unnecessary when using the Series accessor). As a "minimum working example" it's fine; a short note that vectorized .str.contains is preferable for real workloads would help new plugin authors.
• get_column_emoji() returns "x" in implement.md where the old example.md used "✖️". Intentional simplification is fine, but x looks like a placeholder — consider a real emoji so readers don't copy a bare letter into their log output.
• Import style is consistent across the three tab examples: from __future__ import annotations + TYPE_CHECKING for pandas when only used in annotations. Good — this mirrors the style guide's fast-import guidance.
• Multiple-plugins-per-package section dropped the tests_e2e reference. The removed example.md pointed at tests_e2e/ as a concrete example of this pattern; implement.md's "Multiple plugins in one package" section just shows a TOML snippet. If that e2e directory is still a working example, add the link back — it's a cheap pointer that saves plugin authors from guessing.

Documentation accuracy

• assert_valid_plugin coverage. implement.md says "Data Designer provides a testing utility for common plugin structure checks" and shows a single example. The deleted example.md explicitly listed what it validates ("config is subclass of ConfigBase", etc.). The new, terser wording is fine for most readers, but the deleted enumeration was genuinely useful for the "what will this catch?" question. Worth preserving one sentence about it.
• Discovery troubleshooting bullets in overview.md are good and concrete (discriminator must be a string, regex-filter → REGEX_FILTER, etc.). This replaces an entire "Experimental" framing with something actionable — nice improvement.
• Processor callback table in processor.md accurately lines up with the three process_* methods. The async-engine caveat note about row-count-changing pre/post-batch processors under DATA_DESIGNER_ASYNC_ENGINE=1 is a useful landmine callout.

Style / conventions

• Docstring additions follow the existing Attributes-block format used elsewhere in base.py and plugin.py. No drift.
• New __init__.py files use the right SPDX header (2026, Apache-2.0). Consistent with other engine packages.
• mkdocs.yml alphabetization applies to Code Reference only, not to the top-level nav, which matches the comment # Keep code reference pages ordered alphabetically by nav label.. Confirm whether "Plugins" in the top-level nav should similarly list Catalog before Build Your Own (it currently goes Overview → Build Your Own → Catalog, which reads as a natural user journey and is probably preferable to alphabetical).

Risks

• Low. This is almost entirely docs plus docstrings. The only runtime-observable change is the three new __init__.py files; because they are beneath a single installable package and do not introduce a data_designer/__init__.py, they cannot break the cross-package namespace merge.
• One residual risk: if any tooling elsewhere in the repo relied on those three directories being implicit namespace packages (unlikely but worth a grep for pkgutil/find_namespace_packages usage around data_designer.engine.processing), it should still work — explicit subpackages are a strict superset of namespace behavior inside a single distribution.

Suggestions (non-blocking)

1. Confirm DISABLE_DATA_DESIGNER_PLUGINS matches the env var the discovery code actually reads.
2. Replace the "x" emoji placeholder in implement.md's column generator example with a real emoji.
3. Prefer Series.str.contains(...) over astype(str).apply(lambda …) in the processor examples, or add a one-line note that the .apply form is for illustration.
4. Restore the tests_e2e/ pointer in the "Multiple plugins in one package" section.
5. Optional: keep one sentence listing what assert_valid_plugin actually checks, since the deleted page had that and it's useful signal.

Verdict

Looks good to merge. The restructure is a clear improvement over the old single-example layout, the docstring additions are well-scoped and match existing style, and the __init__.py additions are the correct fix for mkdocstrings discovery without breaking the namespace-package invariant. The suggestions above are small polish items and none of them should block this PR.

[greptile-apps]

Greptile Summary

This PR graduates the plugin system out of experimental status by consolidating the plugin authoring guides into build_your_own.md and models.md, restructuring the Code Reference nav into Config / Engine / Interface groups, and correcting docstrings throughout the config and engine packages.

• Docs reorganization: Removed the three older experimental plugin pages, added build_your_own.md (covers column generators, seed readers, and processors) and models.md (model registry access patterns); restructured Code Reference from a flat list into config/, engine/, and interface/ subdirectories with corresponding nav entries in mkdocs.yml.
• Python source changes: Added empty __init__.py files to engine/processing/ and engine/resources/ subpackages so mkdocstrings can discover the classes; corrected docstrings across plugin.py, base.py, processors.py, column_generators/base.py, and related config files, including fixing the actual artifact directory names (dropped-columns-parquet-files, processors-files).
• Remote snippet: docs/plugins/available.md now embeds the NVIDIA plugin catalog from a raw GitHub URL via pymdownx.snippets with url_download: true; this adds a network dependency to docs builds.

Confidence Score: 5/5

This is a documentation-only restructuring with no behavioral changes to library code; safe to merge.

All Python source changes are limited to docstring corrections and empty init.py discovery shims. No logic paths, data flows, or runtime behavior changed. Link updates and nav restructuring are internally consistent. Docstring corrections align with the actual artifact directory names found in the engine code.

No files require special attention.

Important Files Changed

Filename	Overview
mkdocs.yml	Nav restructured from flat Code Reference list into Config/Engine/Interface groups; deleted plugin pages replaced with build_your_own and models; url_download: true added to pymdownx.snippets to enable remote catalog embedding.
docs/plugins/build_your_own.md	New consolidated plugin authoring guide covering column generators, seed readers, and processors with correct import paths and base classes verified against engine source.
docs/plugins/available.md	Replaced placeholder with live plugin catalog; embeds remote GitHub raw URL via pymdownx snippets slice syntax (:5:) requiring the url_download: true flag added in mkdocs.yml.
docs/plugins/models.md	New guide for model-backed plugin patterns; import paths and base class usage (ColumnGeneratorWithModel, ColumnGeneratorWithModelRegistry) match actual engine classes.
packages/data-designer-config/src/data_designer/plugins/plugin.py	Added class-level docstrings to PluginType and Plugin; fixed typo in config_qualified_name field description.
packages/data-designer-config/src/data_designer/config/processors.py	Corrected docstring directory names to match actual artifact storage constants: dropped-columns-parquet-files and processors-files, both verified in artifact_storage.py.
packages/data-designer-engine/src/data_designer/engine/column_generators/generators/base.py	Added class docstrings to ColumnGeneratorCellByCell and ColumnGeneratorFullColumn, and abstract generate() method docstrings; no behavioral changes.
packages/data-designer-engine/src/data_designer/engine/processing/init.py	New empty init.py to make the processing subpackage importable by mkdocstrings/griffe for doc generation.
packages/data-designer-config/src/data_designer/config/seed_source.py	Added class docstring to FileSystemSeedSource with accurate field descriptions for path, file_pattern, and recursive.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Plugin package published] --> B[Entry point registered\ndata_designer.plugins group]
    B --> C{import data_designer}
    C --> D[Discovery scans installed\nentry points]
    D --> E{DISABLE_DATA_DESIGNER_PLUGINS?}
    E -- true --> F[Discovery skipped]
    E -- false --> G[Load Plugin objects]
    G --> H{PluginType?}
    H -- COLUMN_GENERATOR --> I[Register config discriminator\nin column_type union]
    H -- SEED_READER --> J[Register config discriminator\nin seed_type union]
    H -- PROCESSOR --> K[Register config discriminator\nin processor_type union]
    I --> L[Config builder add_column]
    J --> M[Seed source resolution]
    K --> N[Config builder add_processor]

Loading

_{Reviews (6): Last reviewed commit: "docs: clarify plugin model usage" | Re-trigger Greptile}

[andreatgretel]

Codex caught this: "usable immediately after a local install" is only true in a fresh Python process. PluginRegistry snapshots entry points on first init (registry.py:33), and column_types/processor_types/seed_source_types/DEFAULT_SEED_READERS all inject plugins at module-import time. Notebook authors who uv pip install -e . after import data_designer will get no discovery without restarting the kernel. wdyt about a one-line restart caveat?

[johnnygreco]

Good catch. The overview no longer has the old local-install walkthrough, but the current “Use an Installed Plugin” paragraph still needed the same cache caveat. I added a line noting that if data_designer has already been imported, users should restart the Python process so discovery rebuilds from the new entry points.

[andreatgretel]

same caveat as on overview.md:58 — "the editable install registers the entry point so Data Designer can discover the plugin" is true at first import only. PluginRegistry discovers once and caches; the column/processor/seed-source unions are built at import. Worth a one-liner about kernel restart since iterative install in a notebook is the obvious workflow. (Also applies to processor.md:136, which has the same line.)

[johnnygreco]

This is covered now in the install section directly below this sentence: it notes that Data Designer caches the plugin registry on first import, and tells notebook users to restart the kernel or interpreter after uv pip install -e .. The old processor.md page was folded into build_your_own.md, so there is no second copy left to update.

[andreatgretel]

docs/code_reference/plugins.md:5 plus the three new __init__.py files (engine/resources/, engine/processing/, engine/processing/processors/)

The PR description says these __init__.py files exist "so griffe (mkdocstrings) can discover SeedReader, FileSystemSeedReader, and Processor for the new code reference." But I couldn't find any ::: directive in docs/ that targets data_designer.engine.resources.* or data_designer.engine.processing.* — those classes only appear inside code-block from … import … statements, which mkdocstrings doesn't process. Codex flagged the same thing.

As shipped, only column generators get an actual mkdocstrings-rendered API reference (via code_reference/generators.md). Processor and seed-reader authors still have prose examples but no auto-rendered base-class reference like the PR description promises.

Either land code_reference/seed_readers.md and code_reference/processors.md (engine-side) here to mirror the generators.md pattern, or drop the __init__.py files and the docstring-only churn on Processor/SeedReader/FileSystemSeedReader/SeedSource/FileSystemSeedSource/ProcessorConfig until a follow-up PR delivers the rendered pages. wdyt?

[johnnygreco]

@andreatgretel Re: #603 (comment)

This is addressed in the current PR state. The branch now has docs/code_reference/engine/seed_readers.md and docs/code_reference/engine/processors.md, both included under the Engine code-reference nav, and both pages contain mkdocstrings ::: directives for the relevant engine-side classes. I kept the __init__.py files and docstring work because those rendered reference pages now exist.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[johnnygreco]

@nabinchha could you take a close look at the Using Models in Plugins section?

Link: https://github.com/NVIDIA-NeMo/DataDesigner/blob/johnny/docs/plugins-out-of-experimental-mode/docs/plugins/models.md

We want to establish a good pattern for using models in plugins, especially the recommended base class split for single-model vs multi-model generators and the alias validation / health-check behavior.

[nabinchha]

@johnnygreco — took a close pass at docs/plugins/models.md. Cross-checked it against the engine code; the base-class split and the discovery → health-check chain are described correctly. A few things worth tightening, plus one engine-side limitation that came out of the review.

Things I'd change in this PR

1. Tighten the wording around model_alias being required. The doc currently says "the config should keep a primary model_alias field because startup health checks collect that field…". Because _run_model_health_check_if_needed does model_aliases.add(config.model_alias) unconditionally for any column type whose impl inherits ColumnGeneratorWithModelRegistry, it's effectively required, not advisory — without it, plugin users get an AttributeError from inside the health-check loop before the friendlier registry "alias not found" error ever runs. Suggest something like "The config must include a model_alias: str field — startup health checks read it directly off any column config whose generator inherits from ColumnGeneratorWithModelRegistry (including via ColumnGeneratorWithModel)."

2. Show the PairwiseJudgeColumnConfig alongside the multi-model generator example. The single-model example shows both halves; the multi-model one only imports PairwiseJudgeColumnConfig from data_designer_pairwise_judge.config, which makes it harder for readers to see that the config defines both model_alias and judge_model_alias. A small config snippet (or an inline comment) closes the loop with point 1 and makes it visually obvious which alias gets the standard health check vs which one only gets the _validate() resolution.

3. Sharpen the alias-validation note. "Validate additional alias fields in _validate()… so missing aliases fail before generation starts" is true, but readers may infer a model health check happens. Something like "get_model_config(alias) only verifies the alias is registered; it does not call the endpoint. Endpoint reachability is only exercised for the primary model_alias collected by the standard startup health check."

4. Tiny copy nit: "The engine already builds a ResourceProvider for each generator" reads as one-per-generator; in practice it's one ResourceProvider per builder shared with each generator. Easy fix: "…builds a ResourceProvider and exposes its model registry to every generator at:".

Engine-side limitation surfaced by the review

The reason point 3 needs to exist at all is that secondary aliases on a packaged plugin config can't be opted into the standard startup health check today — only CustomColumnConfig.model_aliases (plural) is rolled in via an isinstance branch in the builder. For a packaged plugin with model_alias + judge_model_alias, only the primary alias gets the endpoint ping; the secondary alias's reachability and credentials only surface at first generation call.

I filed #606 to propose a small fix: a get_model_aliases() accessor on SingleColumnConfig that defaults to [self.model_alias] (preserving current behavior) and that plugin configs override to declare every alias they depend on. The builder's isinstance(config, CustomColumnConfig) branch collapses into the same path, and the docs in this PR can switch from "validate manually in _validate()" to "override get_model_aliases()". Happy to do that as a follow-up to #603 once the docs land, or fold it in if you'd rather ship them together.

Verdict on the docs alone: the substance is right, the four items above are polish, and the multi-model alias story will be much cleaner once #606 lands.

[andreatgretel]

Codex caught this: assert_valid_plugin doesn't actually have a PROCESSOR branch in engine/testing/utils.py, so for processor plugins it only checks the config inherits ConfigBase. The impl class isn't type-checked at all. With the processor tab right above, it reads like processors are covered. Maybe add a PROCESSOR branch checking issubclass(impl_cls, Processor), or call it out here? wdyt

[andreatgretel]

nit: worth double-checking that catalog URL resolves before merge, since pymdownx.snippets will hard-fail the build on a 404. also slightly nervous that every docs build now depends on DataDesignerPlugins/main being healthy. pinning to a tag or vendoring a snapshot might be safer, wdyt?

[andreatgretel]

Bundling the code reference reorg with plugins-graduation makes sense given plugin authors now need to navigate engine.column_generators, engine.processing.processors, etc. as a real public surface. The thing I'd still flag: the reorg landed across ~10 commits and grew the diff from 19 files to 72, which makes reviewing harder and links the two for reverts. Not blocking, just wanted to surface it. A sentence in the PR description tying the two together would help future readers.