feat: Add Pydantic v2 serializer support to Python publisher

[kulnor]

Description

Summary

Extends the Python publisher to support generating Pydantic v2 models alongside the existing standard library dataclass implementation. Users can choose between zero-dependency Python dataclass models (default) and validated, strongly-typed pydantic.BaseModel models via the --flavor CLI option or the dedicated publish-pydantic command.

---

Key Highlights

• Unified Python Publisher Architecture: Integrated Pydantic generation directly into PythonPublisher using a PythonFlavor enum (Dataclass vs. Pydantic), sharing common schema analysis and traversal logic.
• Pydantic v2 Model Generation:
  • Emits BaseModel classes with Field metadata for COGS property aliases, default factories, and constraints.
  • Supports polymorphism and subtype hierarchies with SerializeAsAny annotations.
  • Generates pyproject.toml targeting pydantic>=2.0.
• Runtime Serialization & Deserialization:
  • Embedded Pydantic runtime with support for lossless JSON and XML parsing/writing, ItemContainer, Gregorian date/time helpers, and reference resolution.
• CLI & Command Surface:
  • Added -f|--flavor [dataclass|pydantic] option to cogs publish-python (defaults to dataclass).
  • Added cogs publish-pydantic command as a convenient shorthand for --flavor pydantic.
• Documentation & Tests:
  • Added comprehensive unit and integration tests for Pydantic code generation and runtime round-tripping.
  • Added technical guide and CLI documentation for Pydantic publisher options.

---

CLI Usage

# Generate Pydantic v2 models via flavor option
cogs publish-python <SourceDirectory> <TargetDirectory> --flavor pydantic

# Or using the publish-pydantic command
cogs publish-pydantic <SourceDirectory> <TargetDirectory>

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

This PR extends the COGS Python publisher to generate Pydantic v2-based Python models in addition to the existing zero-dependency dataclass flavor, and exposes the choice via CLI (publish-py --flavor ...) plus a dedicated publish-pydantic command.

Changes:

• Adds a PythonFlavor switch to PythonPublisher and wires it through the CLI (--flavor + publish-pydantic).
• Introduces a dedicated embedded Pydantic runtime template and a thin PythonPydanticPublisher wrapper.
• Adds docs and new unit/integration tests covering Pydantic generation and round-tripping.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.md	Mentions Pydantic output as an authoritative COGS output.
OVERVIEW.md	Adds a technical overview document (currently includes non-portable local file:///Users/... links).
generateIntegrationTest.bat	Attempts to incorporate Pydantic generation into the integration generation script (current placement/paths are problematic).
docs/source/technical-guide/generation/pydantic.rst	Adds Pydantic generation technical guide content.
docs/source/technical-guide/generation/index.rst	Adds Pydantic page to generation docs index.
docs/source/technical-guide/command-line/publish-pydantic.rst	Documents the new publish-pydantic command.
docs/source/technical-guide/command-line/index.rst	Adds publish-pydantic to command-line docs index.
docs/source/technical-guide/command-line/generated-reference.rst	Updates generated CLI reference output to include --flavor and publish-pydantic.
Cogs.Tests/PythonPydanticPublisherTests.cs	Adds unit tests for Pydantic publisher output shape and behaviors.
Cogs.Tests.Integration/PythonPydanticIntegrationTests.cs	Adds integration test for C# ↔ Python(Pydantic) JSON/XML round-tripping.
Cogs.Publishers/PythonPydantic/Runtime.py	Adds the Pydantic v2 runtime implementation used by generated models.
Cogs.Publishers/PythonPydantic/PythonPydanticPublisher.cs	Adds a wrapper publisher that selects the Pydantic flavor on PythonPublisher.
Cogs.Publishers/Python/Runtime.py	Tightens dataclass runtime internals (field filtering, stream/path I/O handling, etc.).
Cogs.Publishers/Python/PythonPublisher.cs	Adds flavor selection, emits Pydantic fields/pyproject deps, and selects the appropriate embedded runtime.
Cogs.Publishers/Python/PythonFlavor.cs	Introduces the PythonFlavor enum (Dataclass vs Pydantic).
Cogs.Publishers/Cogs.Publishers.csproj	Embeds the new Pydantic runtime as a resource.
Cogs.Console/Program.cs	Adds publish-pydantic and publish-py --flavor.
.gitignore	Adds OS-specific ignores (macOS/Windows explorer artifacts).

Suppressed comments (2)

Cogs.Publishers/PythonPydantic/Runtime.py:1065

• When serializing an item reference to JSON, the runtime checks assignability but ignores allow_subtypes. For properties that disallow subtypes, a derived item instance should be rejected (or serialized as its declared type), but the current code always allows derived instances.

        expected = ITEM_TYPE_REGISTRY[metadata["type_name"]]
        if not isinstance(value, expected):
            raise TypeError(f"Invalid item type for {metadata['cogs_name']}.")
        return value.to_reference_dict()

Cogs.Publishers/PythonPydantic/Runtime.py:1116

• When serializing an item reference to XML, the runtime checks assignability but ignores allow_subtypes. For properties that disallow subtypes, derived items should be rejected to match the COGS property-local subtype rules.

        expected = ITEM_TYPE_REGISTRY[metadata["type_name"]]
        if not isinstance(value, expected):
            raise TypeError(f"Invalid item type for {metadata['cogs_name']}.")

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[kulnor]

@DanSmith, let me know if you need anything else from my end in order to process this PR. Tks!

[DanSmith]

Thanks for the follow-up.

As I mentioned earlier, the existing Python generation should not be duplicated, that will make the codebase harder to maintain over time. Pydantic support should be added as an option (via a flag) to the existing Python publisher rather than as a parallel implementation.

I can see this was partially done, but there is still a separate PythonPydanticPublisher that just wraps the existing Python publisher, along with redundant documentation and the unnecessary PythonPydanticPublisherTests. Ideally the majority of the changes would be limited to:

• Adding the flavor command flag in Cogs.Console/Program.cs
• The small amount of Pydantic-specific customization inside the existing Python\PythonPublisher.cs and Python\Runtime.py

The PythonPydanticIntegrationTests are mostly a direct copy of the existing PythonIntegrationTests with a few changed output file names. Any needed changes should be combined into the existing PythonIntegrationTests, with only the small differences required for exercising the Pydantic flavor.

A main concern, a second Runtime.py is still duplicated from the existing Python publisher Runtime.py and largely overlapping. Since it is an embedded resource and generated at publish time, any small differences needed for Pydantic should be handled via the templating (the same approach used in the C# publisher). Why are there still changes in both Runtime.py files if two copies currently exist? This will require further review on my part.

The pydantic flavor also does not participate in the conformance testing or packaging of the pydantic flavor added to the workflow build. I can help with this once the updated publisher is in good shape.

A couple of other items that were already flagged in the review are still present:

• On the generateIntegrationTest.bat changes, it should simply add one call to the Python publisher using the Pydantic flavor. It should not delete directories and regenerate a large set of outputs.
• Local file:///Users/pascal/Library/CloudStorage/Dropbox/git-dartfx... links in the docs
• Subtype checking that is not fully handled (as noted in the GitHub review comments)

Happy to re-review once the structure is cleaned up along these lines. Let me know if anything above is unclear.

[kulnor]

Thanks for this. As you can see, I do delegate most of this work to agents to maximize my productivity and multitask. This obviously highlights the weaknesses of this AI-driven approach. I will revisit this when I get the chance by including all your requirements in the prompts and instructions and adding a human-in-the-loop review process... Or I'll just go back to the drawing board and write this myself...

…

On Tue, Aug 25, 2026 at 8:12 PM Dan ***@***.***> wrote: *DanSmith* left a comment (Colectica/cogs#364) <#364 (comment)> Thanks for the follow-up. As I mentioned earlier, the existing Python generation should not be duplicated, that will make the codebase harder to maintain over time. Pydantic support should be added as an option (via a flag) to the existing Python publisher rather than as a parallel implementation. I can see this was partially done, but there is still a separate PythonPydanticPublisher that just wraps the existing Python publisher, along with redundant documentation and the unnecessary PythonPydanticPublisherTests. Ideally the majority of the changes would be limited to: - Adding the flavor command flag in Cogs.Console/Program.cs - The small amount of Pydantic-specific customization inside the existing Python\PythonPublisher.cs and Python\Runtime.py The PythonPydanticIntegrationTests are mostly a direct copy of the existing PythonIntegrationTests with a few changed output file names. Any needed changes should be combined into the existing PythonIntegrationTests, with only the small differences required for exercising the Pydantic flavor. A main concern, a second Runtime.py is still duplicated from the existing Python publisher Runtime.py and largely overlapping. Since it is an embedded resource and generated at publish time, any small differences needed for Pydantic should be handled via the templating (the same approach used in the C# publisher). Why are there still changes in both Runtime.py files if two copies currently exist? This will require further review on my part. The pydantic flavor also does not participate in the conformance testing or packaging of the pydantic flavor added to the workflow build. I can help with this once the updated publisher is in good shape. A couple of other items that were already flagged in the review are still present: - On the generateIntegrationTest.bat changes, it should simply add one call to the Python publisher using the Pydantic flavor. It should not delete directories and regenerate a large set of outputs. - Local file:///Users/pascal/Library/CloudStorage/Dropbox/git-dartfx... links in the docs - Subtype checking that is not fully handled (as noted in the GitHub review comments) Happy to re-review once the structure is cleaned up along these lines. Let me know if anything above is unclear. — Reply to this email directly, view it on GitHub <#364?email_source=notifications&email_token=AABUZKT6465E3F7IJWVQNRL5LZBRFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRHE3DGNRVGM4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5419636538>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUZKXYARI5P73ZB5ZRDFD5LZBRFAVCNFSNUABEKJSXA33TNF2G64TZHM4TGMBYHAYTEMJ3JFZXG5LFHM2TCNJVGUYTCNZWGWQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AABUZKRQVPHMNRDW55S3PBT5LZBRFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRHE3DGNRVGM4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AABUZKUFCJCYNEYVM4VKC7L5LZBRFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRHE3DGNRVGM4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>