Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support msgspec output #1551

Merged
merged 18 commits into from
Sep 23, 2023
Merged

Conversation

indrat
Copy link
Contributor

@indrat indrat commented Sep 16, 2023

Adds basic support for msgspec.Struct as an output format which possibly addresses #1278

I have not used this in anger yet, only to transform some of the examples so I cannot say whether it is completely correct but I figured it was a good idea to push it up for review. I basically followed the bouncing ball of changes from the previous two PRs to add dataclass and TypedDict support.

Things that I haven't figured out:

Class args

Class args to Struct for things like kw_only, forbid_unknown_fields, or omit_defaults, i.e

class Pet(Struct, kw_only=True, omit_defaults=True, forbid_unknown_fields=True):
    id: int
    name: str
    tag: Optional[str] = None

this is probably relevant for the dataclasses output as well for passing args to the @dataclass decorator

discriminated unions

Somewhat relatedly - discriminated unions. msgspec uses class args, tag and tag_field, to denote when Structs participate in a tagged Union whereas the pydantic output includes that information on the Field. Not sure how to solve this one. Ideally output similar to the ClassVar approach in jcrist/msgspec#338 (comment)

pydantic output

# pydantic output of jsonschema discriminator_literals.json
class Type1(BaseModel):
    type_: Literal['a'] = Field('a', title='Type ')

class Type2(BaseModel):
    type_: Literal['b'] = Field('b', title='Type ')

class Response(BaseModel):
    inner: Union[Type1, Type2] = Field(..., discriminator='type_', title='Inner')

with this branch msgspec output has no way to add the tag or tag_field class args.

# msgspec output of jsonschema discriminator_literals.json
class Type1(Struct):
    type_: Annotated[Literal['a'], Meta(title='Type ')] = 'a'

class Type2(Struct):
    type_: Annotated[Literal['b'], Meta(title='Type ')] = 'b'

class Response(Struct):
    inner: Annotated[Union[Type1, Type2], Meta(title='Inner')]

Possible approach using ClassVar from jcrist/msgspec#338 (comment) requires being able to add class args like tag / tag_field.

class Base(msgspec.Struct, tag_field='type_'):
    type_: ClassVar[EventType]

class Type1(Struct, tag='a',):
    type_: Annotated[Literal['a'], Meta(title='Type ')] = 'a'

class Type2(Struct, tag='b'):
    type_: Annotated[Literal['b'], Meta(title='Type ')] = 'b'

class Response(Struct):
    inner: Annotated[Union[Type1, Type2], Meta(title='Inner')]

@codecov
Copy link

codecov bot commented Sep 16, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (e9b6edf) 100.00% compared to head (63160f6) 100.00%.
Report is 723 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##            master     #1551     +/-   ##
===========================================
  Coverage   100.00%   100.00%             
===========================================
  Files           11        34     +23     
  Lines         1020      3727   +2707     
  Branches       201       873    +672     
===========================================
+ Hits          1020      3727   +2707     
Flag Coverage Δ
unittests 99.65% <99.60%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
datamodel_code_generator/__init__.py 100.00% <100.00%> (ø)
datamodel_code_generator/__main__.py 100.00% <100.00%> (ø)
datamodel_code_generator/arguments.py 100.00% <100.00%> (ø)
datamodel_code_generator/format.py 100.00% <100.00%> (ø)
datamodel_code_generator/http.py 100.00% <100.00%> (ø)
datamodel_code_generator/imports.py 100.00% <100.00%> (ø)
datamodel_code_generator/model/__init__.py 100.00% <100.00%> (ø)
datamodel_code_generator/model/base.py 100.00% <100.00%> (ø)
datamodel_code_generator/model/dataclass.py 100.00% <100.00%> (ø)
datamodel_code_generator/model/enum.py 100.00% <100.00%> (ø)
... and 24 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@@ -283,6 +283,9 @@ def merge_args(self, args: Namespace) -> None:
if getattr(args, f) is not None
}

if set_args.get('output_model_type') == DataModelType.MsgspecStruct.value:
set_args['use_annotated'] = True
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@indrat
Doesn't msgspec use Field? and is it always annotated?
You define some logic for Field. But, the unit tests don't cover the parts.
We should change the coverage to 100% before merging.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added some output tests for when a msgspec.field is generated and tried to boost the coverage for the const/alias fields as well.

Regarding forcinguse_annotated I believe the constraints such as gt, lt, etc must be attached using Annotated[..., Meta(gt=..., lt=...)] etc. So aiming for the least surprise I felt it made sense to always force use_annotated on otherwise you won't get the constraints or description metadata generated.

msgspec.field only accepts default,default_factory, and name so it's a little different to pydantic.Field

@koxudaxi
Copy link
Owner

@indrat
Thank you for creating the great PR.
I fix the isort problem of unittest.

The unittests don't cover all lines.
I left one comment.
Could you please check it?

@koxudaxi
Copy link
Owner

Thank you very much!!

@koxudaxi koxudaxi merged commit bac8868 into koxudaxi:master Sep 23, 2023
73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants