-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Add nested json encoding #3941
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nested json encoding #3941
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks great, a few things to change and will need to wait for v1.10, but doesn't need to wait until v2.
class Config: | ||
json_encoders = { | ||
timedelta: lambda v: v.total_seconds(), | ||
WithCustomEncoders: lambda _: 'using parent encoder', | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this makes the example more complicated than it needs to be - I think it's sufficient just to show that ParentModel
uses child.Config.json_encoders
.
If there's a more subtle point to be made about how ParentModel
interacts, either mention it in the code or add another example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this was highlighting that ParentModel
doesn't use child.Config.json_encoders
when models_as_dict=False
, whether use_nested_encoders=True|False
. That's mentioned in the code as a comment, should it be more explicit?
Co-authored-by: Samuel Colvin <samcolvin@gmail.com>
…nto nest-json-encoders
Once this is ready to review, please use the special phrase in a comment so I know it's my responsibility. |
please review |
@lilyminium i think this will satify this feature request #3910 👍 (i need to serialize date times time deltas and other python stuff) also if you will add tests for negative time deltas there is small bug in it time #3909 so do not do it yet or do it after this fix is merge (but i think curent state will suffice. This is just heads up since i saw it in test used.) |
@lilyminium thanks for working on this, it's a feature I need. When I've tried running these tests using the above code though, if you define any primitive types, string for example it raises a from datetime import datetime, timedelta
from pydantic import BaseModel
from pydantic.json import timedelta_isoformat
class CustomChildModel(BaseModel):
dt: datetime
diff: timedelta
text: str
class Config:
json_encoders = {
datetime: lambda v: v.timestamp(),
timedelta: timedelta_isoformat,
}
class ParentModel(BaseModel):
diff: timedelta
child: CustomChildModel
class Config:
json_encoders = {
timedelta: lambda v: v.total_seconds(),
CustomChildModel: lambda _: 'using parent encoder',
}
child = CustomChildModel(dt=datetime(2032, 6, 1), diff=timedelta(hours=100), text='hello')
parent = ParentModel(diff=timedelta(hours=3), child=child)
# default encoder uses total_seconds() for diff
print(parent.json())
# nested encoder uses isoformat
print(parent.json(use_nested_encoders=True))
# turning off models_as_dict only uses the top-level formatter, however
print(parent.json(models_as_dict=False, use_nested_encoders=True))
print(parent.json(models_as_dict=False, use_nested_encoders=False)) Also, in the following example I couldn't get it to json encode the from datetime import datetime, timedelta
from pydantic import BaseModel
from pydantic.json import timedelta_isoformat
class GrandChildModel(BaseModel):
dt: datetime
diff: timedelta
class CustomChildModel(BaseModel):
dt: datetime
diff: timedelta
grand_child: GrandChildModel
class Config:
json_encoders = {
datetime: lambda v: v.timestamp(),
timedelta: timedelta_isoformat,
GrandChildModel: lambda v: 'foobar',
}
class ParentModel(BaseModel):
diff: timedelta
child: CustomChildModel
class Config:
json_encoders = {
timedelta: lambda v: v.total_seconds(),
CustomChildModel: lambda _: 'using parent encoder',
}
grand_child = GrandChildModel(dt=datetime(2040, 7, 13), diff=timedelta(hours=10))
child = CustomChildModel(dt=datetime(2032, 6, 1), diff=timedelta(hours=100), grand_child=grand_child)
parent = ParentModel(diff=timedelta(hours=3), child=child)
# default encoder uses total_seconds() for diff
print(parent.json())
# nested encoder uses isoformat
print(parent.json(use_nested_encoders=True))
# turning off models_as_dict only uses the top-level formatter, however
print(parent.json(models_as_dict=False, use_nested_encoders=True))
print(parent.json(models_as_dict=False, use_nested_encoders=False)) |
thanks so much for this. |
@samuelcolvin did you see my notes above. Just noticed you merged this. |
So sorry @barringtonhaynes, I've been crashing through PRs and missed your comment. I guess we should either revert this or work on a separate fix. Do you think you could submit a fix? |
I would love to but don't have any time at the moment due to other commitments, it's also possible that I missed something and those use cases do work. It's a super useful feature, but think it needs a little TLC. If it's still around in a month or so I'm happy to have a look. |
Okay, I'll look into it. Thanks for the heads up. And thanks again to @lilyminium for the PR. |
Okay, having looked into it, this introduces a lot of problems, for example even in this very simple case the output is not valid JSON: from pydantic import BaseModel
class Child(BaseModel):
b: str
c: int
class Config:
json_encoders = {
str: lambda v: 'custom value!',
}
class Parent(BaseModel):
a: str
child: Child
x = Parent(a='a', child=dict(b='b', c=1))
print(x.json(use_nested_encoders=True)) Outputs: {"a": "\"a\"", "child": {"b": "\"b\"", "c": "1"}} Note::
I'll revert the PR, and if someone else wants to take a stab at this before V1.10 is locked next week, I'll try and review. But realistically, we'll probably need to wait for V2 when serialisation will be completely re-thought. |
This reverts commit b42fae0.
Huge apologies for falling off the map -- Google was helpfully filtering out a substantial chunk of my GitHub notifications. Thanks for noticing the problems, @barringtonhaynes and sorry for any complications this introduced. |
#3935 contains a pin to jinja; unsure if I should wait on that or pin myself.
Change Summary
encode_as_json
keyword to a bunch of serialisation functions to try getting nested JSON encodersuse_nested_encoders
toBaseModel.json
use_nested_encoders
does not pair withmodels_as_dict=False
; in that case, only the global encoder is usedRelated issue number
fix #2277
Checklist
changes/<pull request or issue id>-<github username>.md
file added describing change(see changes/README.md for details)