New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON serialzation #133

Closed
xunto opened this Issue Feb 21, 2018 · 19 comments

Comments

6 participants
@xunto

xunto commented Feb 21, 2018

How to properly turn pydantic schemas into json?

The problem is that the BaseModel.dict() method returns dict, it's okay but it doesn't convert some default types into string (datetime.date, Enum and etc.).

It seems there is no other way to translate pydantic object into dict and serialize those types which is strange as pydantic can deserialize them and even force specific format in some way (datetime_parse.date_re for example which is hardcoded and hardly changeable).

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Feb 21, 2018

pydantic doesn't take care of serialisation because:

  1. people do it in lots of different ways - for example you might know that your data doesn't contain any complex types and therefore want to use ujson or similar, or you might want to use a completely different serialisation protocol like msgpack.
  2. it's very simple - use a custom encoder like this and json.dumps(m.dict(), cls=MyEncoder)

datetime_parse.date_re for example which is hardcoded and hardly changeable

This is unrelated to this issue, if you have a request for a change and a suggestion of how it might work, please create a separate issue. Hint: it is changeable, just add your own validator.

@xunto

This comment has been minimized.

xunto commented Feb 27, 2018

@samuelcolvin thanks for an answer. I know about the way to change it with my own validator (as I reported some bug with it recently).

I'm just trying to understand why pydantic take care of deserialization and doesn't take care of serialisation when your arguments applicable for both of them.

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Feb 27, 2018

I guess we should add a json() method that serialises using a default and best effort encoder class but allows and optional argument of a different encoder.

@xunto

This comment has been minimized.

xunto commented Feb 27, 2018

@samuelcolvin yeah, sounds good.

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Feb 27, 2018

great, PR welcome.

Otherwise, I'll work on it in a couple of weeks. The link about should be a good starting point, the only thing I can thing of to add would be enums. (obviously RowProxy can be removed as that's specific to sqlalchemy)

@xunto

This comment has been minimized.

xunto commented Feb 27, 2018

@samuelcolvin I'm not sure that I'll have time for this but if I will, I would be happy to help.

@samuelcolvin samuelcolvin referenced this issue Mar 25, 2018

Open

Version 1 Release #143

5 of 8 tasks complete
@nandoflorestan

This comment has been minimized.

nandoflorestan commented May 5, 2018

-1.

In my opinion almost every other issue in the github tracker is more important than this (as of May 2018). Serialization depends on intended usage and it can be done independently of pydantic. Getting the other parts right is much more important and urgent.

@nandoflorestan

This comment has been minimized.

nandoflorestan commented May 5, 2018

By the way, the method should not be called "json()", unless it returns a string. Since it returns a dictionary, it should be called "to_dict()". JSON is another step after we have the dictionary.

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented May 5, 2018

Yes, this is what I was saying.

We already have the dict() method which returns a dict.

The problem is that (quite correctly) dict() doesn't return a dict which can necessarily be serialised straight to a dict without a custom serialiser.

@xunto

This comment has been minimized.

xunto commented May 6, 2018

There are a lot of types that will be serialized in the same way from project to project. The point is that nobody wants to implement the same encoder every time and duplicate code.

‎Moreover It's just strange that pydantic can decode string into date but can't encode date into string. It just doesn't make sense. As if you could read but not write. My opinion is that If you strongly against encoding for user, decoding should be removed too.

@dsully

This comment has been minimized.

dsully commented May 20, 2018

+1 to @xunto's reasoning here. Serialization that handles aliases would be extremely nice.

@Gr1N

This comment has been minimized.

Collaborator

Gr1N commented Jun 11, 2018

@samuelcolvin I have simple project (code base mostly taken from Django plus added support for enums) with extended JSONEncoder which can be possible used to solve this issue. What do you think?

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Jun 12, 2018

I think it would be simplest to keep this code inside harrier. Moreover, I disagree with some of django's decisions, eg. it's generally best to encode timedelta as seconds rather than P1DT02H00M03.400000S

@jasonkuhrt

This comment has been minimized.

jasonkuhrt commented Jun 21, 2018

I would welcome a non-intrusive default in pydantic that allows json.dumps(some_model) to just work, without blocking customization. I'm new to Python so take my opinion with grain of salt.

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Jun 21, 2018

It won't quite be json.dumps(some_model) it'll be more like some_model.json() or equivalently json.dumps(some_model, cls=PydanticEncoder).

I'm going to work on this soon.

samuelcolvin added a commit that referenced this issue Jun 28, 2018

@samuelcolvin

This comment has been minimized.

Owner

samuelcolvin commented Jun 28, 2018

this is fixed by #210.

Feedback welcome, I'll wait a few hours then add it to the next deploy.

However, I'm happy to extend the feature in future if I've missed anything.

samuelcolvin added a commit that referenced this issue Jun 28, 2018

implement JSON serialisation (#210)
* implement JSON serialisation, fix #133

* documenting JSON serialisation

* fix coverage
@xunto

This comment has been minimized.

xunto commented Jul 16, 2018

@samuelcolvin just noticed, cool! But what about enums?

@samuelcolvin

This comment has been minimized.

@xunto

This comment has been minimized.

xunto commented Jul 17, 2018

@samuelcolvin oh, sorry. I didn't notice. Cool! Good job!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment