New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for `datetime` #86

Closed
mrocklin opened this Issue Feb 13, 2014 · 23 comments

Comments

Projects
None yet
5 participants
@mrocklin

mrocklin commented Feb 13, 2014

Should simplejson support datetime objects? If so then what is the natural encoding?

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 13, 2014

Member

There is no natural encoding because the JSON spec doesn't have one. For some use cases it's appropriate to encode them as a number (epoch seconds OR epoch milliseconds are both in common use), for others an ISO-8601 string is more appropriate. simplejson can't know what the context is so it can't make an appropriate choice. There isn't even a real de facto standard that I'm aware of.

Member

etrepum commented Feb 13, 2014

There is no natural encoding because the JSON spec doesn't have one. For some use cases it's appropriate to encode them as a number (epoch seconds OR epoch milliseconds are both in common use), for others an ISO-8601 string is more appropriate. simplejson can't know what the context is so it can't make an appropriate choice. There isn't even a real de facto standard that I'm aware of.

@etrepum etrepum closed this Feb 13, 2014

@mrocklin

This comment has been minimized.

Show comment
Hide comment
@mrocklin

mrocklin Feb 13, 2014

:( Thanks for the explanation.

mrocklin commented Feb 13, 2014

:( Thanks for the explanation.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Feb 13, 2014

Member

Note that for a given use case, it's pretty easy to specify a default argument to dumps to do the encoding. On the decoding side you're pretty much out of luck, you need to use something higher level to do a post-process to handle that kind of "schema" if it's encoded as a string or number.

Member

etrepum commented Feb 13, 2014

Note that for a given use case, it's pretty easy to specify a default argument to dumps to do the encoding. On the decoding side you're pretty much out of luck, you need to use something higher level to do a post-process to handle that kind of "schema" if it's encoded as a string or number.

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

I implemented this stuff several times, first as a custom version of cjson, later on top of simplejson, and again today, in my own fork, this time both as pure Python and in the C speedups, simplifying my previous cjson implementation.

If I agree with etrepum when he says "There is no natural encoding", IMHO the ISO format is the closer we can hope to a de facto standard: it is surely common enough, also in the JS framework around nowadays, that justified my effort.

If there is a chance you would reconsider the option, I'll happily adapt my implementation to your desire.

Hope this helps (you too), thanks&bye.

lelit commented Mar 13, 2014

I implemented this stuff several times, first as a custom version of cjson, later on top of simplejson, and again today, in my own fork, this time both as pure Python and in the C speedups, simplifying my previous cjson implementation.

If I agree with etrepum when he says "There is no natural encoding", IMHO the ISO format is the closer we can hope to a de facto standard: it is surely common enough, also in the JS framework around nowadays, that justified my effort.

If there is a chance you would reconsider the option, I'll happily adapt my implementation to your desire.

Hope this helps (you too), thanks&bye.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Mar 13, 2014

Member

I think you're right that if there is going to be a de facto standard, it will be ISO. If for no other reason than that's what JSON.stringify(new Date()) does in Javascript these days. (e.g. "2014-03-13T16:00:18.863Z").

I'd be willing to accept a pull request for datetime encoding if it was disabled by default (for compatibility reasons). The output must be ISO-8601 formatted in UTC (Zulu time), even if the datetime object doesn't have timezone info attached. Should work like this:

>>> import datetime
>>> import simplejson
>>> simplejson.dumps(datetime.datetime.now())
…
TypeError: …
>>> simplejson.dumps(datetime.datetime.now(), iso_datetime=True)
"2014-03-13T16:00:18.863Z"

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toJSON

Member

etrepum commented Mar 13, 2014

I think you're right that if there is going to be a de facto standard, it will be ISO. If for no other reason than that's what JSON.stringify(new Date()) does in Javascript these days. (e.g. "2014-03-13T16:00:18.863Z").

I'd be willing to accept a pull request for datetime encoding if it was disabled by default (for compatibility reasons). The output must be ISO-8601 formatted in UTC (Zulu time), even if the datetime object doesn't have timezone info attached. Should work like this:

>>> import datetime
>>> import simplejson
>>> simplejson.dumps(datetime.datetime.now())
…
TypeError: …
>>> simplejson.dumps(datetime.datetime.now(), iso_datetime=True)
"2014-03-13T16:00:18.863Z"

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toJSON

@etrepum etrepum reopened this Mar 13, 2014

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

Great, I will then tweak the implementation to match your suggestion! Thank you.

lelit commented Mar 13, 2014

Great, I will then tweak the implementation to match your suggestion! Thank you.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

simplejson.dumps(datetime.datetime.now(), iso_datetime=True)

"2014-03-13T16:00:18.863Z"

You already can do this with the default parameter.

json.dumps([datetime.datetime.now()], default=datetime.datetime.isoformat)
'["2014-03-13T18:27:14.372063"]'

Contributor

serhiy-storchaka commented Mar 13, 2014

simplejson.dumps(datetime.datetime.now(), iso_datetime=True)

"2014-03-13T16:00:18.863Z"

You already can do this with the default parameter.

json.dumps([datetime.datetime.now()], default=datetime.datetime.isoformat)
'["2014-03-13T18:27:14.372063"]'

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

Well, what would happen if what is dumped is a more complex structure containing also simple dates?

lelit commented Mar 13, 2014

Well, what would happen if what is dumped is a more complex structure containing also simple dates?

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

etrepum, is that a requirement to dump only the millisecs, and not microsecs as Python supports? I know that most JS engine does not handle microsecs...

lelit commented Mar 13, 2014

etrepum, is that a requirement to dump only the millisecs, and not microsecs as Python supports? I know that most JS engine does not handle microsecs...

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

Well, what would happen if what is dumped is a more complex structure
containing also simple dates?

Then use lambda d: d.isoformat(), or operator.methodcaller('isoformat'), or
more general function:

def conv(x):
if isinstance(x, datetime.datetime):
return datetime.datetime.isoformat(x)
if isinstance(x, datetime.date):
return datetime.date.isoformat(x)
...
raise TypeError

Contributor

serhiy-storchaka commented Mar 13, 2014

Well, what would happen if what is dumped is a more complex structure
containing also simple dates?

Then use lambda d: d.isoformat(), or operator.methodcaller('isoformat'), or
more general function:

def conv(x):
if isinstance(x, datetime.datetime):
return datetime.datetime.isoformat(x)
if isinstance(x, datetime.date):
return datetime.date.isoformat(x)
...
raise TypeError

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

Serhiy, I know that (as said, I already implemented several time the datetime support, and given a link to the pure Python solution I'm currently using), but I think that the point here is having such support out-of-the-box, possibly supported at the C level, and with the ability to load as well...

lelit commented Mar 13, 2014

Serhiy, I know that (as said, I already implemented several time the datetime support, and given a link to the pure Python solution I'm currently using), but I think that the point here is having such support out-of-the-box, possibly supported at the C level, and with the ability to load as well...

@mrocklin

This comment has been minimized.

Show comment
Hide comment
@mrocklin

mrocklin Mar 13, 2014

Personally I would enjoy seeing datetimes handled by default. I use default for my own custom types. Trying to shove lots of functionality in through that one keyword quickly becomes awkward.

The default interface also requires some understanding on the part of the user. While those present are obviously capable of handling this a more novice user might be turned away.

mrocklin commented Mar 13, 2014

Personally I would enjoy seeing datetimes handled by default. I use default for my own custom types. Trying to shove lots of functionality in through that one keyword quickly becomes awkward.

The default interface also requires some understanding on the part of the user. While those present are obviously capable of handling this a more novice user might be turned away.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

On my opinion types which are not specified in RFC 4627 shouldn't be supported,
especially if this conversion is not reversible. This conversion looks
application specific.

Contributor

serhiy-storchaka commented Mar 13, 2014

On my opinion types which are not specified in RFC 4627 shouldn't be supported,
especially if this conversion is not reversible. This conversion looks
application specific.

@lelit

This comment has been minimized.

Show comment
Hide comment
@lelit

lelit Mar 13, 2014

What do you mean with "is not reversible"?

lelit commented Mar 13, 2014

What do you mean with "is not reversible"?

@mrocklin

This comment has been minimized.

Show comment
Hide comment
@mrocklin

mrocklin Mar 13, 2014

On my opinion types which are not specified in RFC 4627 shouldn't be supported, especially if this conversion is not reversible. This conversion looks application specific

This sounds like a reasonable concern to me. I'm only in favor of disregarding official standards if de facto standards exist. This is only a reasonable approach to the extent that other systems also support it or are likely to do so in the near future.

mrocklin commented Mar 13, 2014

On my opinion types which are not specified in RFC 4627 shouldn't be supported, especially if this conversion is not reversible. This conversion looks application specific

This sounds like a reasonable concern to me. I'm only in favor of disregarding official standards if de facto standards exist. This is only a reasonable approach to the extent that other systems also support it or are likely to do so in the near future.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Mar 13, 2014

Member

@serhiy-storchaka isoformat does not include the time zone by default, which is not acceptable. Javascript's encoding for Date objects is a de jure standard for that language. It's outside the JSON spec and not backwards compatible with current versions of simplejson, which is why this would be behind an option that with False as the default for the foreseeable future.

Member

etrepum commented Mar 13, 2014

@serhiy-storchaka isoformat does not include the time zone by default, which is not acceptable. Javascript's encoding for Date objects is a de jure standard for that language. It's outside the JSON spec and not backwards compatible with current versions of simplejson, which is why this would be behind an option that with False as the default for the foreseeable future.

@shakefu

This comment has been minimized.

Show comment
Hide comment
@shakefu

shakefu Mar 13, 2014

Contributor

Since there's no standard, I don't see why you wouldn't just wrap
dump/dumps within your own project to serialize to ISO, unix timestamp, or
JS millisecond timestamp, depending on what your project uses? So +1 for
this being application specific.

FYI, if you want an alternate solution, I implemented datetime encoding to
ISO in Pytool (based on simplejson) using the for_json and default hooks:
http://pytool.readthedocs.org/en/latest/pytool.html#module-pytool.json

/shamelessplug

You could easily lift the code from Pytool for whatever project you want if
you don't want extra dependencies:
http://pytool.readthedocs.org/en/latest/_modules/pytool/json.html

Jake Alheid
http://about.me/jacob.alheid

On Thu, Mar 13, 2014 at 10:51 AM, Bob Ippolito notifications@github.comwrote:

@serhiy-storchaka https://github.com/serhiy-storchaka isoformat does
not include the time zone by default, which is not acceptable. Javascript's
encoding for Date objects is a de jure standard for that language. It's
outside the JSON spec and not backwards compatible with current versions of
simplejson, which is why this would be behind an option that with False as
the default for the foreseeable future.

Reply to this email directly or view it on GitHubhttps://github.com//issues/86#issuecomment-37564368
.

Contributor

shakefu commented Mar 13, 2014

Since there's no standard, I don't see why you wouldn't just wrap
dump/dumps within your own project to serialize to ISO, unix timestamp, or
JS millisecond timestamp, depending on what your project uses? So +1 for
this being application specific.

FYI, if you want an alternate solution, I implemented datetime encoding to
ISO in Pytool (based on simplejson) using the for_json and default hooks:
http://pytool.readthedocs.org/en/latest/pytool.html#module-pytool.json

/shamelessplug

You could easily lift the code from Pytool for whatever project you want if
you don't want extra dependencies:
http://pytool.readthedocs.org/en/latest/_modules/pytool/json.html

Jake Alheid
http://about.me/jacob.alheid

On Thu, Mar 13, 2014 at 10:51 AM, Bob Ippolito notifications@github.comwrote:

@serhiy-storchaka https://github.com/serhiy-storchaka isoformat does
not include the time zone by default, which is not acceptable. Javascript's
encoding for Date objects is a de jure standard for that language. It's
outside the JSON spec and not backwards compatible with current versions of
simplejson, which is why this would be behind an option that with False as
the default for the foreseeable future.

Reply to this email directly or view it on GitHubhttps://github.com//issues/86#issuecomment-37564368
.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

@serhiy-storchaka isoformat does not include the time zone by default, which
is not acceptable.

datetime.datetime.now() is naive datetime object and doesn't contain the time
zone.

Contributor

serhiy-storchaka commented Mar 13, 2014

@serhiy-storchaka isoformat does not include the time zone by default, which
is not acceptable.

datetime.datetime.now() is naive datetime object and doesn't contain the time
zone.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

What do you mean with "is not reversible"?

You can't automatically deserialize serialized datetime object because you
can't distinguish serialized datetime from serialized string. All this is
application specific, only your application knowns which string should be then
converted to datetime.

Contributor

serhiy-storchaka commented Mar 13, 2014

What do you mean with "is not reversible"?

You can't automatically deserialize serialized datetime object because you
can't distinguish serialized datetime from serialized string. All this is
application specific, only your application knowns which string should be then
converted to datetime.

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Mar 13, 2014

Member

@serhiy-storchaka well, I won't accept a PR that doesn't include a time zone in the output. A reasonable default is to assume that a naive datetime object is in the local TZ. There should be no ambiguity in the output.

Yes, it is not a trivially reversible encoding, but this is the tradeoff made in the ECMA standard for Javascript's JSON implementation (which almost makes it a de facto standard) and it is reasonable for us to do the same. I don't want to enable this by default, but I will accept a PR to implement it behind an option that defaults to False.

Member

etrepum commented Mar 13, 2014

@serhiy-storchaka well, I won't accept a PR that doesn't include a time zone in the output. A reasonable default is to assume that a naive datetime object is in the local TZ. There should be no ambiguity in the output.

Yes, it is not a trivially reversible encoding, but this is the tradeoff made in the ECMA standard for Javascript's JSON implementation (which almost makes it a de facto standard) and it is reasonable for us to do the same. I don't want to enable this by default, but I will accept a PR to implement it behind an option that defaults to False.

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Mar 13, 2014

Contributor

I worry about API overburdening. Wouldn't it be more flexible to provide
several helper functions and a way to combine them? For example:

simplejson.dumps((datetime.datetime.now(), decimal.Decima('0.10'), myobj()),
defaults=[
(datetime.datetime, simplejson.datetime_formatter(localtimezone)),
(decimal.Decima: simplejson.decimal_formatter)
(tuple: simplejson.tuple_formatter)
(myobj, myobj_formatter),
])

Contributor

serhiy-storchaka commented Mar 13, 2014

I worry about API overburdening. Wouldn't it be more flexible to provide
several helper functions and a way to combine them? For example:

simplejson.dumps((datetime.datetime.now(), decimal.Decima('0.10'), myobj()),
defaults=[
(datetime.datetime, simplejson.datetime_formatter(localtimezone)),
(decimal.Decima: simplejson.decimal_formatter)
(tuple: simplejson.tuple_formatter)
(myobj, myobj_formatter),
])

@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Mar 13, 2014

Member

The tuple and decimal ships have already sailed, they're both built-in and the encoders have been enabled by default since 2011 (but can be turned off for backwards compatibility). There's already a precedent for this kind of one-way transformation.

>>> import simplejson
>>> simplejson.dumps(('foo', 'bar', decimal.Decimal('1.0')))
'["foo", "bar", 1.0]'

I'm not opposed to coming up with a set of sensible default implementations and a way to compose them. It could basically look like this:

def defaults(fns):
    def _defaults(o):
        for fn in fns:
            try:
                return fn(o)
            except TypeError:
                pass
        raise TypeError(repr(o) + ' is not JSON serializable')
    return _defaults
Member

etrepum commented Mar 13, 2014

The tuple and decimal ships have already sailed, they're both built-in and the encoders have been enabled by default since 2011 (but can be turned off for backwards compatibility). There's already a precedent for this kind of one-way transformation.

>>> import simplejson
>>> simplejson.dumps(('foo', 'bar', decimal.Decimal('1.0')))
'["foo", "bar", 1.0]'

I'm not opposed to coming up with a set of sensible default implementations and a way to compose them. It could basically look like this:

def defaults(fns):
    def _defaults(o):
        for fn in fns:
            try:
                return fn(o)
            except TypeError:
                pass
        raise TypeError(repr(o) + ' is not JSON serializable')
    return _defaults

lelit added a commit to lelit/nssjson that referenced this issue Mar 14, 2014

Comply with Bob's requirements expressed in issue simplejson#86
* rename the flag to activate the new behaviour from "handle_datetime"
  to a more clear "iso_datetime"

* serialize datetimes and times with an ending "Z" to make it explicit
  that they are naive values; the parser does recognize either forms
  though

Correct also the Python 3 variant of the C speedups to properly handle
the different FSR internal representations.
@etrepum

This comment has been minimized.

Show comment
Hide comment
@etrepum

etrepum Mar 14, 2014

Member

I'm going to close this for now. @lelit wanted it to automatically interpret strings as datetime on load, which is not something I would like to add. I am only considering encode support, not decode. If another champion for this feature comes along, we can re-open.

Member

etrepum commented Mar 14, 2014

I'm going to close this for now. @lelit wanted it to automatically interpret strings as datetime on load, which is not something I would like to add. I am only considering encode support, not decode. If another champion for this feature comes along, we can re-open.

@etrepum etrepum closed this Mar 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment