Date / time support

[shish]

Most of the methods I've seen for encoding dates and times in JSON seem hacky and non-standard; if you're going to update the standard, doing dates in a nice way would be good.

(I'm not sure what that nice way would be)

[aseemk]

Great point.

Someone pointed me to JSON3, which is still a regular JSON implementation, but which serializes Date objects to ISO 8601 date-time strings.

Interestingly, it seems it doesn't do anything special to parse the generated strings back to Date objects. But maybe it's fine for that to be application-level logic, not sure.

I thought maybe serializing to Date(...) could be an idea, but it turns out that in ES5 (see link above), calling Date(...) without new is actually supposed to ignore the parameters and always return the current date-time.

That leads to the thought of serializing to new Date(...), but that strikes me as a bad idea, putting object construction in a data format.

So maybe the best thing to do is to follow JSON3 in at least serializing dates to a standard format. What do you think?

[aseemk]

It also seems that V8 (in Node and Chrome) does this natively too:

> JSON.stringify(new Date())
"2012-05-29T04:21:20.031Z"

So this is probably the right thing for this JSON5 parser to do too?

[arunoda]

Why we need Date serialization since we have data.getTime() ?

[aseemk]

I think it's for convenience, when Date instances are nested within objects or arrays.

[chilts]

Of course, instead of

JSON.stringify(new Date())

you can always

(new Date()).toISOString();

I kinda like the latter better. :)

[aseemk]

Indeed -- but that's not always convenient to do when the Date is nested. =)

[aseemk]

Actually, I just realized that we've been talking about serialization, and this JSON5 implementation defers to the regular JSON.stringify() for serialization, so it'll already do this in V8.

Need to investigate whether there are still issues there that could be improved, but in general, I'd love to output regular JSON if possible. JSON5's purpose is really to make JSON more write-/maintain-friendly.

[ForbesLindesay]

This input JSON.stringify(new Date());

Gives this output "2012-05-29T13:00:29.886Z" (at least it does in Chrome)

So I think all we need to do is parse that and turn it into a date? It would mean that a string in exactly that format would also be parsed as a date, which isn't ideal. But I think it's fair to say that if you get something in that format it will be a date.

[aseemk]

Someone on the Node.js thread pointed out this great gist:

https://gist.github.com/2504336

Since JSON5's parse() is built off of Douglas Crockford's own parser, it too supports a reviver argument, so that helper code will work with JSON5 too. =)

Let's keep in mind though that the goal of JSON5 is to make writing JSON easier. I'm not sure that ISO 8601 strings are the most human-friendly way to write dates, but I'm not sure that introducing new types is a good idea, either.

[ForbesLindesay]

Hmm, I take your point about not necessarily being easier to write, but long term it might be nice to view this as a potential replacement for traditional JSON even when the JSON is generated, in which case date representation would be high on people's priorities. We could consider supporting dates of the form

{myDateField:16/06/1981, myTimeField:16:08}

The only problem then is you make a lot of decisions (english vs. american, 24hr vs.12hr). ISO saves us from those judgements.

[aseemk]

Yes, that's a fair point, thanks. I'm hesitant to introduce syntax that's no longer "just JS" though.

The reviver example linked above though gives me another idea — we could have an option to e.g. auto-parse Dates, and the reviver could simply try calling Date.parse(str) on every string value. So you could write dates in any human-readable format that Date.parse() recognizes, like "May 29, 2012" or "6:08 PM".

[ForbesLindesay]

I like that idea

[polotek]

In v8 (not sure about other engines), the Date prototype has a toJSON method which the spec for JSON.stringify respects. The toJSON method seems to just return the ISO 8601 string. So in v8 you're already set up to stringify dates properly. You just need to parse them back out. I left this on the node list, but I'll leave it here too.

https://gist.github.com/2504336#gistcomment-281565

Note that this uses try/catch which has perf implications. But I've heard that JSON.parse is already pretty slow anyway :)

[aseemk]

Definitely, thanks Marco!

[shish]

When representing dates as strings, anything other than ISO 8601 is a world of pain, as you get into special cases, custom parsers, and guesswork that differs between implementations...

I'm pretty sure a perfect implementation would be impossible - storing as strings and then attempting to turn all strings into dates would have false positives; storing as non-strings breaks the purity of the format :S

A possibility:

{
    mydate: {
        advanced_type: "date",
        advanced_data: "2012-05-02T12:03"
    }
}

? That would also allow for other advanced data types, though I can't think of any other practical ones. At a stretch, {advanced_type: "image", advanced_data: "base64 encoded pixels go here"}.

That is should be fairly unambiguous and easy to process, I imagine; it is quite a lot more typing than a single string though v.v

[aseemk]

@shish, thanks for the comment. You're right that JSON5 probably shouldn't specify arbitrary or open-ended date string formats.

I don't think using advanced types like that is a good either, though. The purpose of JSON5 is to be human-friendly when writing, and having to specify such things to help the machine parse your data defeats that purpose. =) It also greatly muddies the spec if we start reserving keys.

Let's keep things simple here and stick to the "no new data types" philosophy, and for developer convenience, this JS implementation will add an option to auto-try-parse all strings into JS Date objects, via @polotek's reviver.

[aseemk]

Folks, just a heads-up that I started a JSON5 google group:

https://groups.google.com/group/json5

If you're interested in news and updates to JSON5, it'd be great to have you join. Cheers.

[rlidwka]

You can do this:

{"type":"Date","data":"2013-10-07T18:06:03.048Z"}

Note that starting with node 0.11.x (nodejs/node-v0.x-archive@840a29f) Buffer type is going to be serialized this way:

{"type":"Buffer","data":[116,101,115,116]}

So it has a chance of becoming "kinda standard", translating {type:XXX,data:YYY} to new XXX(YYY). Same could be done for regexps and any user-defined type somebody would want.

[aseemk]

Very interesting, @rlidwka! That's good to know, but in this case, the native Date#toJSON() already outputs an ISO 8601 string (and not an {type: "Date", data: "ISO 8601"} object), so we probably can't expect that to change.

[rlidwka]

but in this case, the native Date#toJSON() already outputs an ISO 8601 string

The key issue here is that when you're doing Date.toJSON(), type information is lost. In other words, you can't possibly make so both these assertions are true while preserving current Date.toJSON() serialization:

var x = new Date();
var y = String(x);
assert.deepEqual(x, JSON.parse(JSON.stringify(x)));
assert.deepEqual(y, JSON.parse(JSON.stringify(y)));

JSON fails first one, reviver suggestion fails second one.

Is that important so you convert a date to JSON5 and back, and get the same date object again? I guess that's the answer we have to answer here.

[emirotin]

Microsoft (sorry for that :)) has it solved like that

1. http://stackoverflow.com/questions/726334/asp-net-mvc-jsonresult-date-format
2. http://msdn.microsoft.com/en-us/library/bb299886.aspx

[ashmind]

I have thought on this for a while.

I see two main user groups:

1. Services generating/consuming JSON that want to pass dates around (especially without defining field schema in advance)
2. People writing JSON by hand (configuration, complex command-line args, etc)

Main problem with strings is that you have to define common schema between sender and receiver. Since only dates need that, it is a really weird limitation. Another problem is that in JavaScript, your JSON object would not represent the same JS object, so either you have to do postprocessing, or include schema support in JSON lib, or do some random hacks based on string format (similar to what MS does).

Here are the options I see for non-string solutions:

1. Push for ISO-8601 date literals in JS vNext (e.g. ES7). This works well for both groups, but obviously hard to do and with unknown timelines.
2. Parse new Date("ISO8601String"). Literally hardcode this on the parser level, so new with anything else or non ISO8601 string parameters are a syntax error. This is valid JS, and works well for web services, but kind of annoying to write manually.

I would much prefer option 1, but if it not realistic, I think that option 2 is a good substitute that solves problems with string approaches.

[jordanbtucker]

I move to close this as a won't fix. This is outside the scope of JSON5. Use the reviver argument of JSON5.parse if you want to restore types other than primitives, objects and arrays.

[aseemk]

I'm not sure personally. I agree that fortunately, there is a simple workaround (reviver as mentioned), but I think lots of good arguments have been made in this thread that adding some type of support aligns with the goals of JSON5: to help people write dates by hand. Tough call!

Let's keep this issue open. I think I'm still open to pull requests that give some implementation a stab and can demonstrate the value.

[jordanbtucker]

mofo syn posted on the JSON5 Google Group:

I wonder if anyone thought about custom data types e.g. ISO8601

{"data": !ISO8601 "2013-10-07T18:06:03.048Z"}`

Above is how yaml would approach this. Obviously not JSON compatible.

Below is how MongelDB approches it:

{"data":{"$date":"2013-10-07T18:06:03.048Z"}}`

---

Would be interesting to see if there is a way to define custom datatypes in a way that is consistent with javascript V5.

I think @shish summed it up nicely:

storing as strings and then attempting to turn all strings into dates would have false positives; storing as non-strings breaks the purity of the format

Also, YAML's method would violate one of the core values:

JSON5 remains a strict subset of JavaScript.

Probably the best solution would be to allow new Date() or Date.parse() as values. When JSON5 is serialized as JSON, it would output the date string, but when serialized as JSON5, it would output new Date() or Date.parse().

BTW, new Date() and Date.parse() are equivalent, so it comes down to which one feels better. I think my vote is for new Date() since it's two characters less.

[jordanbtucker]

Correction, new Date() and Date.parse() are not equivalent, since new Date() returns a Date object and Date.parse() returns a Number.

new Date() may cause some confusion as to what type of parameter is accepted since it has four overloads:

new Date()
new Date(value)
new Date(dateString)
new Date(year, month[, day[, hour[, minutes[, seconds[, milliseconds]]]]])

I think JSON5 should only accept strings as parameters (for readability) so new Date(dateString) would be the only acceptable overload. But we're still left with ambiguity about what date formats are acceptable. I think ISO8601 should be allowed exclusively.

new Date('2015-10-08T10:29:00.000Z') // acceptable
new Date('October 8, 2015 10:29 AM') // unacceptable

Note that stringify will output all ISO8601 fields, but all fields do not need to be specified when parsing:

// all acceptable
new Date('2015')
new Date('2015-10')
new Date('2015-10-08')
new Date('2015-10-08T10:29')
new Date('2015-10-08T10:29Z')
new Date('2015-10-08T10:29-08:00')
new Date('2015-10-08T10:29:30')
new Date('2015-10-08T10:29:30Z')
new Date('2015-10-08T10:29:30-08:00')
new Date('2015-10-08T10:29:30.123')
new Date('2015-10-08T10:29:30.123Z')
new Date('2015-10-08T10:29:30.123-08:00')
new Date('2015-10-08T10:29:30.123456789')       // these last three are acceptable,
new Date('2015-10-08T10:29:30.123456789Z')      // but lose resolution in some environments
new Date('2015-10-08T10:29:30.123456789-08:00') // like Node.js and browsers

Any values without a time zone designation would be treated as UTC.

[tracker1]

An iso format date, with at least yyyy-mm-dd should be parsed and turned into a Date object, it's easy enough to detect.. yes, a hydration script could be used, but at this point the ISO serialization has been standard in ES for over half a decade, so deserializing makes sense, the trouble is the offset, though it's probably best to just use new Date(Date.parse(str)) and leave it at that for detection.

The following regex should match the above serializations.

var reSerializedDate = /^\d{4}\-\d\d-\d\d([T ]\d\d(\:\d\d(\:\d\d(\.\d{3,9)?)?)?(Z|[\+\-]\d\d\:?\d\d)?)?$/;

if (typeof str === 'string' && str.length > 9 && str.length < 36 && reSerializedDate.test(str) {
  return new Date(Date.parse(str));
}

The reason to use Date.parse over just new Date(str) is because Date.parse is more likely to be polyfilled than the Date itself... (I wrote one of the first polyfills for new Date, and it was rather evil). There are better fills for Date.parse out there, and in wider usage.

[jordanbtucker]

I'm in the process of finalizing v1.0.0, so I've taken another look at Date support. I've been testing out an option for parse that will automatically find strings that look like dates and parse them as Date objects. Here's what the API looks like.

const result = JSON5.parse("{dateTime: '2017-09-23T23:53:40.303Z'}", {dates: true})
// `result` is equivalent to { dateTime: new Date('2017-09-23T23:53:40.303Z') }

• The date string must be in ISO format with all fields present and in the UTC time zone. In other words, it must be in the format YYYY-MM-DDTHH:mm:ss.sssZ. This is the same format returned by Date.prototype.toJSON().
• The second argument of parse can be a reviver function or a new options object. The options object can have a reviver property and a dates property. If dates is truthy, then each string will be tested to see if it conforms to the format described earlier, and if so, converted to a Date object.
• If a reviver function is also defined, then it will be called after all date strings have been converted to Date objects.

It has not yet been decided whether this feature will make it into v1.0.0. Please let me know what you think.

This is not an extension to the JSON5 document specification. It is only an extension to the API of this library. JSON5 implementations are not required to implement this API.

You can try this out by using npm install json5@dates.

[aseemk]

I have the same opinion here as #91 (comment).

[jordanbtucker]

See my reply at #91 (comment).

[jordanbtucker]

Here's a polyfill

function json5DatesPolyfill() {
    const JSON5 = require('json5')

    const origParse = JSON5.parse

    JSON5.parse = function parse(text, reviver) {
        let root = origParse(text)

        if (reviver != null && typeof reviver === 'object') {
            const dates = reviver.dates
            reviver = reviver.reviver

            if (dates) {
                root = internalize({ '': root }, '', dateReviver)
            }
        }

        if (typeof reviver === 'function') {
            return internalize({ '': root }, '', reviver)
        }

        return root
    }

    function dateReviver(name, value) {
        if (
            typeof value === 'string' &&
            /^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d\.\d\d\dZ$/.test(value)
        ) {
            return new Date(value)
        }

        return value
    }

    function internalize(holder, name, reviver) {
        const value = holder[name]
        if (value != null && typeof value === 'object') {
            for (const key in value) {
                const replacement = internalize(value, key, reviver)
                if (replacement === undefined) {
                    delete value[key]
                } else {
                    value[key] = replacement
                }
            }
        }

        return reviver.call(holder, name, value)
    }
}

[icefoxen]

Parsing "any string that looks like a Foo" is almost never what you actually want in the end; it sounds like a glorious way to get some very annoying and subtle edge cases.

Yes you would hope that an ISO date string is specific enough to not be a problem but that's still an assumption. Assumptions in specifications is why we need JSON5 over normal JSON in the first place.

[davidwkeith]

A few thoughts:

• Dates are great, but we now have Maps, Sets, URLs, and more.
• Hex literals start with 0x, so there is precedence for 'special strings' defining types in ECMAScript.
• ECMAScript should have other literal formats, it would be nice to have a standard form to define literals.

I agree with the desire to make JSON5 valid ECMAScript, but that doesn't mean we can't advance EMCAScript to meet JSON5's needs. Ultimately we should move both forward in tandem using something like the following syntax:

{
  date: @d 05 October 2011 14:48 UTC, // quotes optional
  map: @m {2:"two"}, // or nested array notation
  object: @o {two, 2}, // or nested array notation
  array: @a [1, 2, 'foo', 'foo'], // brackets optional
  float32array: @f32a [21, 31],  // brackets optional
  float64array: @f64a [21, 31],  // brackets optional
  set: @s [1, 2,'foo'],  // brackets optional
  regex: @r /match/i, // or string, but without flags
  function: @f "(arg1, arg2) {alert('hi'); //Kidding, don't do this.}"
}

Ultimately I am proposing the following:

• @<hint-string> could be an optional way to disambiguate any literal past what the parser would consider the default for a given notation.
• Consider serializing RegExp objects to strings #91 might work better with a hint (see above).
• The hint could be both a short code (@f32a) and long (@float32array) for clarity.
• Parse errors for anything that does not match the <hint-string> object type.

I chose @ as it matches the Obj-C literal identifier, there may be better choices given the additional information we would want to convey. I can't think of other language conventions at this time.

[icefoxen]

Or you could use type constructors as have been used by languages based off of formal type systems for the past 45 years:

{
  date: Date("05 October 2011 14:48 UTC"),
  map: Map({2:"two"}),
  object: Object({two, 2}),
  array: [1, 2, 'foo', 'foo'],
  float32array: F32([21, 31]),
  float64array: F64([21, 31]),
  set: Set([1, 2,'foo']),
  regex: Regex("/match/i"),
  function: Function(arg1, arg2, null)
}

Is it pretty? Eh, not really. Is it simple, unambiguous, extensible, easy to read, easy to parse, and used widely in other languages, including Elm, Rust, and type constructors for just about everything else? Yes. How difficult does one really need to make things?

That said, it's probably not valid JS, so.

[dcleao]

Finally... Thanks @davidwkeith (although that syntax...) and @icefoxen!
Besides human friendliness, we definitely need an extensible serialization format...
The case we all miss the most is surely Date. Afterwards, there's Regex and Function.
This could be feasible and safe, as long as the acceptable function names were white-listable and given to the parser. Maybe, even, these should be given to it, as a map of name -> function, so that we don't depend on global variables.
A reviver will never be able to do this, alone, without a separate/implied/fixed schema.

[icefoxen]

Serialization of functions seems like a special case that increases the hazards involved immsensely. At that point it's not a matter of "move this data from point A to point B", it becomes "move this program from point A to point B". Doing that safely in a way general enough to be useful is not, as yet, an easily solved problem.

I'd rather go for the easy 80%, or, since JSON already did that, the mostly-easy 95%, than the people-have-been-trying-for-20-years-and-it-still-sucks 100%.

edit: I just realized I may be entirely misunderstanding what you meant, so, disregard this if so. :-)

[dcleao]

When I talked about functions, I was referring to your use of Function(arg1, arg2, ..., body), which I presumed would end up being "revived" with the Function constructor. So yes, this would allow creating functions and any other objects, but note: it is up to you what you would pass to JSON5.parse(jsonString, env), in an environment argument. You also know the source of jsonString; if it is trustworthy or not.

To make it super clear what I think your proposal would sum up to:

var jsonString = "{" + 
"  prop1: Function(\"a\", "\b", \"return a + b;\")" + 
"}";

var env = {
  "Function": function() {
     return Function.apply(null, arguments);
  }
};

var result = JSON5.parse(jsonString, env);

assert(typeof result === "function");
assert(result.prop1(1, 2) === 3);

This is just as dangerous as the existing reviver can be...
The only difference is that there would be a special JSON syntax that would make the feature much more interesting. This syntax would make it much more probable that people would end up using common conventions for serialization of dates, etc. Right now, some use {"type": "Date", value: "2004-01-01"}, others use {"_": "Date", v: "2004-01-01"}, others use "/Date(...)/", etc.

Even if you say that, generally, this is unsafe, there are applications where this is a required risk, like component/dashboard editors, where users do specify functions, and these need to be serialized and deserialized.

[jordanbtucker]

Hey, everyone. I'm going to close this issue because it is incompatible with the official specification. If you would like to continue this discussion, please open an issue on the official specification repository. Please see the new issues template for more information. Thanks.

[jordanbtucker]

This discussion has been continued on the official specification repository.