Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new JSON formats for some DAML-LF data #2463

Merged
merged 18 commits into from
Aug 13, 2019
Merged

Conversation

S11001001
Copy link
Contributor

@S11001001 S11001001 commented Aug 8, 2019

The ApiCodecCompressed uses some formats by default we don't want to use anymore, like {"Some": {}} for Some (). We'll make the encoder use new formats, keeping the branches for the previous formats whenever no inconsistency arises, but adding new branches.

  • int64s from JsNumber
  • int64 to Number encode mode
  • decimals from JsNumber
  • decimal to Number encode mode
  • nullable root Optional
  • nested Optional as lists
  • decimal banker's rounding
  • reject non-Z timestamps
  • absent record fields in object syntax are None

Fixes #2361. Documented in #2519.

Pull Request Checklist

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

- restores totality checking for interface ADT
- makes alternate input formats obvious at a glance
- lets us restructure in the future to cache type branch lookups
@S11001001 S11001001 marked this pull request as ready for review August 12, 2019 21:30
@S11001001 S11001001 requested a review from leo-da as a code owner August 12, 2019 21:30
Copy link
Contributor

@rautenrieth-da rautenrieth-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the use cases for enabling encodeDecimalAsString and encodeInt64AsString? Storing data in a database? Serializing numeric data as JsNumber is not safe unless you have total control over who is going to deserialize the JSON object (now or in the future).

FYI, we used to treat the Navigator GraphQL API (which the JSON encoding is part of) as a stable API, but we gave up on this a while back when we decided not to offer the Navigator code base as a framework for writing UIs. Doing breaking changes in the Navigator JSON encoding is fine as far as Navigator is concerned.

V.ValueOptional(Some(jsValueToApiValue(v, prim.typArgs.head, defs)))
case Some(_) => deserializationError(s"Can't read ${value.prettyPrint} as Optional")
case None => deserializationError(s"Can't read ${value.prettyPrint} as Optional")
(prim.typ, value).match2 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using match2 is to preserve exhaustiveness checking? Or is there any other advantage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes; additionally, we might decide to resolve the first match layer in advance of having the value in the future, notwithstanding DAML recursive cases.

val id = Model.DamlLfIdentifier(
typeCon.name.identifier.packageId,
typeCon.name.identifier.qualifiedName)
val id = typeCon.name.identifier
// val dt = typeCon.instantiate(defs(id).getOrElse(deserializationError(s"Type $id not found")))
val dt = Model.damlLfInstantiate(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this commit: AFAIR Model.damlLfInstantiate is a copy of typeCon.instantiate, introduced when the daml.lf version had bug and Navigator could not upgrade to the latest daml.lf library version.

New issue for fixing this: #2506

@S11001001
Copy link
Contributor Author

What are the use cases for enabling encodeDecimalAsString and encodeInt64AsString? Storing data in a database?

@rautenrieth-da That's a good one. We do plan to replace extractor's JSON encoder with this one.

The real reason it appears in this PR is "that's what's written in the spec".

Serializing numeric data as JsNumber is not safe unless you have total control over who is going to deserialize the JSON object (now or in the future).

I could equally say that a deserializer could discard every character in a number-string after the seventh. This is not without precedent 🙂 so I wouldn't say this is a reason it's "unsafe".

Instead, I would say that our choice of default for the HTTP JSON API (and Navigator) "avoids common mistakes", whereas alternate cases e.g. extractor will simply make alternate choices.

Even in the cases you're referring to, it's not necessarily a concern that some data loss might occur, because not all use cases require the JSON encoding of contracts to be injective. Consider an application that charts some decimal values from a set of contracts, for example. We might want to provide such applications a way to [explicitly] request number encoding in the future.

Copy link
Contributor

@leo-da leo-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@S11001001
Copy link
Contributor Author

@leo-da Please check b4c0072, too, which was added after your review.

@leo-da
Copy link
Contributor

leo-da commented Aug 13, 2019

Even in the cases you're referring to, it's not necessarily a concern that some data loss might occur, because not all use cases require the JSON encoding of contracts to be injective. Consider an application that charts some decimal values from a set of contracts, for example. We might want to provide such applications a way to [explicitly] request number encoding in the future.

@S11001001 I don't think we can make it injective, there always will be a case when
DAML Decimal 9007199254740992 represented as JSON String: "9007199254740992"
and the case when
DAML Text "9007199254740992" represented as the same JSON String "9007199254740992"
or do you mean we that there will be cases when we don't need to operate with large numbers?

@rautenrieth-da see javascript Number.MAX_SAFE_INTEGER https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER
this explains why we want Strings.

@S11001001
Copy link
Contributor Author

@leo-da I should have said "JSON encoding of contracts of the same type".

@leo-da
Copy link
Contributor

leo-da commented Aug 13, 2019

In any case if it is up to me, I would keep the option to represent DAML Decimal as JsNumber...

@S11001001 S11001001 merged commit 0d72f84 into master Aug 13, 2019
@S11001001 S11001001 deleted the 2361-new-lf-value-json-formats branch August 13, 2019 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enhance Navigator's lf-value-json encoding to fully support DAML LF primitives
3 participants