Adding standard Json AST - https://github.com/mdedetrich/scala-json-ast #28

Closed
wants to merge 7 commits into
from

Projects

None yet
@mdedetrich

READ THIS FIRST

This discussion is about designing a library for a representation of JSON that can be shared between many different implementations of JSON parsing/pretty printing/transformation/.... As it's a SLIP (and not a SIP, so no language changes!), this shared API may eventually ship as a module that's included in the standard set of Scala libraries (e.g., it would be like a modern, minimalist scala-xml without the language support).

Everyone's welcome to join the discussion, provided your contribution is truly a contribution to the precise topic of this SLIP, as I tried to capture it above. This SLIP is not trying to solve the general problem of how to modularize the standard library, for example. I'm happy to refine my summary if needed. As always, there are plenty of other channels to discuss other (related) topics.

To ensure a productive discussion, please be kind to each other, stay as focussed as possible, and refrain from comments with very little technical content. As we are still refining this SLIP, it's too early to vote.

Thanks,
@adriaanm

@mdedetrich mdedetrich referenced this pull request in json4s/json4s-ast Nov 4, 2015
Open

Submit for SLIP #12

@xuwei-k
xuwei-k commented Nov 4, 2015

-1

@mdedetrich

-1

?

@SethTisue
Member

@sjrd want to weigh in on the Scala.js aspects of this, and/or call the Scala.js community's attention to it?

@sjrd
Member
sjrd commented Nov 4, 2015

Sure, I already did a surface reading of the proposal. I will have a deeper look at it later.
I'll also mention the PR on the Scala.js gitter channel.

@SethTisue
Member

This question about key ordering in objects is interesting. In the proposal fast preserves key order, safe doesn't. I had expected the opposite.

If it's possible to preserve key order and still get good performance, great! But then I would expect safe to preserve it, too. (After all, the spec doesn't actually forbid order preservation.)

I imagine this will be the subject of further discussion...

@xuwei-k
xuwei-k commented Nov 4, 2015

I believe Scala stdlib does not need json library.

@odersky
odersky commented Nov 4, 2015

@xuwei-k We see that sort comments a lot from the scalaz people. I must admit it is getting annoying. If you have serious concerns about a proposal let's hear them. But I am not interested in the commonly voiced opinion that no improvements should be made to the standard library. That's just obstructionism. It's like me telling you there should be no scalaz, I guess you would not like that either.

@hepin1989

I think a standard way to work with json would be great,there are so many json there in the scala community,and look at the https://github.com/tminglei/slick-pg,it write extends for json4s, play-json, spray-json and argonaut json,and on the play side,I once looked at the google-group,@jroper,said there maybe go with json4j in Play3.0.and the current json4s way I think it's great,share some common thing in the scala community would really help,especially for those new comers.

@milessabin

-1.

This is downright damaging when the FOSS community already contains several first class implementations, each with its own particular emphases and design goals, and all in use in production in systems large and small.

If you want to promote Scala for Json processing, get behind those libraries. By all means fork or create another one if none of the existing ones exactly meet your needs, but then recognize that you're doing this because one size doesn't fit all, which is precisely why promoting one as a so-called "standard" is harmful.

@lihaoyi
lihaoyi commented Nov 4, 2015

We see that sort comments a lot from the scalaz people. I must admit it is getting annoying.

+1. It's totally unconstructive, and adds nothing to discussion.

It's even more annoying because oftentimes the objections come out before the exact thing being proposed has even been decided, which is totally ridiculous because by that point you clearly haven't even considered the proposal (because there isn't any yet!)


@hepin1989 OTOH this won't actually solve your use case. This is purely for interop in JSON ASTs, and doesn't provide any facilities for parsing or serializing.

IMHO that makes it considerably less useful. I don't find myself passing around parsed but un-converted JSON ASTs around enough for it to be a hassle; most of my work involves going straight from String -> case class or case class -> string, but maybe other people have different usage patterns.

Do people find themselves mangling, passing around and transforming parsed-by-untyped JSON ASTs between different libraries often?

@seanparsons

As the maintainer of Argonaut I don't see a huge benefit to Scala itself in there being an inbuilt JSON library, mostly because it runs the usual risk of not changing once first created. Which means it can be a bit of a lowest common denominator implementation that gets rusty really quickly.

I agree with @lihaoyi that it's very rare to need to convert between implementations of ASTs, even to the point where I can't recall a time where I've actually had to do that.

It's also worth mentioning that the JSON standard as I think it stands now calls for arbitrary precision numbers which was a serious piece of work in Argonaut to support in an efficient way, which is why some of this gets hard really quickly.

@odersky
odersky commented Nov 4, 2015

@milessabin You are making a political argument, not a technical one. By
the same yardstick I could argue that TypeLevel should not contain specs2
because users should choose freely between it and ScalaTest or one of the
other available libraries. We have had this discussion before, and decided
we will go with a batteries included model. Recycling this debate on
individual SLIPs is obstructionist IMO.

[Some more thoughts]
This in no way endorses the present SLIP which might be promising or not; I have
not studied it. But I reject the argument that we should not even consider adding
a JSON parser to the standard library, in particular since the one we have
is so sorely lacking. Neither need the standard library stay monolithic and
unchangeable. We are talking about modules that can be updated at their own
pace, with not all sharing the same stability guarantees as the core.

On Wed, Nov 4, 2015 at 7:12 PM, Sean Parsons notifications@github.com
wrote:

As the maintainer of Argonaut I don't see a huge benefit to Scala itself
in there being an inbuilt JSON library, mostly because it runs the usual
risk of not changing once first created. Which means it can be a bit of a
lowest common denominator implementation that gets rusty really quickly.

I agree with @lihaoyi https://github.com/lihaoyi that it's very rare to
need to convert between implementations of ASTs, even to the point where I
can't recall a time where I've actually had to do that.

It's also worth mentioning that the JSON standard as I think it stands now
calls for arbitrary precision numbers which was a serious piece of work in
Argonaut to support in an efficient way, which is why some of this gets
hard really quickly.


Reply to this email directly or view it on GitHub
#28 (comment).

Martin Odersky
EPFL and Typesafe

@lihaoyi
lihaoyi commented Nov 4, 2015

Here's a few more opinions:

  • scala-offheap should have nothing to do with this. Zero. Nada. Making your core library dependent on a zero-user super-experimental library for no particular gain seems silly. JSON isn't FlatBuffers or protobufs, it's generally accepted you won't be able to just mmap something and start using it.
  • What's the exact treatment of numbers? Ints? Longs? Doubles/ BigNums? Strings? In Javascript people always parse doubles, but I've seen various JSON libraries parse different things.
  • Scala.js isn't just about js.Arrays, we have js.Dictionarys too; would we encode the JSON dicts as those? That wouldn't work with the fast/safe distinction, because Javascript objects are un-ordered and that's that.
  • In Scala.js you'd typically want to wrap "raw" Javascript objects coming from JSON.parse. This is partly for performance, partly due to the fact that other JS libraries will be giving you these things. You hardly ever will want to parse them yourself into your own AST if you can help it.
  • Nobody uses any of the listed libraries in Scala.js, and this proposal as discussed won't be of any use to anyone using Scala.js AFAICT. Cross compiling the AST would be of course trivial, but I do not see any use case as proposed (and I do think Scala.js needs a nice way of handling JSON!)
  • This could very well live outside the standard lib and get adopted by various libraries. In particular, the consumers of this AST are few enough that it's actually feasible to talk to all of them and get them to try and migrate. This is in contrast to e.g. collections where the consumers are every-scala-programmer-ever.
@milessabin

@odersky the very idea of a "standard" is a political one. This whole enterprise is political and it's appropriate to respond to it in kind.

@SethTisue
Member

If you want to promote Scala for Json processing, get behind those libraries

The code in this SLIP will literally be behind those libraries...!

@densh
densh commented Nov 4, 2015

scala-offheap should have nothing to do with this. Zero. Nada. Making your core library dependent on a zero-user super-experimental library for no particular gain seems silly.

I wholeheartedly agree. It's way too young for getting into committed relationship with standard library.

@milessabin

@SethTisue no, it will be a competitor with code which already exists in those libraries. I know you've see the relevant xkcd cartoon.

@fommil
fommil commented Nov 4, 2015

Why not just pick one that already works well? spray-json in really good. Importantly it uses arbitrary precision for numbers and reminds us all that JSON maps do not preserve ordering. If you really must create a new AST (replacing this one tiny file of awesome) then please do not forget these points.

The only change I would ask of spray-json is to remove the Root* traits, and it the marshalling layer to perhaps move all implicits to the companion of JsonFormat.

Also it has a super fast parser. And years of production usage with lots of tiny performance tweaks along the way.

Plus it has spray-json-shapeless ;-)

Let's just get everybody to standardise on spray-json

@fommil
fommil commented Nov 4, 2015

(Actually if the marshallers returned Either instead of throwing exceptions, it would significantly speed up Option marshalling, but that's a rather minor technical point).

@milessabin

@fommil There's no need to "pick one" and give it an official stamp ... if you want to use or advocate for spray-json, that's fine, but there's no grounds for your (or anyone else's) preferences being legislated for.

@fommil
fommil commented Nov 4, 2015

BTW is this just about the AST or an entire JSON framework? Parsing/marshalling/querying/validation/etc?

I don't believe the standard library even has the power to provide what is needed for a proper marshalling framework. We need shapeless for that.

@odersky
odersky commented Nov 4, 2015

the very idea of a "standard" is a political one. This whole enterprise is political and it's appropriate to respond to it in kind.

It's good to know how you feel about it. My "politics" is to make Scala more accessible for newcomers and people who do not want to spend a lot of time trying out different alternatives. I stand by that. I am very happy co-opting a best of breed library that's out there; no need to reinvent the wheel. But I am deadset against leaving stdlib with the crippled JSON parser it has, and also leaving it without one.

@lvicentesanchez

Unless I'm missing the point of this SLIP, this is just a proposal for a unified JSON AST and its goal is to allow different JSON libraries to interop between them; that is... no improvements to the existing JSON parser in the scala standard library.

If the whole purpose of this SLIP is just that, I don't thing it belongs to the standard library, not even as a module. If anything it's a library that the authors of the different JSON libraries might or might not want to use.

As an user I can't even imagine a use case where I would want to have two different JSON libraries in my classpath and rely on a unified JSON AST to interop between them.

@SethTisue
Member

is this just about the AST or an entire JSON framework?

This SLIP is about the ASTs only.

I suggest any broader discussion happen on scala-debate instead, but maybe those cows are already out of the barn.

@SethTisue
Member

I am deadset against leaving stdlib with the crippled JSON parser it has, and also leaving it without one.

Do you want to issue a separate call for a SLIP that addresses that? This one doesn't.

@tperrigo
tperrigo commented Nov 4, 2015

-1. I'm in agreement with @milessabin, and agree with @fommil that a proper JSON framework should provide parsing/marshalling/validation/etc (which, as he points out, would require something like shapeless...Circe, eg, is making excellent progress on this front). I simply don't see this as something that needs to be (or should be) in the standard library.

@odersky
odersky commented Nov 4, 2015

@SethTisue Yes, I think the current JSON parser is overdue for a replacement. It would be good to call for proposals.

@lihaoyi
lihaoyi commented Nov 4, 2015

I don't think the current proposal is useful enough to warrant inclusion, but I think a good AST + a fast parser + a serializer, with the useful knobs (e.g. serialize with proper indentation) would be useful enough to put in there.

@odersky
odersky commented Nov 4, 2015

Almost every language library has a decent JSON parser. Ours sucks, as do quite a few modules that were introduced in the very early days of Scala when every contribution was welcome, because there was so little of it and the community was tiny anyway. It's time we corrected that, don't you think?

@odersky
odersky commented Nov 4, 2015

And, sure, maybe with shapeless one can do a more fancy thing with lots of typelevel stuff. That's no reason not to strive for something that's simple and works efficiently and can be put in the standard library.

@lihaoyi
lihaoyi commented Nov 4, 2015

There are multiple stages that JSON-support can reach

  1. Just AST; bring your own parser and serializer and operations
  2. AST + operations + parser + serializer, bring your own case-class-mapper/etc.
  3. AST + operations + parser + serializer + case-class-mapper a.l.a. shapeless, uPickle, etc.
  4. Fancy operations: lenses, traversals, zippers, etc.

IMHO 1 is not useful enough to warrant inclusion, 2 is useful and straightforward to implement with current technology, 3 is probably just a bit too experimental, even as the author of a library which does exactly that, I would not want my code in the std lib just yet. 4 I have no experience with.

@duncan
duncan commented Nov 4, 2015

A standard built-in awesome JSON parser would be awesome. Yes, there's room for other parsers that do things beyond a threshold, for example all the stuff you can do with shapeless. But personally, I'm finding that I'm happier with projects that stick with the basics than with those that go too far down the rabbit hole.

@wheaties
wheaties commented Nov 4, 2015

-1

We already had an XML library. It was arguably good and bad at the same time (Daniel's Anti-XML was an improvement.) That said, look how it all worked out. It was removed. There's already a ton of approaches to make with so many libraries to "solve" this issue. I have to agree with @milessabin.

That said, I see there being nothing wrong with having a Scala official JSON library the same as the Scala XML library, separate. I could +1 that approach.

@ktoso
ktoso commented Nov 4, 2015

One datapoint (so please refrain from ranting about the JSR process. thank you.) which may be interesting to keep in mind is that the JDK is rolling in their own JSON types and parser.
This may be meaningful in the long-term plan we foresee if we'd roll our own types (or rather not).

https://json-processing-spec.java.net/nonav/releases/1.0/fcs/javadocs/index.html
https://json-processing-spec.java.net/

It might be just that this answers the inter-op story?
I'm not sure if it will make it into JDK9 though.


UPDATE: The one considered for inclusion in JDK9 was JEP198 but seems like it may slip from the release. // pointed out by @soc, thanks! #28 (comment)


UPDATE 2: The JSON JEP was dropped from JDK9, in theory it could come back for JDK10, but that's many many years in front of us. (details: #28 (comment) )

@mandubian

A standard AST in Scala could be useful to stop having multiple AST in many projects...
Specific language syntaxes, certainly not, XML was a failure...
A parser I'm not so sure, there are different approaches to parsing, different requirements... "Official" lightweight extensions to (de)serialize Json AST using existing parsers from the Java world, why not...
More about it, I don't see what: pure functional approaches to manipulate Json AST that are present in several libs will stay there or it would mean pure functional structures have appeared in standard...

@djspiewak

The current JSON situation in the standard library is really awful. It's basically worse than nothing, since it's useless to anyone who knows what it is and how it works, and it entraps newcomers into using it "because it's there".

I think in general, standard library things more often skew towards entrapment than usefulness. This is not an indictment of Scala; most standard libraries fall into the same bucket (see Java). I think this is @milessabin's general point. If we take this observation to its natural conclusion, then the correct response is to apply the scala.xml treatment to JSON: delete with prejudice and don't replace.

With that said, I can see merits in the desire for a "batteries included" language. If you're a newcomer to Scala, you don't want to just fiddle around with encoding your own List and writing merge sort. You probably want to solve an actual problem with actual tasks that you would expect an actual language to have a solution for in the standard library. Like JSON parsing.

Exactly what set of tasks falls into the category of "batteries that should be included" is a very subjective question, and there is no way to answer this definitively in a way that will satisfy everyone. Personally, I don't want JSON parsing in the standard library any more than I want XML. But that's just me.

I certainly don't agree with the nuclear "freeze/delete the standard library" viewpoint.

@odersky
odersky commented Nov 4, 2015

@wheaties XML was put into a separate module, and that's what I would propose for a new JSON library as well. The days of a monolithic standard library are over. I would have loved to have Anti-XML as an alternative, in fact was trying to convince @djspiewak to contribute it, but it never got that far.

@fommil
fommil commented Nov 4, 2015

@odersky shapeless isn't needed for a fast parser, formatter or marshalling framework as spray-json demonstrates. But it is needed to automatically derive marshallers for user domain objects, as in spray-json-shapeless. Various other libraries have chosen to do this with their own macros, but fundamentally they are all doing the same thing. Without this, users need to provide a lot of boilerplate if they want these things to be defined and verified at compile time.

@larsrh
larsrh commented Nov 4, 2015

If it's being put into a module, users would still need to declare it as a dependency in their build definition (just like for XML and parser combinators). What's the point prescribing one library when there are others out there which can be imported in exactly the same way?

@milessabin

@odersky if it's done in the same way as Scala XML, separate repo, separate binary, different release cycles from the language distribution, then it's just another library. And if it's just another library then it should compete for mindshare on its own merits and not get a leg up from an "official" rubber stamp.

You should trust the community (which is much larger and more experienced than all of EPFL and Typesafe combined, by orders of magnitude) to make its own choices about which libraries to use.

@odersky
odersky commented Nov 4, 2015

@djspiewak Fair enough. But I want to re-iterate that we are talking about modules, not a monolithic jar. Modules can evolve at their own speed. There will be modules (such as the core) that are marked as stable, and we would not tolerate any major source incompatibilities between versions for them. Others, in particular more peripheral ones, could evolve at their own speed. And there would be space for alternatives in different libraries. My aim is simply to provide a least common denominator to get something acceptably useful out of the box. What constitutes that is a matter of debate and opinion.

@OlegIlyenko

+1

this was my dream for quite a while now. Recently I was working on scala GraphQL library and had to provide adopters for different scala Json ASTs. To be honest, it becomes really frustrating to support them all (json4s, play-json, spray-json, argonaut, circe, etc.), especially considering how similar their AST representations are. Somehow reminds me of situation with futures a while ago.

In my case parsing/formatting of Json is not that important since library takes Json AST as an input and produces Json AST. Many of these Json libraries actually have very similar AST representation in form of case classes. I really wish for one unified AST which is adopted by different Json parsing/formatting libraries. I think this will make life much easier for libraries that need to work with Json, but don't do Json parsing/formatting themselves.

@nafg
nafg commented Nov 4, 2015

@larsrh good question. Maybe the answer is the "default distribution" -- i.e. the zip download?

@davidhoyt

IMO, there is other work to be done and this is one thing that seems best to leave to the FOSS community to pursue. Put the expertise of Typesafe, EPFL, and other contributors to work in areas that will provide maximum benefit.

@natewave
natewave commented Nov 4, 2015

I wholeheartedly support @odersky's vision when it comes to a "batteries included" model. Scala should encourage newcomers and not just leave them there to the mercy of usually abandoned libs. I don't see any technical arguments supporting the other position, and thus it seems to me like a narrowed vision that can't get us anywhere. JSON is a big deal, there's no reason not to strive for support for it in stdlib.

A big +1

@lrytz
Member
lrytz commented Nov 4, 2015

If you're a newcomer to Scala, you don't want to just fiddle around with encoding your own List and writing merge sort.

This is not only about newcomers, I think it's really about the 90% of users / use cases. I cannot imagine that more people ever wanted to write their own List or merge sort.

Having something in the standard library doesn't prevent any other libraries (with different focuses, for the other 10%) from existing.

@tixxit
tixxit commented Nov 4, 2015

A few concerns about a standard AST... I went to some lengths to ensure that arbitrary numbers were supported in Argonaut (and Circe by extension). This is not just using BigDecimal (eg spray-json), but also ensuring we only pay for the number type we use (eg don't force parsing to BigDecimal, if we're using Double). On top of this, it supports numbers that can't be stored in a BigDecimal, due to the limited exponent.

It's possible to create an AST that is open enough to allow these kind of optimizations, but I don't really hold out hopes that this will happen.

Edit: Just realized this is to make json4s-ast the standard. I'm actually pretty happy with that AST! That said, why not just promote json4s-ast? What other modules would benefit?

@odersky
odersky commented Nov 4, 2015

@milessabin You are thinking mindshare and competition, I am thinking usefulness "out of the box". Different concerns, obviously.

Also, your characterization of the SLIP committee as Typesafe + EPFL is somewhat disingenuous, given that a major Typelevel committer is on it.

@milessabin

If you want a "batteries included" model make your choice(s) from the existing libraries, recommend them and put some effort and resources behind them.

@fommil
fommil commented Nov 4, 2015

I don't see any value in adding new modules to Scala. They are a good way to git rid of mistakes. Incorporating extremely stable codebases into the standard library has the advantage of getting in the door at large corporations with the language, thereby helping avoid the temptation to write it yourself. And if I've learnt anything from big corporates, it is that they have lots of very bad developers who just love to write things like JSON frameworks and logging APIs, but it's perhaps too late frankly.

@lvicentesanchez

A JSON module that evolves independently is just a library, I don't see the point of creating yet another library instead of choosing one of the existing ones as the preferred/endorsed. It seems a waste of "resources".

@larsrh
larsrh commented Nov 4, 2015

I don't see any technical arguments supporting the other position, and thus it seems to me like a narrowed vision that can't get us anywhere.

@niseh, please re-read this thread. You don't appear to be interpreting the arguments charitably.

@kings13y
kings13y commented Nov 4, 2015

+1 I'd like a small modular core, but either RI or preconfigured bundles for when I don't want the hastle of picking the best of breed, or to point newcomers at to be up and running quickly. Don't want another maven like grab bag of snowflakes, but I do think diverse and dedicated community could build better dedicated libs than could be reasonable expected of a core team.

@non
non commented Nov 4, 2015

I think an interesting question is whether library/framework authors who desire a unified JSON AST are already planning to move to json4s-ast from their own libraries, or only plan to do so if it is included in the standard library.

I helped to design these ASTs, I think they are reasonable, and I plan to add support for them in Jawn (along with all the other ASTs that are supported). However, I would feel more comfortable with this proposal if the library had more usage in the wild and some non-snapshot releases.

There's one point in the SLIP text that is worth pointing out:

The issue presents itself more in frameworks/libraries that have to deal with JSON.
As a quick example, slick-pg, an extension ontop
of slick for Postgres extensions, needs to provide implements of the JSON type for each of
the commonly used JSON libraries, instead of just needing to support one.

My interpretation here is that if this SLIP were accepted, then libraries like slick-pg would only support json4-ast as opposed to other libraries that it currently does support. So it does seem like libraries like argonaut, rapture, rojoma, circe, jawn, etc. would be at a significant disadvantage when competing with json4s-ast if this proposal were accepted, whereas now everyone competes on a (relatively) level playing field.

This isn't necessarily entirely bad: libraries need to justify their existence versus some status quo. But it does mean that we should be very careful about standardizing on something that is not yet proven, since it will be difficult/impossible to change our minds later.

@odersky
odersky commented Nov 4, 2015

@larsrh Why do you say "prescribing"? Nothing prevents alternatives to flourish. There are at least two reasons I can think of to put it in the standard distrubution: (1) It means that other modules in the standard distribution can depend on it, without fear of being rendered inoperable by a change. (2) it gives one suggestion to newcomers and generally non-experts what libraries might work well together. If you do not believe in these reasons, why do you invest time in Typelevel?

@fommil
fommil commented Nov 4, 2015

@tixxit if somebody wants support for numbers that can't be represented as BigDecimal then perhaps JSON isn't the right protocol in the first place!! My main concern in financial data.

@gpampara
gpampara commented Nov 4, 2015

I'm very much against this being in the stdlib. I simply don't understand the need.

If newcomers come to scala and ask about anything, what is so wrong with having a wiki/web page stating a list of recommended libraries? There could be a whole list of JSON related libraries that users could look at and choose themselves, thus allowing the community to benefit from the competition. Having something in the stdlib for JSON (or anything else) is prescribing what a user should be using.

Instead of pigeon-holing them into something that is "standard" and possibly underwhelming, support the community projects that already exist.

@fommil
fommil commented Nov 4, 2015

@odersky 3) corporate uptake. We don't all have access to maven central! But this is not true for modules.

@DarkDimius
Member

I'd like to have a actively maintained replacement for scala.util.parsing.json.
It does not need to part of main artifact of standard library: it could be a separate maven artifact with own releases. But, for convenience, I would want it to be part of standard distribution, as JSON became very common as an interchange format and I'd like to be able to have a library that just works everywhere including standard REPL, both for me and for newcomers.

But as far as I understood the SIP, it's not about user-facing JSON library, but about common interchange format for multiple already existing JSON libraries. In that case, this is entirely a different situation, as the format has no use in standard distribution if there is no implementation that uses it.

@larsrh
larsrh commented Nov 4, 2015

@odersky As has already been pointed out, third-party libraries have a hard time competing against the standard library. Compare the amount of supported alternative JSON libraries with the same for XML. Also, there are large parts of the ecosystem which do not depend or do not interact with typelevel libraries, and this is just fine, so I don't see the point.

@non
non commented Nov 4, 2015

We have an example where Scala went in the other direction: actors. Akka actors proved to be the most successful actor library and are now a de facto "standard" despite being a separate library. They were so successful that the competing standard library implementation was removed. Similarly, my understanding is that Akka's design for Futures and Promises was the basis for SIP-14, which gave us the modern implementation of futures.

Akka was is (and was) the status quo actor library for Scala. If there is no status quo JSON library for Scala then it's interesting to wonder why that is. Is the problem too easy? Too large a design space? Too many competing requirements? Something else?

@larsrh
larsrh commented Nov 4, 2015

@non I'm confused – are Akka actors a module, just like parser combinators?

@nafg
nafg commented Nov 4, 2015

I think we should also make it easier for people to add libraries to their project. Maybe something like http://ls.implicit.ly/#finding? Maybe @softprops can comment

@non
non commented Nov 4, 2015

@larsrh I just meant that we used to have built-in actors, and now they are just provided by a separate de-factor standard (Akka) even though it is a separate library.

@mandubian

@non I think Json multi libraries prove there are different approaches about it... the smallest common denominator is the AST and having contributed to a Json library and a validation one, I must say I wouldn't have been against having a common AST because Json is a very stupid format and I'm not interested in it...

@odersky
odersky commented Nov 4, 2015

@non Completely agree that if there is a standard library in a domain out there we should not duplicate it in stdlib. In fact the migration from scala.actors to Akka was done with the active support of the standard library maintainers. Phillip Haller, who originally did scala-actors, spent a lot of effort to make sure the migration to Akka was as smooth as possible. But Akka is huge compared to the modules we are talking about. If you think you need an actor library, you'll probably find out quickly that Akka is the one to turn to. But for JSON parsers no such reference implementation exists, so confusion ensues.

@channingwalton

I am interested in the 'batteries included' notion. Is it to help newcomers, and also provides comfort that the batteries have a reasonable longevity and support.

They certainly are not going to solve any problems as far as projects finding multiple libs for json or anything else, it'll just add one more to the mix.

Martin, what is the problem being solved by 'batteries included" as you see it?

@non
non commented Nov 4, 2015

I do think that @nafg is on to something. Most languages I've used (besides Java of course) have a reasonable directory of libraries (CPAN, PyPI, rubygems, npm, hackage, opam, etc.) so users don't tend to have problems finding these things, and it's relatively easy to see which are the most active, complete, well-supported, etc.

@non
non commented Nov 4, 2015

I guess what I am trying to say is that if no one is agreeing on a standard JSON library in the wild, I think it's worth asking why that is before immediately proposing a new library as a standard. Otherwise we run the risk of duplicating the XKCD comic about competing standards [1].

Are there libraries out there who expose a JSON library dependency and are intending to move to json4s-ast on its own merits?

[1] https://xkcd.com/927/

@fommil
fommil commented Nov 4, 2015

@non its because it's an interesting intellectual exercise to create another one. Just like logging. And that hurts everybody. It is exactly the sort of thing that I'd like to see standardised at the AST level. But putting it in a module is useless because that's just another maven central artefact.

@jeantil
jeantil commented Nov 4, 2015

I agree that the current Json implementation in the std lib has to go.

Not having a standardized AST makes it harder to switch serializer and parser implementation (to and from string), It also makes it harder to switch typeclass implementation (to and from class/caseclass) as each AST as its own idioms to manually create a tree. and of course they are close but not identical.

Not having a default parser/serializer in the std lib is not necessarily a big issue. I believe newcomers will simply google json with scala anyway and pick the first library returned.

What the std lib and the community should strive for is to make it easy to switch to a different implementation once they realize their initial choice is not the one best suited for their problem.

@hepin1989

Why there are so many json lib in scala,a fork and a fork of fork?just for fun?or just for practice or just make a voice?how many star and how many contributor and how activity them are?how wildly they are used?are they depended by spark or something like those?I think we should be nice to the new comers and normal usage,I really agree with @non about his cats,why they try do add more document and test,that's the way to think about users,even them have not nake money from it.

90% use case are simple,why coulden't you try to share something and make it more nice to new comers,I really want scala to be more useable and nice for a greater and bigger community,for good thing for the whole world,not only some of the expects.

@odersky
odersky commented Nov 4, 2015

@non Completely happy to co-opt an existing library. But having some automatism is important. Let's face it: Our average user does not care at all what kind of JSON library is provided and what the tradeoffs between possible alternative libraries are. They just want to have some way to parse JSON if they need it. So any library that is not completely broken will fill this need. The importance is simply to have something always available.

If I would try out another language, did not find a standard JSON parser, and was pointed to an extensive body of discussions on Stackoverflow (or, worse, Twitter!) discussing the merits of the various contenders, I would be gone at this very instant.

@henridf
henridf commented Nov 4, 2015

If I would try out another language, did not find a standard JSON parser, and was pointed to an extensive body of discussions on Stackoverflow (or, worse, Twitter!) discussing the merits of the various contenders, I would be gone at this very instant.

Exactly the feelings of this dabbling Scala user (who won't be gone in an instant due to an ongoing project, but still remains baffled that this discussion should even happen...)

@adriaanm
Member
adriaanm commented Nov 4, 2015

I agree with @odersky and @non, with the clarification that the standard library's goal is to standardize, not to exclude alternatives, and that the promoted/"standard" JSON API is a module with a default implementation, that should be easily swapped out for another module.

The abstractions should be standard, the choice of implementation is up to you. Especially in the JSON space, having a standard AST would be a huge boon to all the other libraries, as they could compete on performance etc, rather than on "what you happened to standardize on and now you can't move because migration is too expensive".

@odersky
odersky commented Nov 4, 2015

@larsh

As has already been pointed out, third-party libraries have a hard time competing against the standard library. Compare the amount of supported alternative JSON libraries with the same for XML.

I think that has to do with the relative complexity of XML and JSON. Stdlib had both. XML was problematic in some aspects but hard to do better in all dimensions. JSON was super-easy to improve upon. But as I wrote before, it's not a matter of "competition" but of maximal usefulness for the community as a whole. If you disagree with that opinion, I'd like to point to the "Lisp Curse" as a supporting argument:

http://www.winestockwebdesign.com/Essays/Lisp_Curse.html

@sjrd
Member
sjrd commented Nov 4, 2015

because Javascript objects are un-ordered and that's that.

Not anymore. Current engines all agree on keeping the order of insertion, and this de facto behavior has been spec'ed in ES 6, so we can actually rely on it.

@adriaanm
Member
adriaanm commented Nov 4, 2015

/cc @jroper, who did some work on a standard json api for play recently, I believe

@hepin1989

just take a look at golang,in the src/ encoding ,they have json,csv ...

@mdedetrich

@lihaoyi

IMHO that makes it considerably less useful. I don't find myself passing around parsed but un-converted JSON ASTs around enough for it to be a hassle; most of my work involves going straight from String -> case class or case class -> string, but maybe other people have different usage patterns.

We currently have to interopt between 3 different JSON AST's in our codebase, mainly because we have a client side REST API that was written in Spray, and another that was written Scalatra (json4s). There is an older one that also uses Finatra (which we want to remove)

Any new library that uses JSON in some way, and wants to be used by the community, has to support like 5 JSON AST's, all of which differ in trivial ways

What's the exact treatment of numbers? Ints? Longs? Doubles/ BigNums? Strings? In Javascript people always parse doubles, but I've seen various JSON libraries parse different things.

According to the JSON spec (referenced in the SLIP), JSON numbers have unlimited precision, in other words they are unbounded real numbers (hence why both this SLIP, and Jawn, uses String as an underlying representation of JNumber, at least for the fast versions)

Scala.js isn't just about js.Arrays, we have js.Dictionarys too; would we encode the JSON dicts as those? That wouldn't work with the fast/safe distinction, because Javascript objects are un-ordered and that's that.

Are you talking about safe or fast? Fast has ordering, but thats only because it uses an Array (Array or js.Array). Array is the fastest, most memory efficient data structure for speed, particularly when you are building a JArray/JObject. The safe version has a proper implementation of what JArray/JObject are meant to represent

In Scala.js you'd typically want to wrap "raw" Javascript objects coming from JSON.parse. This is partly for performance, partly due to the fact that other JS libraries will be giving you these things. You hardly ever will want to parse them yourself into your own AST if you can help it.

Its more for libraries that use a JValue, and want to target both JVM and Javascript.

Nobody uses any of the listed libraries in Scala.js, and this proposal as discussed won't be of any use to anyone using Scala.js AFAICT. Cross compiling the AST would be of course trivial, but I do not see any use case as proposed (and I do think Scala.js needs a nice way of handling JSON!)

Originally one of the reasons was triviality, but the other reason is that I would (when Scala.js ecosystem grows enough) use some of the other libraries (that would use this AST) to do my client side JSON querying, mainly because its typesafe

This could very well live outside the standard lib and get adopted by various libraries. In particular, the consumers of this AST are few enough that it's actually feasible to talk to all of them and get them to try and migrate. This is in contrast to e.g. collections where the consumers are every-scala-programmer-ever.

If this is an official module rather than in stdlib, I don't really have many issues with that

@fommil

Why not just pick one that already works well? spray-json in really good. Importantly it uses arbitrary precision for numbers and reminds us all that JSON maps do not preserve ordering. If you really must create a new AST (replacing this one tiny file of awesome) then please do not forget these points.

The fast version here is almost the same as spray-json. The safe version is meant to represent a valid JSON structure that has good all round performance for querying values (i.e. eC lookup time in JObject)

@larsrh

If it's being put into a module, users would still need to declare it as a dependency in their build definition (just like for XML and parser combinators). What's the point prescribing one library when there are others out there which can be imported in exactly the same way?

It has an "official" label on it, but also the AST is designed to basically never change, so it has the feature of stability

@OlegIlyenko Precisely what this is trying to solve

@non

My interpretation here is that if this SLIP were accepted, then libraries like slick-pg would only support json4-ast as opposed to other libraries that it currently does support. So it does seem like libraries like argonaut, rapture, rojoma, circe, jawn, etc. would be at a significant disadvantage when competing with json4s-ast if this proposal were accepted, whereas now everyone competes on a (relatively) level playing field.

This is one of the main justifications, however thats why json4s-ast (current reference library) was made with the huge amount of feedback from all (or at least most of) those library creators. This is just meant to be an AST, and the AST for JSON is downright trivial. If you look at the implementations for those AST's, they basically are almost practically the same (if you split them into fast or safe versions).

Its like having 3 different versions of String

@adriaanm

The abstractions should be standard, the choice of implementation is up to you. Especially in the JSON space, having a standard AST would be a huge boon to all the other libraries, as they could compete on performance etc, rather than on "what you happened to standardize on and now you can't move because migration is too expensive".

Precisely

@sjrd

Not anymore. Current engines all agree on keeping the order of insertion, and this de facto behavior has been spec'ed in ES 6, so we can actually rely on it.

Yes, but the JSON standard as is about unordered objects

@odersky
Regarding using a parser as part of the module. I am not in disagreement of that, however it was outside of the scope for this SLIP, and there is a huge (and amount of good competition) for parsers. If scala wants to pick one of the current existing parsers so users have a solution, I am not against that.

@dwijnand
Member
dwijnand commented Nov 4, 2015

Trying to be "batteries included" but then actually be a module means:

  • you still need to actually declare it as a dependency, like an other library
  • but you would get it in the distribution REPL (scala on the command-line)

👎 I don't think this should be a part of the Scala distribution, I think it should be part of the larger ecosystem.

I hope json-ast picks up interest again and reaches success as I think its goals are important.

@cb372
cb372 commented Nov 4, 2015

I don't understand the benefit of adding a json library to the stdlib.

  • It's more bloat for the core Scala developers to maintain, even if it's in a separate jar
  • Anything tied to the Scala release cycle will inevitably evolve/fix bugs/etc more slowly than an independent library
  • What's the point of a batteries included solution that quickly becomes outdated and superseded by one or more libraries?

As for the argument for adding just an AST to the stdlib, I don't really think this is necessary either. I don't think I've ever had to interop between two different json libs in the same Scala project, so who cares whether different libs share the same AST?

Also I don't feel like the current Scala ecosystem is as bad as people make out. Sure, there are a few different libs available, but they serve different purposes: json4s is for people who just want to get shit done, play-json is more type-safe, argonaut is for people who love FP, etc.

@mdedetrich

It's more bloat for the core Scala developers to maintain, even if it's in a separate jar

If you have a look at the implementation, and the spec for JSON, its been virtually unchanged for years. In fact, I don't see the current source changing (apart from version bumps) for years

What's the point of a batteries included solution that quickly becomes outdated and superseded by one or more libraries?

There is a lowest common denominator for interpretability. Thats like arguing "why do we have a common String". Have a look at how much pain multiple "string" types has caused in the Haskell community (text vs List[Char] vs others)

As for the argument for adding just an AST to the stdlib, I don't really think this is necessary either. I don't think I've ever had to interop between two different json libs in the same Scala project, so who cares whether different libs share the same AST?

I do this all the time. Its not even so much about interoperability, but the fact it makes it that much harder for library creators (that have their library using JSON in some way) need to target 5 different AST's that differ slightly.

Let me point out to everyone, that what is now 5 AST's, used be 2/3. That number keeps on increasing, and we don't need to get to a ridiculous situation in 5 years time where "if you want to do something with JSON, you need to target/maintain 8 JSON AST implementations"

Also I don't feel like the current Scala ecosystem is as bad as people make out. Sure, there are a few different libs available, but they serve different purposes: json4s is for people who just want to get shit done, play-json is more type-safe, argonaut is for people who love FP, etc.

Not sure about argonaut, but there is commonality in all of these AST's. Where all of these libraries differ greatly is their parsing/querying, which is not something I want to standardise (but there is good argument for a standard parser)

@sschaef
sschaef commented Nov 4, 2015

I don't like that we have this discussion at all.

The problem of being beginner friendly can't be solved with a stdlib because it is a tooling problem. People ask questions like "which library should I use?", "where can I find documentation to this problem?", "which IDE should I use?", "which build tools should I use?", "how is the community organized?", "how do I make my library visible?", "how can I find out if people like my library?", "how many people use my library?" and so on. Solving any of these questions would contribute to making a language more beginner friendly. Adding more content to a stdlib solves the first question for a short time (probably months, maybe even years) but ignores all of the other questions completely. Not an efficient way to solve a problem.

With better tooling support all of the questions above could easily be addressed but Scala - as all other languages before it - has chosen to ignore that option completely. Taking the short route of only solving the immediate problem rarely is the path that should have been taken in the first place. Most people believe that creating a unified interface, an ecosystem for libraries, documentation, compilers, IDEs and other tools is too difficult to solve. But it is not - with the possibilities we have today we no longer have to repeat the mistakes of most other programming languages for forever by always going the short route.

As long as we have discussions about such "trivia" like adding a JSON AST to a stdlib, we miss the time we need to discuss how to solve the ecosystem problem - which is (for me) by far the largest unsolved problem Scala has (and not just Scala but most other programming languages too). Scala spread because it solved problems at the roots instead of fighting the whole day against symptoms. I would appreciate it if we could bring back this spirit.

@fommil
fommil commented Nov 4, 2015

@sjrd you're thinking purely in terms of browsers. In my world both ends of the pipe are servers and can be ancient. We cannot assume or rely on ordering, of course being ordered by happy accident is a different thing.

@bwmcadams

-1

... with right to change my opinion presented different arguments/evidence reserved.
I was hesitant to wade in on this, and my voice added to this may not be as useful as I think... but here goes. And this is going to be a long one, sorry.

From a technical standpoint I have a few things here:

  1. The merits of the argument entirely aside, I don't feel that the proposed SLIP makes any valid, reasoned arguments for a standard AST or JSON Library in Scala. It points out that there are many libraries, and someone once tried to make a standard one (which has been recently resurrected, but see my further point on that below). It presents drawbacks, which is of course something I also see as a requirement. But I see it failing entirely to make any reasoned case for its existence or implementation. This is a flawed proposal on that omission alone, and a strong reasoning for my -1. It seems that we're voting on whether to allow this SLIP be proposed, rather than “are we, as a community, voting to approve it?”. If I'm wrong there, I'd appreciate correction. But my impression is if we feel the flaws like what I've just raised are fixed we are voting whether to merge a proposalnot approve “do we do this“–and argue / vote later if the SLIP should be actually approved.
  2. Having thought it through thoroughly, I would also support voting against the approval of the proposed idea of a centralized JSON Library and/or AST. I don't see merit in it, or any advantage being provided to Scala or its users. It seems to come from a place of “there are too many JSON libraries and it is hard to choose”. But this proposal would do nothing to fix that and it is not a problem that needs to, or can be, fixed. Do we really feel Scala can force consensus or compliance of JSON library usage/implementation standardization within the community with meaningful impact or justification?
    • Again, I question what problem is really solved by introducing this? Sure, we give new Scala users a “default” JSON library to choose from. But it doesn't obviate the advantages of other libraries which may be more appropriate to their use case. In contrast, it guides them unnecessarily to use something that is deliberately intended to be more “broadly generalized” in its approach. Do we feel that's a good or productive thing? I am not using this point to stand on a hard “no” – I'm using it to point out that this question needs to be part of our discourse.
    • I find myself – having used many of the different mentioned “competing” JSON libraries in various contexts – swayed by the argument that there's merit in having multiple libraries, each of which can focus on a use case optimized AST instead of a “generalized” one that seems to provide no advantage, and disadvantages which are quite literally outlined in the SLIP. On a counterpoint, it is fairly noted in the SLIP that there has been some recent collaboration to have many of these libraries use a standalone community driven standard AST. Which I'll address thusly...
      • A question we also need to ask: Will the libraries mentioned as part of the fragmentation actually adopt or value an AST that is baked into Scala? If we (well, the JSON lib authors) end up agreeing that “hey, a common AST isn't a bad idea”, that doesn't translate automatically/implicitly/transitively to mean that one should be baked into Scala. Scala moves more slowly than an open source library can. If these libraries standardized on a central AST that means “improvements” are locked to waiting on the next release of Scala – as presumably you're not likely to see a major AST change in a minor release, that could be a long wait.
    • I am hesitant to say it, but “bloat” concerns me. This circles back to the what value do we derive from this?, and are we being consistent in decisions to add/remove things from StdLib? Are we thinking about the justification for adding more to the standard library? XML support is being moved out of the StdLib, IIRC – you'll need (or already need?) a separate dependency for it. Actors did the same thing, though much of that was driven by Akka's existence (again an IIRC). But my point, I hope, stands or at the very least argues for itself from here. I'm not fully in either of the camps arguing for a smaller standard library or a more comprehensive one. My opinion for "smaller standard library" with regards to bloat applies to this case, specifically, based on the other questions I've already posed.

That ends my technical arguments. I'd be remiss without a final statement, however...

A Final Thought on the Poor Behavior Exhibited Throughout This Thread

Or, How I Learned To Stop Arguing And Learned To Prefer Discourse

I'm disappointed in all of us (I include myself as I'm not without blame for doing same in past) on how quickly this thread devolved the way it did. Worsened by the nauseousness of the fact that it devolved at all.

Are we are a community, or a collection of competing tribes? I like to think we're a community - but the kind that doesn't resort to Scarlet Letters, Tarring & Feathering, Stockades, or Good Ol' 'Murican Smear Campaigns. Personally, I'm somewhat mortified that there was assumptions made that a mere -1, or a disagreement without qualification or even fairly assumable faction-driven opinion lumps someone into a targetable group such as "scalaz people".

I, for example, disagree with this proposal. It happens to be that yes, I'm a user of scalaz. I've given conference talks on it. I've helped write a scalaz test library which led me to be involved in Typelevel. I even got involved in cats, contributing some code to it already (but IIRC someone else helped me get it right after my first draft was somewhat wonky, so I feel weird taking credit). But I've also worked for Typesafe. I've been a longtime contributor to–and evangelist for–Akka. I am currently employed by an organization whose primary focus is on services related to Typesafe's product offerings. I'm a strong supporter of all of the above projects and organizations... because there are different tools for different jobs. I admit that there are strong voices here who are capable of swaying my opinion at times; I also try to stop and think about the merit of them before deciding for myself. But hey, now you all have a list of possible tribes to lump me into based on how I vote in the future?

I have spent a significant amount of my life for the past 6+ years advocating and evangelizing for Scala. When large organizations or representatives thereof ask me to justify moving from say, Java to Scala, beyond just the technical advantages... “Strong, Healthy Community” is something I hold up as a badge of pride.

I would feel uncomfortable ever making that argument again, if this is genuinely how we all think we should behave when we disagree. We are introducing unnecessary partisanship, when we should treat each other as equals who have varying opinions that may, at times, clash. How do you think this kind of behavior looks from the outside to someone considering if making the investment (be it time or money or anything else that qualifies)?

Debate & discourse is healthy: but only if it's a proper debate. Without digressing unnecessarily into a lesson in American history, we should look to something like The Lincoln-Douglas Debates in our discourse and conduct. Roughly, what I remember being taught many years ago in school WRT to good discourse...

  1. An initial statement should never be deliberately antagonistic, but reasoned with logic & facts. Passion is good, but never ever allow your passion to pass into fervor, lest you be lost forever.
  2. The response you give should follow the same rules as above. With the even more important need to avoid your counterpoint being based in any way on an attack on your opponent. They aren't your adversary: they are someone with a valid opinion that happens to clash with yours.

There is a staggering amount of unbelievable minds in this thread, many of whom are the best educated and knowledgeable people I have ever met in my life. Please act like it.

@mdedetrich

@bwmcadams

Having thought it through thoroughly, I would also support voting against the approval of the proposed idea of a centralized JSON Library and/or AST. I don't see merit in it, or any advantage being provided to Scala or its users. It seems to come from a place of “there are too many JSON libraries and it is hard to choose”. But this proposal would do nothing to fix that and it is not a problem that needs to, or can be, fixed. Do we really feel Scala can force consensus or compliance of JSON library usage/implementation standardization within the community with meaningful impact or justification?

By extension, you can say there is no merit in having a standard String?. Obviously the inclusion of this into stdlib (or a supported module), doesn't automatically solve the problem, it does so indirectly. There is already a lot of support for this library (scalatra/lift/play/spray for now), and if something is including and is official, then it means that over time libraries will mainly target that implementation

I'm disappointed in all of us (I include myself as I'm not without blame for doing same in past) on how quickly this thread devolved the way it did. Worsened by the nauseousness of the fact that it devolved at all.

Agreed. I expected to have a nice discourse about the merits of this SLIP, but instead its turned into some political mudball where people provided non helpful feedback or have standoff'ish comments

@OlivierBlanvillain

@non

Are there libraries out there who expose a JSON library dependency and are intending to move to json4s-ast?

https://github.com/jto/validation

But the story is not as simple as "move to json4s-ast". Play users still want something that works out the box with the AST returned by the Play parser, Scala.js users want something compatible with the js.Any/js.Dictionary returned by the native js parser and so on.

In this context (wanting to be out of the box Play/Spray/Scala.js/... compatible) I think having a standardised json API could be very useful to circumvent the current "let's be compatible with 7 ASTs" situation that is found in many json related libraries.

@non
non commented Nov 4, 2015

@OlivierBlanvillain Thanks, that's really good to know. I agree that supporting N different ASTs is a bit annoying (I wrote a bunch of code to solve that in Jawn using type classes).

What do you think your migration path will be? Presumably you'll add support for json4s-ast now and then deprecate the others later?

@puffnfresh

👎

My professional life is hard enough as it is because of the current stdlib having many poor performing and broken things. I want less stdlib, not more.

@sjrd
Member
sjrd commented Nov 4, 2015

@fommil

you're thinking purely in terms of browsers. In my world both ends of the pipe are servers and can be ancient. We cannot assume or rely on ordering, of course being ordered by happy accident is a different thing.

I think you misunderstood my comment. I am not saying we should standardize on JSON ASTs that guarantee to preserve ordering. I was only replying to @lihaoyi's concern that, on the JS platform, we should use js.Dictionary to represent objects, but that using those would not be compatible with the fast "requirement" (of the proposal as it currently stands) to preserve order. I was just saying that yes, js.Dictionary does preserve order, and that therefore, should we choose to standardize on something that needs to preserve order, we can use js.Dictionary anyway.

@hepin1989

@sschaef yes,but slip,dotty,and so on is just a part of the ecosysterm too.and in fact scala ide and idea's plugin both have been improved a lot in the last year.more scala user,more bigger community,then more commercial investment from company.

@fommil
fommil commented Nov 4, 2015

@sjrd ah sorry, I missed the reference to the JS impl. Yes that sounds sensible.

@mandubian

@bwmcadams I feel like this discussion isn't so insane except the few usual ones... globally it tells something interesting about the topic...

@shawjef3
shawjef3 commented Nov 4, 2015

A standard JSON AST that is appropriate for most uses and is official in the same sense as the XML library would help with Scala adoption. If someone wants to do something that uses JSON, they shouldn't have to spend a day reading about and weighing different implementations; that's time wasted that could be used getting something useful done. An advanced user can choose to use a non-official library for whatever reasons.

@mdedetrich

@puffnfresh

👎
My professional life is hard enough as it is because of the current stdlib having many poor performing and broken things. I want less stdlib, not more.

If you have issues with the performance/brokenness of this SLIP, then please show them

@mdedetrich

@non

What do you think your migration path will be? Presumably you'll add support for json4s-ast now and then deprecate the others later?

Thats what I had roughly in mind. More importantly we need the broad community to support json4s-ast (hence why when creating it, I was trying to get everyone onboard in regards to whether or not they would use this AST, and what they need from it)

@adriaanm
Member
adriaanm commented Nov 4, 2015

Please be kind and remain on topic XOR refrain from commenting.

If too many cannot abide this basic form of politeness, I will have no choice but to lock this thread, with my apologies to those contributing constructively.

EDIT: We have deleted comments with zero technical contribution and 100% unkindness, and will continue to do so.

@Ichoran
Ichoran commented Nov 6, 2015

@jedws - Oh, sorry, I misread. Indeed, it does.

However, the bandwagon point was what I was really intending to get at. That there are so many libraries indicates that it's something that lots of people want to do. The point of a standard library is to do those things that people want to do a lot (or want to do rarely but can't do any other way, but Scala's really good at making that set small).

So I'm not sure why you think something being a bandwagon means that keeping people from having to reinvent the same bandwagon wheel is a bad thing?

@mdedetrich

That is not a reason, that is a bandwagon. There are plenty of features many of those languages share that we should eschew. Scala should choose its own path for reasons that are clearer and more useful than "X has it, we should too".

Sure, but no one is providing substantial reasons why, and its not in the scope of this SLIP. If this is going to be a community decision, than this thread has already provided preliminary evidence that there is a lot of support for JSON inclusion.

Its also not an all or non exercise, if you don't like the implementation, you are not forced to use it.

But… it does, it just sucks. And as you point out, this SLIP has nothing to do with a standard way of handling JSON, and you aren't at all interested in creating one – or even two – just an AST that is difficult to use.

This is just being pedantic. It does provide a standard way of handling the JSON type, it doesn't provide a standard way of parsing JSON (if you are going to parse JSON, you need to parse it to some AST form anyways).

An AST has to come first, in some form, else your "parsing JSON" is just validating a String.

@jedws
jedws commented Nov 6, 2015

To summarise, there are several questions raised by this SLIP:

  • Should there be a standard Scala module (endorsed lib) that exposes a standard JSON AST API
  • Should the chosen AST API be a brand new, previously unused library that exposes not one, but two ASTs, or should it be at least based on existing production JSON APIs. (the XKCD problem)
  • Should the AST API be designed to support only a subset of legitimate JSON, and what should it do if it encounters legitimate but uncommon JSON.
  • Is a SLIP process the right way to develop a new API, or should it be taken outside the SLIP, implemented in a few interested projects and once an API has settled then submitted as a SLIP.

Out of scope but people really want to argue about:

  • Scala should replace the existing stdlib JSON parser that doesn't suck

There is yet to be any consensus on any of these afaict. Personally, I would strongly counsel removing this SLIP and getting a few libs to start actually using your proposed API before resubmitting it – that way you don't have the added pressure of submitting what is essentially a hypothetical API and trying to develop it into a concrete one as well as all the other open questions.

@jedws
jedws commented Nov 6, 2015

This is just being pedantic. It does provide a standard way of handling the JSON type, it doesn't provide a standard way of parsing JSON (if you are going to parse JSON, you need to parse it some AST form anyways).

No, it isn't pedantic, and when designing APIs pedantry is a virtue anyway.

It provides a very low-level way to model a data-structure that could be JSON, but not an API that you actually expect anybody to use. As @jroper points out, the design of it deliberately eschews any useful combinators for navigating, extracting or updating a structure, and expects that other non-standard libraries will provide that. It isn't an API you expect anyone to use directly.

And, without a parser or encoder, it has no relation to actual JSON at all.

@marekzebrowski

I think JSON is not a core language concept. It's just a representation of data - nothing more. It happens to be convenient to use when interacting with web browsers, but that's all. It has no meaning in Scala - because it has to wrap language constructs like values, classes and so on. It is necessary to somehow be able to get data from outside world into Scala from JSON format, and export them. There are very few situations when actually manipulation on JSON objects is valuable, and for those few specialized library, independent of language relate cycle probably is better way to go.
I think there are two bad things that could potentially happen when JSON AST will be incorporated in the language itself:

  1. it may come with crappy implementation, so first time experience will be bad, and people would get impressions that scala is bad at JSON
  2. it might slow/impede development of other libraries - as it happened with XML in Scala
  3. as other libraries are already well entrenched, it will not be a "standard", but just "another way" of processing JSON in Scala, just adding to the complexity, not reducing it.
  4. It might promote bad programming practices - like creating JSON objects and manipulating them inside scala applications, instead of operating on classes and objects and serializing them into JSON for external communication.
    Overall my view is that JSON inside Scala will do more harm than good.
@dwijnand
Member
dwijnand commented Nov 6, 2015

@mdedetrich

If this is going to be a community decision, than this thread has already provided preliminary evidence that there is a lot of support for JSON inclusion.

It has also provided lots of evidence that there's a lot of support for non-inclusion.

@hohonuuli

@Ichoran

I would LOVE a native HDF5 implementation

Just a note that the NetCDF-Java libraries can read HDF5 (well, most HDF5 files). Writing them ... not so much.

@martijnhoekstra

@marekzebrowski nobody is proposing to create language support for JSON in the Scala language proper.

@SethTisue
Member

Is a SLIP process the right way to develop a new API, or should it be taken outside the SLIP, implemented in a few interested projects and once an API has settled then submitted as a SLIP

This one I can settle. The SLIP committee very strongly encourages proposals just like this one — proposals that are promising, but also certainly aren't ready yet for an immediate accept-or-reject decision.

Submitting a proposal calls everyone's attention to the proposal and lets the community give a first round of feedback on the initial draft and offer to help. The SLIP committee will also decide whether the proposal will become "active", added to the SLIP list, and assigned a number. That lets the submitter know whether the proposal is even worth putting further effort into.

This is happening exactly the way it's supposed to happen.

Please, more SLIP proposals! You don't need a bulletproof proposal ahead of time and you don't need consensus ahead of time.

@Ichoran
Ichoran commented Nov 6, 2015

@hohonuuli - Writing is pretty important for data interoperability! And the Java version won't write NetCDF4 except by wrapping the C version.

@dwijnand - It has provided indication that there is a sentiment against inclusion, to some extent, but few reasons against, and very few that are detailed and carefully reasoned. There's a lot of worry about it being bad but much less input on why it could only possibly be bad or on how to fix it. So it seems pragmatic to just ignore that as noise and get on with the business of fixing it (since it doesn't seem to some of us that it must be bad, though it could of course always end up bad for technical or political reasons).

@milessabin

@Ichoran the burden of proof should be the other way around: the default position should be against additions to the "standard" library unless extremely compelling reasons can be found for the opposite. Thus far no good reasons have been given why this library couldn't stand on it's own two feet perfectly well outside.

@lihaoyi
lihaoyi commented Nov 6, 2015

the default position should be against additions to the "standard" library unless extremely compelling reasons can be found for the opposite

That's your default position, not everybody's =)

@Ichoran
Ichoran commented Nov 6, 2015

@milessabin - I agree, actually. I just think the burden of proof has been met here: 8+ nearly identical yet incompatible representations of JSON is prima facie evidence that JSON is in high demand, and that the lack of a standard is causing duplication of effort and compatibility headaches.

It is exactly to avoid duplication of effort that this library should exist, and also why it should have some somewhat priviledged position, assuming buy-in of some reasonable fraction of people who maintain the independent copies.

So now the burden of proof shifts back the other way: it looks very much like a good case for an addition. Let's hear compelling arguments why it's not. For instance,

  1. If nobody will adopt it, regardless of technical merit, it's pointless.
  2. If it is not technically possible to simultaneously satisfy the demands of many of the existing implementations, it's pointless.
  3. If there is insufficient buy-in to devote enough attention to make it work well in practice, then it's a bad idea.
  4. If we can quickly and effectively replace scala.util.parsing.json with something usable and fast, except this proposal impedes that, it may be counterproductive.

I don't think any of these are true, but if they were it's important to find out. If there are others, it's important to hear them.

@chaotic3quilibrium

+1 to Ichoran's excellent summary and more effectively framing the debate

On Fri, Nov 6, 2015 at 11:10 AM, Ichoran notifications@github.com wrote:

@milessabin https://github.com/milessabin - I agree, actually. I just
think the burden of proof has been met here: 8+ nearly identical yet
incompatible representations of JSON is prima facie evidence that JSON
is in high demand, and that the lack of a standard is causing duplication
of effort and compatibility headaches.

It is exactly to avoid duplication of effort that this library should
exist, and also why it should have some somewhat priviledged position,
assuming buy-in of some reasonable fraction of people who maintain the
independent copies.

So now the burden of proof shifts back the other way: it looks very
much like a good case for an addition. Let's hear compelling arguments why
it's not. For instance,

If nobody will adopt it, regardless of technical merit, it's pointless.
2.

If it is not technically possible to simultaneously satisfy the
demands of many of the existing implementations, it's pointless.
3.

If there is insufficient buy-in to devote enough attention to make it
work well in practice, then it's a bad idea.
4.

If we can quickly and effectively replace scala.util.parsing.json with
something usable and fast, except this proposal impedes that, it may be
counterproductive.

I don't think any of these are true, but if they were it's important to
find out. If there are others, it's important to hear them.


Reply to this email directly or view it on GitHub
#28 (comment).

@dickwall
Contributor
dickwall commented Nov 6, 2015

I have until this point refrained from commenting on this since the discussion, despite lacking professionalism in places, seemed otherwise healthy, and I didn't want to add opinion. What I will add is what I intend to happen to this discussion framed in terms of the actions the SLIP committee will take, at least while I have a say in it.

Probably this PR should have been a discussion issue first, and I will be amending the process to suggest that (let's discuss before we PR). It also should not have been given a number (the documents are already quite clear on that matter). The number makes it appear that the PR has in some way been ratified, accepted or approved and it has not.

The following actions are what should happen from here:

  1. The issue will be discussed at the next SLIP meeting. Clearly there is enough interest in it that the discussion should continue. It will be put into a 1 month official public review cycle.
  2. We may need to figure out some kind of survey mechanism (and try and make it fair) for this one to capture the will of the majority better. See issue #29
  3. The SLIP meeting after that we will re-examine and try and figure out the will of the majority fairly. If there is a clear decision to be made based on that, the SLIP will either be officially accepted or rejected at that point. If the decision is not clear, it will remain in public review for another month, etc.
  4. Accepting the SLIP is not the same as accepting the implementation into the library (core, module, recommended or otherwise, some of this still has to be sorted out yet - I suggest a separate issue for that discussion, not this one).

For now, the discussion seems healthy, the SLIP has not been accepted nor rejected, and the discussion will not be shut down. I believe that makes the position clear. If anyone still has doubts, please let me know and I will attempt to clarify further.

My request, please remain professional and civil in the discussions. Argue your corner and argue it well. In my opinion, non-professionalism in presenting your arguments weakens rather than strengthens your case.

Dick

@SethTisue
Member

We may need to figure out some kind of voting mechanism (and try and make it fair) for this one to capture the will of the majority better. Frankly I am not an expert on voting and anti-gaming, so I could use some help in this area to make it fair and unbiased. Volunteers?

But let's definitely not hash that out on this ticket! If anyone wants to discuss this aspect of the SLIP process, please open a new ticket in this repo, or a thread on scala-debate.

@dickwall
Contributor
dickwall commented Nov 6, 2015

Quite right - sorry Seth and everyone. Please see #29

@adriaanm
Member
adriaanm commented Nov 6, 2015

+1 on @Ichoran's great summary.

@mdedetrich

Probably this PR should have been a discussion issue first, and I will be amending the process to suggest that (let's discuss before we PR). It also should not have been given a number (the documents are already quite clear on that matter). The number makes it appear that the PR has in some way been ratified, accepted or approved and it has not.

This is my bad, will remove the number right now. Was writing this late at night and missed that part and I completely forgot about it after being bogged down in the discussion

@hohonuuli

@Ichoran

Writing is pretty important for data interoperability! And the Java version won't write NetCDF4 except by wrapping the C version.

That is true. I will point out that I've been working with NetCDF for about 18 years and have never needed to write NetCDF4 though.

@mdedetrich

@milessabin

@Ichoran the burden of proof should be the other way around: the default position should be against additions to the "standard" library unless extremely compelling reasons can be found for the opposite. Thus far no good reasons have been given why this library couldn't stand on it's own two feet perfectly well outside.

I don't think I have read one convincing argument that it shouldn't be included. Most of the exclusion actually tend to be one or two liners, or just -1. They also seem to come from an already assumed presumption that "nothing should be added" mainly due to personal and/or political reasons (i.e. I don't think a stdlib/prelude should exist, even though every major language has one, which arguably means this position should be argued for, not the other way around).

The most compelling (but not completely convincing, yet) argument is that the standard hasn't been adopted by the other frameworks/json libraries. This has however been clarified, since the SLIP is a proposal, there is nothing wrong with suggesting something before its being adopted

I am with @Ichoran , the clear majority of rejections can be summaries with a "no" or "I don't like this". A lot of rejections are also not applicable because it was clear that the person didn't actually read the SLIP, and so their concerns aren't really applicable (although people arguing for the SLIP have also done this)

@Ichoran
Ichoran commented Nov 6, 2015

@mdedetrich - Stronger indications that the proposal would be adopted would be very nice, though, if we are to accept it.

@mdedetrich

@mdedetrich - Stronger indications that the proposal would be adopted would be very nice, though, if we are to accept it.

This is mentioned in the SLIP, it was actually made with collaboration with almost all the major JSON library creators/web frameworks. The gitter log for json4s/json4s has an entire chat history demonstrating this, its also shown by the issues/some git commits. @non, @sirthias for example provided some very good performance techniques, and @eed3si9n provided some very valuable general feedback (such as the importance of ordering in specific rare practical situations). @non works with jawn, @sirthias with Spray and @eed3si9n wants to use an AST for SBT. We also got feedback from @farmdawgnation (lift), and play (@jroper). Then obviously there was the json4s community itself (@rossabaker primarily, who is also now spearheading http4s)

Personally I think I managed to push it as far as I could without it being "official". One of the goals of json4s-ast was to be ultra stable, so I didn't really want to release anything until it was good enough for everyone. I think the SLIP was good in the sense that we got a lot of feedback which I normally wouldn't have gotten otherwise, but more importantly, people took it more seriously (for better or for worse)

People can read the chat history themselves here https://gitter.im/json4s/json4s

@Ichoran
Ichoran commented Nov 6, 2015

@mdedetrich - That's good, but not very discoverable from reading the SLIP. Details matter, and direct quotes from people who matter are better evidence than your summary thereof. I found where James Roper indicates that he'd be inclined to adopt it. I haven't found any others.

I also think the naming is unacceptable (lots of confusion on this list, and on the previous gitter chat). Safe isn't always safe (you lose things, at least coming from fast) and fast isn't always fast (no O(1) map lookup).

Now that there is more feedback, if you think there are any good points at all, you should incorporate them and go back to the people who thought it was good before and make sure they still think so.

@mdedetrich

@Ichoran

@mdedetrich - That's good, but not very discoverable from reading the SLIP. Details matter, and direct quotes from people who matter are better evidence than your summary thereof. I found where James Roper indicates that he'd be inclined to adopt it. I haven't found any others.

Sure thing, will add this to the SLIP

I also think the naming is unacceptable (lots of confusion on this list, and on the previous gitter chat). Safe isn't always safe (you lose things, at least coming from fast) and fast isn't always fast (no O(1) map lookup).

We actually discussed naming a lot, this was the best we could come up with. Obviously happy if there are more valid names, just couldn't come up with them at the time

Now that there is more feedback, if you think there are any good points at all, you should incorporate them and go back to the people who thought it was good before and make sure they still think so.

Already happening https://github.com/json4s/json4s-ast/issues

@Sciss
Sciss commented Nov 6, 2015

@Ichoran OT - but re NetCDF - in our SysSon project we do write NetCDF files using the UCAR library, so that's definitely possible and it's not a C wrapper.

@Ichoran
Ichoran commented Nov 6, 2015

@mdedetrich - I'd suggest "strict" and ("lax" or "permissive"). That gives you a better idea of what the use cases are. "mapped" and "raw" would be my second choices. Maybe you can come up with others, but I think the existing names have demonstrated themselves as not being good enough. It'd be better to name them sqyrg and ziskob than what they're lableled now, as then people would be forced to read the docs to figure out what the distinctions are.

@sciss - I'm pretty sure that's NetCDF3 only. Anyway, thanks--I'll look into it more when I get back to that project. For now, we shouldn't hijack the thread any more.

@mdedetrich

@Ichoran
I have updated the SLIP to include quotes from people. If anyone wants their quote removed for any reason, please let me know and I will do it. They were taken from gitter, which is a public chat platform

@mdedetrich - I'd suggest "strict" and ("lax" or "permissive"). That gives you a better idea of what the use cases are. "mapped" and "raw" would be my second choices. Maybe you can come up with others, but I think the existing names have demonstrated themselves as not being good enough. It'd be better to name them sqyrg and ziskob than what they're lableled now, as then people would be forced to read the docs to figure out what the distinctions are.

Will make an issue on it!

@jedws
jedws commented Nov 6, 2015

Clearly the burden of proof needs to be in favour rather than against the proposal.

Merely showing that some people like the idea of a standard is not enough. Showing actual usage in the wild, would be helpful. Simply proposing that libraries could use it , or would use it if it was available – or _should_use it because it is officially blessed – is not a convincing argument. You need to show that the library solves actual problems.

@mdedetrich mdedetrich referenced this pull request in json4s/json4s-ast Nov 6, 2015
Open

Change name from fast/safe to something else #23

@mdedetrich

Clearly the burden of proof needs to be in favour rather than against the proposal.

Its not clear at all. Basically all common mainstream language have stdlibs (obviously to varying sizes). Languages like Python/Java/Ruby have huge stdlibs, and other (mainly older ones, like C) have smaller stdlibs

I think thats the actual problem. It may be anecdotally clear for yourself, but its far from generally clear. If basically almost every other mainstream language (ones with large enough user bases to matter) have a differing point of view, the burdern of proof is not on the SLIP process to argue that the stdlib should never be changed/frozen/removed, its on the people asking for it, and a lot of people who have been asking for it no/frozen stdlib haven't provided anything substantial

We should obviously be careful about what does end up getting added in (and there is a reason why future additions are being added as modules, and not a monolithic library, similar to how its being done to Clojure), but shutting down conversation from a prerogative which is not established, or clear, isn't helpful

You need to show that the library solves actual problems.

I have, many times. I have to deal with what is essentially 3 versions of String, which is giving me performance issues as I have to convert between them. I also have certain libraries frozen because those 3 frozen strings are moving at different speeds

@jedws
jedws commented Nov 6, 2015

You need to show that the library solves actual problems.

I have, many times. I have to deal with what is essentially 3 versions of String, which is giving me performance issues as I have to convert between them. I also have certain libraries frozen because those 3 frozen strings are moving at different speeds

No, you haven't at all. All you've shown is that you are introducing the 4th – even slower moving – version of String.

@Ichoran
Ichoran commented Nov 7, 2015

@jedws - Can you indicate what would constitute good evidence?

@jedws
jedws commented Nov 7, 2015

@jedws - Can you indicate what would constitute good evidence?

@Ichoran of what? Of the idea? The approach? Specifics of this API? The need for it in the standard modules/library?

None of these are well argued at the moment. The best arguments being presented at the moment are XKCD and bandwagon. Actual usage in the wild by multiple projects would be more compelling.

@adriaanm
Member
adriaanm commented Nov 7, 2015

@mdedetrich, thanks for patiently arguing your case, but I would suggest you focus your energy on refining your design, code & proposal based on technical input. It's looking great already!

This is not the forum for influencing our policy on how we evolve our standard library -- which is not up for a vote to begin with -- so let's not let ourselves get sidetracked.

@mdedetrich

@mdedetrich, thanks for patiently arguing your case, but I would suggest you focus your energy on refining your design, code & proposal based on technical input. It's looking great already!

Yup, already doing that!

@adriaanm
Member
adriaanm commented Nov 7, 2015

I know, I didn't mean to tell you what to do, but rather what to feel free not to do ;-)

@Ichoran
Ichoran commented Nov 7, 2015

@jedws - What would constitute good evidence that this library solves actual problems that warrant inclusion in a somewhat-officially-blessed module? Or of anything that you listed, for that matter?

You are not doing well at arguing that it is poorly-argued. The reason is that when you make an argument, as opposed to merely stating an opinion, you need to present some combination of relevant, factual, and sufficient evidence to support your conclusion. (If this claim seems controversial, I am happy to cite support.)

You have offered one question (recently): what need does it fill?

Here is one. All sorts of things have data stored in JSON. If I want to write, say, a Scala Asana API layer, my first step is to pick a JSON framework and then I either have to write oodles of abstraction layers and converters, or my API can only be used by people who use Play (let's say). This really puts a damper on my enthusiasm for building such an API. (This is not a hypothetical example. I would have actually built a Scala Asana API if there was a global JSON solution.)

You also have implied that the standard for inclusion in the standard library is that it's already widely used. But the claim here is that wide use is conditional on inclusion (following discussion with library authors). This makes sense: you can only gain the benefit of broad deployment if it's actually broadly deployed, while you immediately suffer pain from incompatibilities if you switch before others do. If that is true, asking that it already be widely used is unreasonable, or you need to indicate in what way this conditional acceptance is untrustworthy or undesirable.

@ghost
ghost commented Nov 7, 2015

By the same yardstick I could argue that TypeLevel should not contain specs2

@odersky Whilst I do get your general point, your choice of example is unfortunate as there is new typelevel library catalysts that abstracts the underlying test framework, helping test writers and consumers to choose freely.

And @milessabin is an official maintainer 😉

@fiadliel
fiadliel commented Nov 7, 2015

My bikeshedding opinion:

I believe from the discussion that one of the main aims of the SLIP would be to reduce the number of implementations of integration with existing JSON libraries (Play, json4s, argonaut, circe, etc.) that somebody might need to write to create a complete solution for some system.

So let's say, this aim is successful, and people only target this AST, rather than multiple ASTs. That in itself sounds great! But...

There have been examples earlier in this discussion of people pointing out the difficulty of making an AST that matches exactly the semantics of JSON; also that details can be important for, as an example, numerical computation.

I think you should be very careful before saying "we won't support that use case", and that people should use their own library instead -- because the AIM of this SLIP is that there will no longer be integration support for any such libraries. So it could make life a lot more difficult for some people.

And then the technical requirements for satisfying everybody can be... challenging 😄 It will be interesting to see the final design proposal.

@jeffmay
jeffmay commented Nov 13, 2015

I think we should compare the relative success for the language and community as a whole for standardizing on a library versus not.

Python, Ruby, JavaScript, (soon) Java, and Golang all have standard libraries for json. I believe it is a goal of Scala to make a static language feel more like a dynamic language (at least, I strongly appreciate this goal). One of the advantages the aforementioned languages have so much traction is how easy they are to use... like all the time, and not just when I setup some project directory structure. For example, if I want to make a webserver, I just type python -m SimpleHTTPServer. If I want to send json, I can just open up a REPL and fire off a request. If Scala does not have any Json library (assuming we remove the outdated one in the standard lib), then the only way to use it is to set up some sbt project, decide between some really great alternatives (a very time consuming task at many companies with also smart and opinionated developers who love to extrapolate the merits of choice to infinity) and then start coding your stupid simple script. My belief is that this hurts the community more than having a standard library version that is simple (even if to the point of being uncool), and then competing Json libraries that make this even cooler. I don't think it hurts to pull something basic like an AST in to the standard library. I know there is some rough edges around how to represent numbers, but that is precisely why "we the community" should come together around the recommended best way to do it. Just as we have for collections, string builders, and pretty much everything that we use all the time. Despite all of its problems, the Scala collections library provides a standard that allows people the freedom to work on something without having to make a choice. It is precisely this freedom from choice that high powered language users often forget about. Is it really freedom to have to sift through stack overflow discussions over Json library just to start on a project with your friend who is a newcomer to Scala, while they scratch their head over how complicated it is to make a simple HTTP POST request that they can write in JQuery in one line of code?

Having different use cases is fine. Once you can make a case for choosing between A or B you are helping the community. If you can't make a strong case for when to use A over B, even if it is because they are both good, then I think A and B are too similar to be useful. They stifle development and split the community. When a library is able to distinguish itself enough from a standard to become a new standard or provide a competing paradigm, then I think you have real evolution. I honestly can't see how we would not be able to agree on an AST being the standard way to build Json. Is there any good library out there that does not use an AST? If not, then why is there not a standard AST? How many different ways is there to represent JSON? In JavaScript, Python, and Ruby, there is only one way: an object, dict, and hash respectively. What is it in Scala? Can't we just pick one?

+1 to standardizing. I want Scala to be a language I can use to write scripts as well as big data applications.

@marekzebrowski

If goal is to make it easy why not include pickling or similar and show in every example
Map("a" ->1).asJson or something similar, and make the over way - parse json to Map or case class easy?
that is what exactly dynamic languages do.

@Sciss
Sciss commented Nov 13, 2015

@marekzebrowski as long as it is not scala-pickling in it's current state. I looked at it already three times I believe, it's still a mess, complicated and the code is horrible to read, things don't always work as expected, etc. This is ok for a journal paper, and I appreciate the research, but not for inclusion in the standard library (yet).

@marekzebrowski

@Sciss pickling is just an example. It can be done in any other ways as current libraries do it. Point is to make it easy to go from Scala representation to external JSON format and back. Otherwise everybody will still have to do it twice - from class to JSON AST and to external JSON. In result users will start to write their code against JSON AST instead of Maps, Lists etc. That will make things more complex, not easier. In JavaScript JSON IS the data representation inside the language. dict and object are roughly the same.

@sschaef
sschaef commented Nov 13, 2015

On 11/13/2015 10:10 AM, Jeff May wrote:

+1 to standardizing. I want Scala to be a language I can use to write
scripts as well as big data applications.
A problem that can't be solved through a stdlib. Just because other
people believe that it can be solved through a stdlib, doesn't mean
Scala have to make the same mistake and go that direction. I already
argued that this problem needs to be solved through a module system but
I can understand that most people don't want to think that far. Go ahead
with your AST standardization, but don't be mistaken by thinking that it
would solve the problems you pretend it would solve.

@Ichoran
Ichoran commented Nov 13, 2015

@sschaef - Can you justify your comment

A problem that can't be solved through a stdlib.

a little more? I use Scala for a lot of script-like tasks all the time, because I've built up a personal library that is good at those tasks. To me it seems like it's exactly the standard library which is at issue. For simple stuff like reading column 15 out of a CSV file, picking out all the numbers, and adding them up, I will spend literally 10x longer telling SBT what to get and waiting for it to get it than to actually do what I want.

@sschaef
sschaef commented Nov 13, 2015

On 11/13/2015 05:01 PM, Ichoran wrote:

@sschaef https://github.com/sschaef - Can you justify your comment

A problem that can't be solved through a stdlib.

a little more? I use Scala for a lot of script-like tasks all the
time, because I've built up a personal library that is good at those
tasks. To me it seems like it's /exactly/ the standard library which
is at issue. For simple stuff like reading column 15 out of a CSV
file, picking out all the numbers, and adding them up, I will spend
literally 10x longer telling SBT what to get and waiting for it to get
it than to actually do what I want.

The reason why people want a stdlib is because there is no simple way to
load dependencies regardless where they are coming from. If Scala would
have a module system that is backed into the language one could say
something like

import module "org.csvparser" %% "csvparser" % "1.0"
import org.csvparser.CsvParser

// use class CsvParser in Scala code

It would be the responsibility of the compiler to load the dependencies
and the responsibility of the build tool to do everything else. Loading
dependencies from non-local places also should be a no-brainer, it can
be done in the blink of an eye nowadays.

@Sciss
Sciss commented Nov 13, 2015

@sschaef What's the advantage of adding an import in a source file versus adding libraryDependencies += "org.csvparser" %% "csvparser" % "1.0" in build.sbt? That's equally simple, why would you want to have a language construct for that?

@Ichoran
Ichoran commented Nov 13, 2015

@sschaef - You're answering a different question, which is why a module system would solve the problem. Why will the standard library not solve the problem?

Also, that's way too much boilerplate. import module org.csvparser.CsvParser ought to be able to figure out where to look. If it's not, I agree with @sciss that it's not that big an improvement over SBT. (It is an improvement since you don't need the separate directory, build.sbt file, and so on, but it's less than it could be.)

@sschaef
sschaef commented Nov 13, 2015

On 11/13/2015 06:09 PM, Ichoran wrote:

@sschaef https://github.com/sschaef - You're answering a different
question, which is why a module system /would/ solve the problem. Why
will the standard library /not/ solve the problem?

It would only solve the problem for some time. The moment when someone
finds a better way on how to build a library or when simply a new
library appears (which may be used by a lot of people) the discussion of
what to include in the stdlib and even what to remove from it needs to
be repeated. A stdlib simply does not have a built-in update mechanism,
something a module system can have.

Also, that's way too much boilerplate. |import module
org.csvparser.CsvParser| ought to be able to figure out where to look.
If it's not, I agree with @sciss https://github.com/sciss that it's
not that big an improvement over SBT. (It is an improvement since you
don't need the separate directory, |build.sbt| file, and so on, but
it's less than it could be.)

First, don't be mistaken that even the most slightly reduction of
boilerplate can't make a huge difference in usefulness. A lambda literal
is also just a slight improvement over an anonymous class but among
other things it simply redefined how software is written in Scala in
contrast to Java.

Second, you do only compare my example with what sbt can do (or what you
are familiar with) but a module can be anything, not just a maven
artifact. A module system done right is a formalism that allows static
reasoning and dynamic re-compositions for any program. But that is a
discussion I would like to not have here, it is not exactly on topic.
Also I'm not yet finished by writing the spec for it, therefore I
couldn't present a waterproof proposal anyway. ;)

@hepin1989

@sschaef The module system you described here sounds great,The only thing I know could do something like that is http://lihaoyi.github.io/Ammonite/#ArtifactLoading.

Sounds you think the Json AST may be a module then?

@lihaoyi
lihaoyi commented Nov 13, 2015

One of my long-term stretch goals is to make Ammonite a viable dynamic execution environment to replace SBT. Clearly @hepin1989 has seen the same thing 👍

One step at a time though.

@Ichoran
Ichoran commented Nov 15, 2015

@sschaef - The standard library does have an update mechanism: cycles of deprecation and replacement and so on, and it can be formalized to whatever extent is desired. It's not as automatic as modules, but that doesn't mean it doesn't work, or that it isn't useful. Indeed, it solves a problem that modules don't solve, which is, "I'm new here, how do I do X?". If there's something in the standard library, it's really easy to answer that question (i.e. there is a curated "okay" way to do it--maybe not the best, but adequate, and curated adequacy is really valuable when you don't know your way around a particular space).

I'm not 100% sold on modules in all cases (I think they have the potential to mix concerns in a way that is counterproductive, or at least will make you lean a lot more heavily on git), but I think they do have really nice use cases esp. if they come with extremely minimal syntactic overhead. I agree that small wins in boilerplate can have disproportionate benefits. So can big wins. My argument was that the win you were proposing wasn't enough! Why should I have to know two different names for the same thing that I want? Are computers really so slow and primitive that they can't figure out where to get fastparse or scala-graph from? But though I think they're a nice tool to have in the toolbox, I don't think it replaces the need for a decent standard library. And I don't think that jeffmay's goal is one that won't be solved by a standard library and will with modules, that is, "the freedom to work on something without having to make a choice".

Modules make choices easier. You still have to make them.

@sschaef
sschaef commented Nov 15, 2015

On 11/15/2015 02:42 AM, Ichoran wrote:

Modules make choices easier. You still have to make them.

Once again, I think this is a tooling problem. A module system doesn't
prevent standard modules (or better: modules that are more suggested to
use than others). Built-in quality management is more scalable, see my
thought about this here:
https://github.com/sschaef/ide-research/wiki/Goals-of-This-Research-Platform

@dickwall
Contributor

There is clearly enough interest in this issue to keep it open for discussion. Further more, it will be assigned SLIP number 0028 and enter a public review phase.

Please assemble an expert group to answer questions like:

  • Scope of the effort (just a lightweight minimal JSON representation vs best of breed parser or anything in between)
  • Should it go into the Scala name space? Separate namespace?

It seems like accepted wisdom is that this would not be a core library implementation but more likely a platform library level.

Please read the full thread to familiarize yourself thoroughly with this issue.

@sschaef
sschaef commented Nov 16, 2015

On 11/16/2015 06:35 PM, Dick Wall wrote:

Please read the full thread to familiarize yourself thoroughly with
this issue.

What if I read it but already forgot what is mentioned there, do I have
to reread everything again?

@mdedetrich

Please assemble an expert group to answer questions like

I think the current list of active contributors/developers as listed here https://github.com/json4s/json4s-ast/blob/master/build.sbt#L41-L115 should be part of the group, how exactly does this process work?

Scope of the effort (just a lightweight minimal JSON representation vs best of breed parser or anything in between)

Personally I don't have an issue with implementing a parser, I would probably use some stripped down version of https://github.com/non/jawn to target the AST

Should it go into the Scala name space? Separate namespace?

I would argue that the AST should go into the scala namespace (something like scala.json) so it makes it obvious that people should target the AST, however the parser can sit in its own namespace if desirable.

It seems like accepted wisdom is that this would not be a core library implementation but more likely a platform library level.

Agreed

@densh
densh commented Nov 17, 2015

Scope of the effort (just a lightweight minimal JSON representation vs best of breed parser or anything in between)

In my opinion, we need to agree on AST first and then make an open call for parser proposals with clearly defined requirements & quality evaluation criteria (i.e. shared set of benchmarks and unit tests). Similarly to dotty's strawman proposals regarding collections redesign this can be used to obtain the best parser through competition and constructive discussion.

@SethTisue
Member

I agree that parsing should be a separate SLIP.

@SethTisue
Member

how exactly does this process work?

@mdedetrich this is all brand new, we don't have a structure for how these "expert groups" should work. (it isn't even clear we need to formalize that.) actually the phrase "expert group" doesn't even occur in https://github.com/scala/slip/blob/master/README.md . but "Help this is all too informal!" does :-)

if you want to discuss that, we could talk on a new issue at https://github.com/scala/slip/issues. (and mention @dickwall since I think this concept of expert groups is his?)

@mdedetrich

@mdedetrich this is all brand new, we don't have a structure for how these "expert groups" should work. (it isn't even clear we need to formalize that.) actually the phrase "expert group" doesn't even occur in https://github.com/scala/slip/blob/master/README.md . but "Help this is all too informal!" does :-)

I guess I am kind of unclear what goes on from here. Like am I meant to wait for this "expert group" to formalise so we can discuss the different adjustments for the JSON AST? Are we going to be using json4s-ast as a reference?

I just don't want to be doing work that is wasted. @dickwall , can you help out here?

@SethTisue
Member

I guess I am kind of unclear what goes on from here

I think we got this sorted out on Gitter: https://gitter.im/scala/slip?at=564d337675346dea413351d2

@eed3si9n
Member
eed3si9n commented Dec 5, 2015

As an sbt maintainer, I'd like the new JSON AST to be designed with long-term binary compatibility in mind. One way to do this might be to declare support window like 24 months and/or putting the major version number in the package name like scala.data.json1, and. Having long-term stability allows other libraries to depend on it more reliably. (e.g. Scala 2.9+ promising binary compatibility across minor releases)

Another requirement the AST should go for, I think, is literal round trip ability from a legal JSON string -> AST -> JSON string. Having more requirements on AST than the JSON spec doesn't interfere with the interop of JSON strings. If not the round trip (aka insertion order), at least we should aim for idempotency from the input type A -> String. Having referential transparency comes with many benefits, like caching.

(I was away from internet when this discussion was active, but I figured I'd put my 2 cents)

@mdedetrich

As an sbt maintainer, I'd like the new JSON AST to be designed with long-term binary compatibility in mind. One way to do this might be to declare support window like 24 months and/or putting the major version number in the package name like scala.data.json1, and. Having long-term stability allows other libraries to depend on it more reliably. (e.g. Scala 2.9+ promising binary compatibility across minor releases)

Definitely agreed. It kind of kills the point if the library is not stable. Regarding versioning, I am not sure if versioning by packages is needed, especially if the release cycle will be so long.

Another requirement the AST should go for, I think, is literal round trip ability from a legal JSON string -> AST -> JSON string. Having more requirements on AST than the JSON spec doesn't interfere with the interop of JSON strings. If not the round trip (aka insertion order), at least we should aim for idempotency from the input type A -> String. Having referential transparency comes with many benefits, like caching.

The current design does deal with this problem

Just letting you guys know, that I have been incredibly busy lately, however I will have a lot more free time in the holiday period to work on this

@furaoing

@jeffmay Huge +1, build-in language support of JSON is the common expectation of programmers with a dynamic language background (python, javascript, Ruby, etc).

@mdedetrich

Hey guys, thought I would give a little bit of an update from my side regarding whats going on. In regards to time I am about to move overseas, so I don't have as much time as I used to, I should be more free around march/april. I am also helping write a Scala book.

@SethTisue , @Ichoran would it be possible to give an update on the progress of this issue. As it currently stands

  • There is an implementation that already exists, which has been developed with feedback from (most) of the major framework providers.
  • I haven't received any extra feedback, technical or otherwise on the proposition that is outlined in the SLIP.

Should I re-namespace the current proposal, "submit" it to the community and see what they say?

@SethTisue
Member

@SethTisue , @Ichoran would it be possible to give an update on the progress of this issue. As it currently stands

I think we covered this in the scala/slip room on Gitter? Perhaps you'd like to summarize for this audience?

@mdedetrich

@SethTisue Sure thing

The current proposed AST at https://github.com/json4s/json4s-ast is going to be repackaged so its under the scala.json package.

Again to be clear, this is not going to be released as part of Scala stdlib. Its going to be a seperate module which uses the scala namespace, which is the exact same situation as scala-xml. It will also only contain a fast and safe variation of the JSON AST. The easiest way to explain the distinction is the difference between a String (safe) and Array[Char] (fast). One is immutable and mainly used for general use, the other is lower level and fast, meant to be used for performance reasons.

There are going to be some minor modifications to the AST, which include the following. Hopefully this will be submitted by next week

  1. Constructors and converters to Byte for JNumber have been removed. Since they aren't idempotent, they are confusing.
  2. For the apply of JNumber which applies to Double, if you have a NaN or Infinity it will return a JNull in the safe version of the AST. In the fast version, this is ignored, which actually means you will get a "NaN" or "Infinity" as a string stored in JNumber. This is done for performance reasons, however there is nothing preventing parsers from manually checking the validity of the double before constructing the fast JNumber
  3. The implicit .to[T] converters have been moved out of a package object (package objects aren't meant to contain classes/object).
  4. JNumberConverter is an abstract class (from a trait), this should improve binary compatibility

@travisbrown has raised an issue with JNumber being a BigDecimal, which is that stuff like JNumber("1e2147483647").to[BigInt] takes a huge amount of time to compute and it can cause security issues if abused for this reason. Although obviously an issue, this has more to do with the limitation of BigDecimal rather than the proposed AST, and I believe the best way to solve this problem would be to create a proper lazy number library for Scala (Scala doesn't have an unbounded lazy real data type in the stdlib). Creating custom number types is obviously out of scope for the AST.

One solution would be to document this, another would be to remove the .to[BigInt]

@travisbrown

Really expensive conversions: Just to clarify, my concern specifically is that library users working with JValue values don't necessarily have any idea that the underlying representation could be a BigDecimal, so there's no reason for them to suspect that to[BigInt] is dangerous and that they need either to avoid it or to sanitize user input.

If my library includes a method that allows valid user input to result in computations that run forever and consume lots of resources (both processing and memory), I'd want that to be painfully clear in the documentation, in the method name, etc.

circe addresses this problem by having BigInt decoding fail immediately for values above a certain size (by default 218 digits). It does this by introducing a pretty minimal new numeric type that isn't intended to be a new number library—just to solve this particular set of problems in a reasonably clean way.

Representing structure vs. decoding: More generally (and probably more importantly), I think the presence of the to method here muddles up different concerns. If this is purely an AST library, I don't see why it needs this decoding-related stuff, but if it's going to include some decoding-related stuff, it should do it responsibly and in a way that isn't going to make life harder for maintainers of JSON libraries with a more complete approach to decoding.

Other clunky number stuff: I have a number of other complaints about the representation of JSON values—e.g. 1e2147483648 is a perfectly legitimate JSON number, but I have to jump through hoops if I want a safe.JValue that represents it—this doesn't work:

scala> org.json4s.ast.fast.JNumber("1e2147483648").toSafe
java.lang.NumberFormatException
  at java.math.BigDecimal.<init>(BigDecimal.java:491)
  ...

In my view this kind of thing should be hidden from the library user (which is not that difficult to do).

Fast vs. safe: I'm also just not convinced that the fast / safe distinction is a good idea. Is there any concrete evidence that the fast implementation is actually faster in any meaningful way? Some of the specific choices about the representation are also kind of surprising or confusing—e.g. I wouldn't expect anyone to guess that lookup by key would be linear in the fast package but constant-time in safe.

Lastly, there's no reason a "strictly adhering" JSON object representation can't preserve field order. What I want is a fast, safe AST that holds on to a few useful facts that aren't strictly mandated by the spec (like field order, the sign of zeroes, etc.). circe has that now, and adopting this AST representation seems like it would be a step backward in every respect.

@mdedetrich

Representing structure vs. decoding: More generally (and probably more importantly), I think the presence of the to method here muddles up different concerns. If this is purely an AST library, I don't see why it needs this decoding-related stuff, but if it's going to include some decoding-related stuff, it should do it responsibly and in a way that isn't going to make life harder for maintainers of JSON libraries with a more complete approach to decoding.

I don't disagree with this at all, and I wouldn't have an issue in removing it (although if we end up using something like BigInt under the covers for large numbers, .to would need to remain so it hides the underlying representation.)

I made an issue about this and didn't get an overwhelming response to remove it json4s/json4s-ast#3 (comment). I think if we leave JNumber as a BigDecimal, we can remove it (and leave it to other libraries to implement their own converters), however if we use BigInt as a fallback for larger numbers, we need to provide the functionality

I have a number of other complaints about the representation of JSON values—e.g. 1e2147483648 is a perfectly legitimate JSON number, but I have to jump through hoops if I want a safe.JValue that represents it—this doesn't work:

Can you point to the source in your code where you do this. I may end up implementing it if there isn't too much ceremony in how it works

Fast vs. safe: I'm also just not convinced that the fast / safe distinction is a good idea. Is there any concrete evidence that the fast implementation is actually faster in any meaningful way? Some of the specific choices about the representation are also kind of surprising or confusing—e.g. I wouldn't expect anyone to guess that lookup by key would be linear in the fast package but constant-time in safe.

Fast takes up much less memory (uses mutable arrays under the hood). More importantly though, it doesn't do any runtime checking and it uses the minimal available datastructure (i.e. String for a JNumber). fast is something that is typically used for parsers, but it does address other concerns. @eed3si9n needs an AST that preserves the semantics of equality under conversion (i.e. if you go from String -> JValue -> String, you get the same string back). This means to represent JObject you need to use something like an Array or List (to preserve ordering, and possible key duplicates)

@mdedetrich

Hey guys, just letting you know that a git repo has been set up at https://github.com/mdedetrich/scala-json-ast, it also includes a gitter room, so feel free to ask questions/file issues!

@mdedetrich

Just giving everyone an update, the final design of the scala json AST is now out, the main change is that JNumber is now represented as a String (validated by a regex at runtime to make sure its a proper JSON number) with implementations of equals and hashcode to make to check for number equality.

Unless there are major objections to this last change, this will probably be the latest revision. The only thing that needs to be updated is to add some tests/benchmarks. Hashcode for the new JNumber may need to be update in later revisions to make the hash more consistent, however this won't break binary compatibliity

@SethTisue As far as I am concerned, from a design perspective the AST is pretty much done, I just want to add some tests/benchmarks to provide some greater confidence. When that is done, I think its a good time for a release?

@furaoing
furaoing commented Apr 6, 2016

Great work !

Rao Fu

On Apr 6, 2016, at 08:13, Matthew de Detrich notifications@github.com wrote:

Just giving everyone an update, the final design of the scala json AST is now out, the main change is that JNumber is now represent as a String (validated by a regex at runtime to make sure its a proper JSON number) with implementations of equals and hashcode to make to check for number equality.

Unless there are major objections to this last change, this will probably be the latest revision. The only thing that needs to be updated is to add some tests/benchmarks. Hashcode for the new JNumber may need to be update in later revisions to make the hash more consistent, however this won't break binary compatibliity

@SethTisue As far as I am concerned, from a design perspective the AST is pretty much done, I just want to add some tests/benchmarks to provide some greater confidence. When that is done, I think its a good time for a release?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub

@SethTisue
Member

I think its a good time for a release?

sure — ideally in tandem with an updated version of the SLIP itself, reflecting the latest discussions and design thinking?

@mdedetrich

@SethTisue The final design is now pretty much complete, I am going to update the SLIP itself to reflect the most recent changes (amongst other things). There may be more to be done (documentation, some more tests for confidence?) but I think its at a point where it can get a good looking at

@chaotic3quilibrium

How can I find a URL to the SLIP?
On Apr 17, 2016 8:08 AM, "Matthew de Detrich" notifications@github.com
wrote:

@SethTisue https://github.com/SethTisue The final design is now pretty
much complete, I am going to update the SLIP itself to reflect the most
recent changes (amongst other things). There may be more to be done
(documentation, some more tests for confidence?) but I think its at a point
where it can get a good looking at


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#28 (comment)

@mdedetrich

@chaotic3quilibrium Its in the SLIP, but for reference its located at https://github.com/mdedetrich/scala-json-ast

I will edit the title so its easier for people to spot

@mdedetrich mdedetrich changed the title from Adding standard Json AST to Adding standard Json AST - https://github.com/mdedetrich/scala-json-ast Apr 17, 2016
@OlivierBlanvillain OlivierBlanvillain referenced this pull request in jto/validation Jun 24, 2016
Merged

Replace json4s-ast with custom AST #62

@akkie
akkie commented Nov 26, 2016

What's the status of this proposal?

@mdedetrich

@akkie Afaik, the proposal has been accepted however we are waiting for the Scala center to release their plans for the SLIP process in order for this to proceed

@SethTisue @jvican Correct me if I am wrong or if something else needs to be added

@akkie
akkie commented Nov 27, 2016

@mdedetrich Thanks, great news!

Next SIP meeting is happening on the 29th of November. Agenda: Static SIP, SIP-24, SIP-27 and the Meta proposals SIP-28 and SIP-29.

https://twitter.com/jvican/status/798506152887730176

@mdedetrich

@akkie I am not sure if this specific SLIP will be discussed in those meetings, those are for sips afaik

@jvican
Member
jvican commented Nov 28, 2016

@mdedetrich Yes, correct, once the new Scala Platform process we will discuss this.
@akkie SIP meetings are meant to discuss proposals related to the Scala language, not libraries. Library changes will be discussed in our future Scala Platform meetings 😄.

@akkie
akkie commented Nov 29, 2016

@mdedetrich @jvican Thanks for the info!

@ktoso
ktoso commented Nov 29, 2016

Can't wait to see the SPP process kick-off, we have a strong interest and would back this proposal as the combination of Lightbend's libraries (as we discussed across the Play/Lagom/Akka/Spray teams internally).

Awaiting the new process then :-)

@mdedetrich

@ktoso Likewise!

Note that there is already a reference implementation if you want something to code against if you find that helpful. Its highly unlikely that it will change in the future, but I don't have a complete guarantee in this area

@SethTisue
Member

The SLIP process has been replaced by the new Scala Platform Process. Matthew has moved this proposal there: https://contributors.scala-lang.org/t/scala-json-ast-sp-proposal/175

@SethTisue SethTisue closed this Nov 30, 2016
@SethTisue SethTisue locked and limited conversation to collaborators Nov 30, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.