[DEP1] Implement SDLang support for package descriptions

[s-ludwig]

There is a (slightly outdated) draft for an SDL based package format. This should be implemented by separating the parser functionality in dub.package_.PacakageInfo into a separate module, where both, JSON and SDL are supported.

[fesor]

But what's wrong with JSON? Why not use yaml instead, it's well readable and have same pros as SDL.

[etcimon]

But what's wrong with JSON?

JSON is a data interchange format without support for comments. It doesn't self-document itself very easily either

Why not use yaml instead, it's well readable and have same pros as SDL

I was going to tell you there's no parser for it yet, but apparently this can fit the bill. There's more documentation so I'd prefer it personally, though it doesn't exclude the JSON or SDL from being a choice as well.

[s-ludwig]

I personally (really) don't like the fact that YAML mixes indentation sensitive syntax with ordinary JSON-like constructs. But worse is the complexity of the format. YAML files can be really hard to understand from the outside if certain features are used/mixed. Finally, it doesn't solve one of the potential goals of reducing the nesting level (although it does admittedly make it a little less painful).

See the forum thread for the previous discussion.

The brace based blocks of SDL also seem to match quite well with the D syntax, while YAML arguably feels a little alien in that ecosystem.

[dcarp]

I red somewhere in a thread that dub doesn't have the ambition to be a full-fledged build system.
On the other side the code.dlang.org is a great collection of software, that can be accessed by a generic build system.

Defining a "proprietary" package description will make more difficult the adoption of the dub registry in a generic build solution. As an example, there is work for supporting D and dub registry in CMake.

How often would you need a comment? Even the DEP has no comment example. I think there should be more advantages, than the 3 enumerated in the DEP, to justify a new package description format.

[s-ludwig]

Note that, for various reasons, accessing the JSON package description directly is not a good idea. The format is non-trivial to support correctly and the right way to go about this is to use dub describe, which outputs all information in a preprocessed and canonical format. This will also always output JSON, regardless of the format used for the package description file.

[dcarp]

How is dub describe output simpler as the current JSON? What information will be missing or consolidated?

[s-ludwig]

All things like configurations, build types, platform specific fields, inherited values from dependencies, dependency selection and so on are already done by dub describe. These and more things would need to be reimplemented by any tool that tries to use dub.json for building. Of course simple cases can work without that, but that would never result in solid support.

[marler8997]

I'd like to see this get done. Even though it would make the json file no longer valid json, I was going to look into enhancing the json format to:

1. allow comments
2. Make surrounding quotes optional
3. Make the root level curly braces optional (there's no reason to have them there).

However in researching the current issues I found this thread and discovered SDL. SDL looks like a perfect format. I'm willing to add the support for this, but I'd like to check that no one is working on this yet. If no one is I'll try to have the enhancement done in the next couple days. I just want an all clear before I put my time into it.

PostNote: If I do implement this, would you want me to import abscissa's sdl parser (using dub itself) or use pure source. I'm asking because I'm not sure how important it is for dub to not have any dependencies.

[s-ludwig]

That would be great. I won't have time for this for at least another two months. I'd like to have the code reorganized a little bit though, so that the JSON/SDL parsing is in separate modules and possibly using a common interface, but that can probably be done at any point of the process without much overhead.

[s-ludwig]

Oh, and there is one notable thing: The SDLang-D repository should be included using GIT subtree, so that there are no "external" dependencies. Not sure, but that is maybe not possible to do in a pull request, so I'll look into that tomorrow.

[marler8997]

I've never used a git subtree before because they seem pretty cumbersome. The way I understand it is developers who want to work on dub will need to manually update both their dub git repo, and each sub-tree independently. Have you used git subtrees before? Are you sure that's the way you want to go?

[MrSmith33]

Why not simply use Git submodules?

[s-ludwig]

One advantage that git subtrees have is that the external files are actually contained in the master repository. This makes working with them generally much easier (no submodule updates or checkouts required) and also doesn't break existing build and package scripts (this is the main reason in this case, because sub modules would break a lot of things). The only thing that is slightly more complicated is an update of a subtree. However, this is nothing that needs to be frequently done and it would be trivial to store the needed command in a little script.

BTW, missed the right link, this time in English. A list of the pros/cons of sub trees: http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

[Paxa]

Why can not use js format? It's like json, but can use single quotes, comments, and object keys without quotes. Looks more readable

{
  name: 'example',
  mainSourceFile: "httpserve.d",
  targetType: "executable"
  // no dependencies for now, will be added in next version
}

or another option to use some scripting language, as mruby, lua, js. Like ruby bundler

[MartinNowak]

Can we rediscuss using SDL please.
There is a simple YAML alternative called TOML and betting on SDL, a XML derived format that hardly anyone uses, seems like a bad decision.

Also see my comment here for why it makes more sense to use a JSON derived format.

There is a TOML parser https://github.com/iccodegr/toml.d.

[s-ludwig]

I've added a motivation section to the DEP: https://github.com/D-Programming-Language/dub/wiki/DEP1#why-sdl

In particular TOML is a lot worse at reducing nesting (although it's definitely an improvement) and wouldn't work for procedural elements with a more or less natural syntax. Also, although this is a matter of personal taste, SDL seems to be much closer to D syntax wise, which is a nice trait.

While TOML somewhat reduces visual nesting by using the [table] syntax, that also makes it awkward to use for things like sub packages:

name = "somepack"

[dependencies]
someLibrary = "~>1.0"
someOtherLibrary = {version = "~>1.0", optional = true}

# doesn't preserve order and either redundant or deviates from the JSON format
[subPackages.subpack]
name = "subpack

# using TOML syntax for sub sections also isn't very pretty
[subPackages.subpack.dependencies]
somelib = "~>1.0"

# fall back to JSON like syntax instead?
subPackages = [
   {
      name = "subpack"
   }
]

# the "arrays of tables" syntax looks better
[[subPackages]]
name = "subpack"
[[subPackages.dependencies]]
somelib = "~>1.0"

The latter syntax looks kind of alright, but I personally found it relatively hard to grok and it seems rather error prone. It also doesn't scale to deeper nesting levels. I also have a considerable dislike for formats that provide so many multiple syntax alternatives. It's confusing for newcomers and paves the way for non-uniformity across different packages.

Finally, and I have no idea how much of a difference this makes in practice, TOML states:

Be warned, this spec is still changing a lot. Until it's marked as 1.0, you should assume that it is unstable and act accordingly.

To me the only upside of TOML is it's presumably higher popularity, but does that really matter in practice? My guess would be that 90% of the people would have to learn a new language no matter which format we choose. IMO, popularity should only be used to decide between two otherwise "equal" contestants.

(edit: fixed some TOML syntax errors)

[marler8997]

@MartinNowak and @s-ludwig
It's been a while since we've talked about this. As I understand...JSON format will always be supported and SDL will become the "user-friendly" format. My question is, was there any reason that we couldn't support SDL and also enhance the JSON format to be more lenient...allow comments, unquoted strings, trailing commas? We could even fix the deep nesting problem by allowing duplicate keys. Adding support for LenientJSON would be very easy to implement, and would likely make people happy until we finish SDL or decide what to do next.

[s-ludwig]

The problem is that the JSON format is supposed to be machine readable, so it would require anyone who wants to read that format to adjust. In reality you could argue, though. Usually (lib)DUB would always be used to read the package description and would then either output a standard JSON description or just the runtime representation.

But really I think there should only be one format endorsed for users and trying to make JSON more comfortable would lead to a mess of formats.

[marler8997]

But really I think there should only be one format endorsed for users and trying to make JSON more comfortable would lead to a mess of formats.

I have a feeling that just about everyone who uses dub would love if it supported these 4 extensions to standard JSON. The disadvantage as you said, would be making it harder for other programs to parse/modify it, and would increase complexity by adding another format. This would leave DUB with JSON, LenientJSON, and SDL. If we only take these factors into consideration I would think supporting LenientJSON would be a win, but I guess that's a matter of opinion.

I do sympathize with wanting to be compatible with other programs. Adding another format makes everyone else's life more difficult. However, there are ways to accommodate this. Like you mentioned, Dub can convert whatever format we use to standard JSON. Then when a program wants to modify the JSON, it can use dub to convert the JSON to whatever format the user is actually using. Something like dub readpackage and dub writepackage.

[mihails-strasuns]

btw https://github.com/Dicebot/dub/tree/sdl head is capable of building a simple hello-world project based on dub.sdl (thanks @marler8997 for majority of work)
Just saying :P

In my opinion SDL is a huge readability win over any JSON derivative.

[MartinNowak]

Great, need help to finish it?
After all it wouldn't be too hard to add more supported formats, and endorsing SDL as default seems like a good choice to try.

[mihails-strasuns]

I could really use example of dub.sdl that features majority of dub parameters in a single file. Adding it straight to https://github.com/Dicebot/libsdl-d test files would help to ensure parser stability.

In general I only need to make a full read through and add test coverage before making PR but I don' t know when it will happen. Getting to existing state took me two evenings but those two evenings took two months :)