New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The iPKG format should be replaced with a known configuration file format. #1825

Open
jfdm opened this Issue Jan 3, 2015 · 30 comments

Comments

Projects
None yet
@jfdm
Contributor

jfdm commented Jan 3, 2015

For now the current ipkg format has served us well, however, with the interest in developing an Idris package manager the format should be replaced with a known configuration file format i.e. YAML.

YAML appears to be the choice de jeur for configuration files. Use of YAML will also allow for definition of a checkable schema for package files.

There are several advantages for switching to a known format. For one, it appears to provide the benefits of XML sans the horror's associated: extensible, well formed, types and namespaces, CDATA et cetera It is essentially, key-value pairs but on steroids and is more human readable than JSON. Secondly, support for a bespoke configuration file parser can be deprecated and work concentrated on using the contained values. Discussions over syntax also becomes deprecated to a degree.

The only disadvantage would relate to the Haskell library used. For example: not all libraries support schema checking, some require external dependancies and unsure if said dependancies are cross-platform. The PKG file parser and format need to be rewritten.

An example iPKG YAML file. Note the default options are explicitly represented here, and need not be represented.

This issue should be seen as a place holder for work in progress by myself. Though, if someone else completes this before me I won't be mad.

@suhr

This comment has been minimized.

Contributor

suhr commented Mar 8, 2015

YAML seems to be pretty complicated.

What about TOML? Rust's Cargo uses it, for example.

@jfdm

This comment has been minimized.

Contributor

jfdm commented Mar 8, 2015

I haven't heard of TOML. Can you clarify more, why you think TOML should be considered over YAML.

@jfdm jfdm changed the title from The iPKG format should be replaced with YAML to The iPKG format should be replaced with a known configuration file format. Mar 8, 2015

@suhr

This comment has been minimized.

Contributor

suhr commented Mar 9, 2015

TOML config example: https://gist.github.com/suhr/2a4747c3b5b3c1062094

I haven't heard of TOML.

https://github.com/toml-lang/toml

Can you clarify more, why you think TOML should be considered over YAML.

The main reason is it's more simple.

@ahmadsalim

This comment has been minimized.

Contributor

ahmadsalim commented Apr 11, 2016

@jfdm Do you think it is still relevant to change format?

@jfdm

This comment has been minimized.

Contributor

jfdm commented Apr 12, 2016

At the moment I do not think we should change the format it is pretty stable after all, but I am still worried about extensibility and maintaining a custom parser. At some point in the future we should revisit it certainly.

@LeifW

This comment has been minimized.

Contributor

LeifW commented Apr 12, 2016

I have work adding support for a yaml format in addition to the current
ipkg format I could push up. I had been hoping to have other tools read the
files e.g. dependency managers.
On Apr 11, 2016 11:51 PM, "Jan de Muijnck-Hughes" notifications@github.com
wrote:

At the moment I do not think we should change the format it is pretty
stable after all, but I am still worried about extensibility and
maintaining a custom parser. At some point in the future we should revisit
it certainly.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1825 (comment)

@siddharthist

This comment has been minimized.

Contributor

siddharthist commented Sep 27, 2016

@LeifW Any progress? It would be great to move to a standard format, rather than maintaining a custom one.

@jfdm

This comment has been minimized.

Contributor

jfdm commented Sep 27, 2016

I don't know about the work of @LeifW, but I do have something in the pipeline. Sadly, work commitments take priority, and won't get back to it until beginning of November.

@LeifW

This comment has been minimized.

Contributor

LeifW commented Sep 28, 2016

I can put up a PR for what I have later this week or this weekend. Jan looks like he's put more thought into this than I have, but perhaps my work can be helpful as a starting point.

@be5invis

This comment has been minimized.

Contributor

be5invis commented Apr 4, 2017

+1 for TOML. It is simple enough

@pierrebeaucamp

This comment has been minimized.

pierrebeaucamp commented Apr 9, 2017

Any thoughts on https://github.com/sol/hpack?

@wizzardx

This comment has been minimized.

wizzardx commented Sep 29, 2017

Hi there.

How about taking a page from how Nim and Nimble handle package management?

The'y're not perfect, but perfect can also be the enemy of good.

Central list of packages is maintained in a json file on github:

https://github.com/nim-lang/packages/blob/master/packages.json

People who want to add packages to the central listing can do a git pull request against the JSON file.

Each package is usually just a git or mecurial repo, but needs to contain a .nimble file with more details, eg for their SDL2 wrapper lib:

https://github.com/nim-lang/sdl2/blob/master/sdl2.nimble

The packages themselves are also listed nicely in the docs, pulled in via javascript to stay current, eg:

https://nim-lang.org/docs/lib.html#nimble-unofficial-packages

https://github.com/nim-lang/nimble

And then the package manager; it downloads the json file and uses that metadata.

It's fairly basic, but works pretty well in practice.

How about making a really basic idris package manger; eg "idris-pacman", that for v1 just uses that kind of basic scheme?

@xaviervia

This comment has been minimized.

xaviervia commented Sep 30, 2017

Configuration files are often manipulated by tools written in a very diverse set of languages, which means there is a clear plus in making them as interoperable as possible. Having a simple, widely known format for the configuration file makes it easier for an ecosystem to flourish.

With that in mind, I propose to go with JSON.

Every language in existence has a way of handling JSON, and it is not going away any time soon. It might not be the prettiest, but that is offset by the fact that there are so many tools to consume it, produce it and manipulate it.

@wizzardx

This comment has been minimized.

wizzardx commented Sep 30, 2017

There's some precedence there: The D lang packaging format:

https://code.dlang.org/package-format?lang=json

eg:

{
	"name": "myproject",
	"description": "A little web service of mine.",
	"authors": ["Peter Parker"],
	"homepage": "http://myproject.example.com",
	"license": "GPL-2.0",
	"dependencies": {
		"vibe-d": "~>0.7.23"
	}
}
@xaviervia

This comment has been minimized.

xaviervia commented Sep 30, 2017

That's verbatim the same as Node’s package.json format. And there is something to be gained from sharing practices: after all most of the keys in the package.json format from Node are general purpose description of programming libraries and have nothing specific about JavaScript or NPM in them. @wizzardx , I’ve seen that same format being adopted more and more for project manifest files outside Node.

I know I’m biased coming from a web-heavy background, but I time and again I find that nobody has a problem with having to deal with JSON when coming to the web since everybody that has been programming for a while bumped into it before and knows how to use it.

@jdevuyst

This comment has been minimized.

Contributor

jdevuyst commented May 5, 2018

How about making Idris the file format?

When I propose this, I’m inspired by Boot, which is build tool for Clojure where the build file is also written in Clojure.

Here’s how this could work. The package.ipkg would be a regular Idris file except that it has an entry point package : PackageInfo.

PackageInfo would be a record type that contains the program name and other metadata. Additionally it could have various IO values, which are ‘tasks’ that serve various purposes:

  • Download (and potentially patch?) dependencies
  • Compile with various configurations
  • Run tests
  • Start the REPL (perhaps with some code injected on launch)
  • Run the Makefile
  • ...

It might then of course help to add a bunch of ‘batteries’ to make writing these tasks easier. The batteries could be convenience functions for downloading and installing packages, for launching a REPL, etcetera.

I do imagine that safety could be a concern. However:

  1. Safety at compile time is already non-existent given that ipkgs may cause make to be called and given that type providers allow I/O to be performed during compilation.
  2. If we want more safety pre-compilation then perhaps that could be solved by adding a ‘safe’ mode to Idris. In this mode really_believe_me, assert_total, unsafePerformIO, and so on might be disabled. The ipkg file could be evaluated in such a mode, meaning that the only unsafe operations would happen through the IO monad (and if that’s still too dangerous, we could add a SafeIO monad).
@luc-tielen

This comment has been minimized.

luc-tielen commented May 17, 2018

I've been tinkering with a "package manager/build tool" of sorts for Idris, but I would really love a standardized format (or natively written in Idris) that's properly supported by the language itself.

This is one of the main things preventing me from building larger projects..
I like @jdevuyst his suggestion, that or the YAML/TOML approach..

@david-christiansen

This comment has been minimized.

Contributor

david-christiansen commented May 17, 2018

The more I learn about YAML, the less it seems like a good idea for something like this. YAML is extremely complicated.

The approach of internalizing the metadata proposed by @jdevuyst sounds ambitious, yet promising. Otherwise, TOML is quite a bit simpler than YAML, while still being friendly for humans. Something simpler than that would be even nicer, as we probably do want to be able to write verified Idris code to process these things in the future, but OTOH the best parser is the one you don't write yourself.

@LeifW

This comment has been minimized.

Contributor

LeifW commented May 17, 2018

You don't have to use all of YAML, just a simple subset. YAML is already used as the package format by a lot of related tools, e.g. stack, hpack, etc. I have a branch that uses yaml instead of .ipkg, and I like how it looks. I guess TOML is used by rust, but I don't have as much experience with it. Ultimately, I'd like something extensible, so I can add other metadata to project files. And also easily parseable by other tools.

@david-christiansen

This comment has been minimized.

Contributor

david-christiansen commented May 17, 2018

The problem with picking a simple subset of YAML is that you're no longer using YAML, so any benefits associated with a reusable format go out the window. It becomes yet another proprietary format, with extra gotchas associated with the difference between how it seems and what it actually is. When written tastefully, YAML definitely looks good, but it's really a very complicated format.

Extensibility is a good goal, however, and whatever we get should support that. If we do get an Idris value representing a package, it'd be pretty easy to render it into whatever format was convenient for any old external tool, wouldn't it?

@jfdm

This comment has been minimized.

Contributor

jfdm commented May 18, 2018

As the original submitter of this issue, I have been musing it for quite a while. My own, publically available, build/packaging helpers for idris do use yaml. I even have a half finished branch which replaces Idris' own format for YAML. I think i was stuck on parsing namespaces.

I too have some misgivings about using YAML, exactly what @david-christiansen has mentioned, but I think we can work with the format regardless.

However, I am not a fan of encoding idris' package metadata in the language itself.

I'll see if I can polish off the half finished branch over the next few weeks and see where I am at. I don't want this issue just being about discussing things, it would be nice to see some action.

@david-christiansen

This comment has been minimized.

Contributor

david-christiansen commented May 18, 2018

I certainly don't want to get in the way of work here that people will find useful. I've registered my concerns.

@doofin

This comment has been minimized.

Contributor

doofin commented Jul 9, 2018

I strongly agree with @jdevuyst . Scala 's sbt and new build tool mill is also build around this idea. No need to use other file format, just idris.

@doofin

This comment has been minimized.

Contributor

doofin commented Jul 12, 2018

If you want write idris everywhere,check https://github.com/idris-industry/ikan

@f-f

This comment has been minimized.

f-f commented Aug 11, 2018

I was wondering about Idris package management and ended up on this issue, so I'll add my perspective (sorry for not contributing more practically): some projects are adopting Dhall (Haskell implementation/bindings here) as the language for their package management, e.g.:

  • Eta (Haskell on JVM) will have it as the default language for build config, replacing .cabal, see here
  • PureScript: there has been discussions about supporting it in psc-package (the main package manager, currently only JSON) and there is a new package manager entirely based on Dhall, Purcel

You can think of the language as "JSON + functions + types + imports", and it's not Turing-complete - so safety is not a concern anymore - and can easily render to JSON/YAML/text/etc.

(Disclosure/disclaimer: I'm a contributor to the lang)

I used the boot-clj tool described above to build some Clojure projects, and in the end I stopped using it, as there is some drawbacks to the "single language for everything": basically you're now writing in a full-fledged programming language and the build/package config is an actual program, so its logic requires maintenance, can get too complex and need refactoring, there are security concerns, etc etc.

@ahmadsalim

This comment has been minimized.

Contributor

ahmadsalim commented Aug 12, 2018

I have seen a lot of discussion about approaches in this issue.
While we appreciate a good discussion, it would be nice to move forward with some approach.

If someone is genuinly interested in making a PR to a particular language, please start a discussion on the mailing list or on IRC. It's probably easier to guide people using a particular approach, and then we can get closer to closing this issue 😄 .

It might be interesting to have something that works well with Blodwen, which is most likely going to be the next version of Idris.

@clayrat

This comment has been minimized.

Contributor

clayrat commented Aug 12, 2018

See also this thread:

As far as I know, there are three Idris-specific package management systems in development:

  • idream, written in Haskell. This one is the most usable at the moment, though development has slowed down a bit lately. It takes strong design cues from Purescript's excellent psc-package.
  • elba, written in Rust. Large amounts of implementation have been done already and development is the most active of these options, but it's currently unusable as a command-line tool. I'm one of the authors of this tool, and in my (very biased) opinion, I think it will nicely synthesize the features of Rust's cargo (general ergonomics), Dart's pub (dependency resolution), and Haskell's cabal (nix-style global caching).
  • ikan, written in Idris by @doofin. This one is still very much in very early development, though it's also fairly active. It's infuenced by Scala's build tools (sbt and mill) as well as Clojure's Leiningen, and its internal design is the most cutting-edge of all three, using the most advanced abstraction tools (embedded DSLs, Free monads, etc.)
@clayrat

This comment has been minimized.

Contributor

clayrat commented Aug 13, 2018

This paper might also be interesting:
https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf
Mokhov, Mitchell, Peyton Jones, "Build systems a la carte"

@doofin

This comment has been minimized.

Contributor

doofin commented Aug 14, 2018

@clayrat great paper to look at, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment