When to release Elvish 1.0

[xiaq]

Last update: 2023-03-08

Here are my thoughts on what Elvish 1.0 means and when to release it.

Background

Elvish has, up until then, been designed and implemented predominantly by one person, namely me. I certainly don't mean to belittle the contributions from and discussions with the community, nor do I reject a future where Elvish becomes more like a community project. Nonetheless, I am the only person who can push directly to the repo, and I have never merged any code that I don't personally like.

Having a small team of designer for a language is a better idea than it might sound (and 1 is the smallest you can get). For instance, C was designed by Dennis Ritchie; Go by Robert Griesemer, Rob Pike and Ken Thompson; Clojure by Rich Hickey. None of those languages are perfect, but they are more coherent and tend to be smaller than languages designed by committees. (Examples abound; Scheme is a notable, perhaps only, counter-example).

To conclude, before the Elvish community evolves significantly, Elvish will remain designed (and "controlled", if you will) predominantly by me. This is both a constraint of reality and a good thing. As a result, this design note and potentially many others to come, will be colored by my honest personal opinions.

With that out of the way let's get to the actual topic...

What Elvish 1.0 means

I treat the significance of 1.0 as pretty much the same way as Go treated its 1.0, namely there is a compatibility promise. What this means is the following:

• There will be a more or less rigorous language specification (this is Go's);

• Any code written against that language specification will continue to work in future releases of Elvish 1.x.

In other words, once a language feature is in Elvish 1.0, it will be in all of Elvish 1.x releases. One can workaround that by explicitly marking some features as unstable and not documenting them in the language specification, but I want to avoid that wherever possible.

I would also think that Elvish 1 should have a lifespan of at least 10 years from the 1.0 release. Combined with the compatibility release, this means that Elvish 1.0 language needs to get things as "right" as possible, because once it is released, I will stick to such features for 10 years (this is not a promise, but rather a goal).

Also note that the UI experience is not put under the compatibility promise. My hope is that by Elvish 1.0, the UI will have become modular enough that it can have its own versioning, and breaking changes in the UI is more acceptable than the language.

What are still not right

So the reason we haven't had an Elvish 1.0 release yet is, of course, some parts of the language are still not "right". This is a strictly subjective matter, and this is where my personal taste kicks in.

Here is a preliminary list of things I don't find "right" yet. As anyone familiar with programming languages will notice, all of those problems are already understood within the PL theory community, and many have multiple solutions. However, the real problem here is a design one: which solutions to pick and mix to make Elvish feel simple and coherent.

• A story for writing safe, concurrent code. The pipeline is a great mechanism of concurrent programming, but it doesn't cover everything. Elvish should provide a rich system of memory-safe concurrency primitives.

  • Have clear rules around when variable access is guarded by mutexes.
  • Have first-class value channels (today channels are parts of pipes, but the pipe made by file:pipe doesn't have channels)
  • Maybe STM. (Clojure has STM, but Rich Hickey said that it's rarely used in practice as people just bundle related states together and make them atomic.)
  • Maybe some sort of reactive programming system. This feels sorely needed when designing the editor API, but I'm not yet sure about use cases beyond that.

• Some sort of type system. Some elements I like from more recent languages:

  • TypeScript's type system, especially the structural parts.
  • Clojure's spec. It's not a type system, but has some promising ideas
  • Shallow and optional typed Racket has some recent research on the tradeoffs of soundness and performance.

• Make it easy to distribute software written in Elvish. The current epm package was implemented with the use case of a traditional "personal plugin manager" in mind, and doesn't really help with distributing standalone scripts. Some elements I like from recently designed systems:

  • Clojure's deps is a small but comprehensive package management system.
  • Deno's remote import makes module dependencies part of the source file.

• Make it easier to write portable shell scripts.

  • A more comprehensive stdlib for interacting with the OS without using external commands
  • Good support for detecting OS differences and external command versions. (Maybe this doesn't have to be builtin, but rather something implemented in Elvish code.)

• Builtin testing support.

• A better story for handling exceptions.

• Good support for REPL-aided dev flow, a la Lisp (if you don't know what this means, this page in Clojure's website may be useful). This isn't purely a tooling matter: language design choices need to be compatible with REPL tooling.

That's it for now; the list will likely grow and shrink over time as I update this post to reflect the status.

[xiaq]

Addendum to what's still not right:

• A system for classifying errors. Perhaps not so much as a "system", but a coherent guideline. I am mainly familiar with those of Python and Go and neither is exactly satisfying:

  • Python provides a few "standard" exception types like ValueError and TypeError. However, they are way too vague and don't provide nearly enough signal when it comes to fine-grained error handling. Programmers can also define their own exception types, but there is not much guideline on how to use them.

  • In Go there are basically two ways you can do errors: 1) build adhoc errors that are basically just glorified messages, via fmt.Errorf or errors.New 2) implement a dedicated type for a class of errors. An example of this is os.PathError. Sometimes, functions are provided to distinguish certain classes of errors from others; an example of this is os.IsPermission. All of these techniques in Go seem to work fine for their specific use cases, but then there isn't much uniformity; programmers are left on their own to decide how to report and classify errors.

[xiaq]

Another more interesting question that I haven't "got right" is the ability of defining new types from Elvish code. (There currently isn't any.)

• On the one hand, Elvish has maps and lists, which can pretty much carry most shapes of data (never mind efficiency); this is indeed how data in languages like Clojure and JavaScript are typically represented. Maps and lists are easy to understand, inspect and serialize. Behaviors cannot be attached to them; instead, you can write functions that manipulate them.

• On the other hand, Elvish has a few builtin custom types, like styled and styled-segment (the type of keys in binding tables are also custom types, but they are not exposed to Elvish code). Those custom types are really just Go structs that implement some methods, which makes sense as this is indeed how you define new data types in Go. They also have some custom behaviors; notably for styled, it knows how to turn itself into ANSI sequences when printed onto the terminal. This is something that maps and lists cannot do unless we introduce something along the line of metatable in Lua.

Another way to summarize this is that there are three different requirements that pull the design in different directions:

• Interoperability with Go code -- including, notably, the implementation of builtin facilities. This encourages defining custom types, because Go code typically defines new shapes of data using struct, and new behaviors are implemented as methods of those structs.

• Uniformity and simplicity within Elvish. This favors having as few custom types as possible -- use lists and maps for everything.

• Go/Elvish code parity. Go code can define custom behavior (styled being the arch-example), Elvish code cannot. This is not a good scenario to be in, as builtin facilities (which are Go code) have access to magic that is inaccessible to Elvish code.

[dumblob]

1. WRT state sharing between concurrent threads - I'd say first implement something simple (an MVP) and later maybe something more complex.

   I'd first go for CSP-style channels as that should be quite easy to expose from Go. But ideally I'd probably go for something like V's shared & (r)lock blocks (they can be nested and guarantee safety!) which is presumably more time-consuming to implement.

2. I'm a bit skeptical about defining new types. I'd prefer the direction "Uniformity and simplicity within Elvish." out of the three choices. I'm too new to grasp everything in Elvish and all the reasons why things in Elvish are as they are but intuitively I'd say such a high-level shell doesn't need any dedicated support of "working with types" in syntax (not even lists/maps/integers/...) as it's anyway futile imagining those thousands of types utilities produce in pipes.

   My idea always was to add own "method" (i.e. new functionality) to an existing shell command (thus leveraging its existing implementation details!) which would have a standardized name/API and which would ensure compatibility on demand. On of potential ways how to implement it could be as follows:

   1. each command has to accept pipe data in the most generic format (I'm deliberately not saying what format it is, but I'm sure it has to be extensible at least in compile time as newly defined commands can require new data types which won't fit the existing set of data types the generic format will support) - imagine something like TLV or any other stream-oriented extensible (unlike msgpack/cbor/...) efficient (in terms of native binary data support unlike JSON/XML/...) format like uSX or ston

   2. each command is in addition to (1) encouraged to accept also data in other formats - especially the one format which the command itself finds most efficient to work with (targetting zero-copy)

   3. each command is required to produce pipe data in the most generic format (the same one as in (1) )

   4. each command is in addition to (3) encouraged to produce also data in other formats - especially the one format which the command itself finds most efficient to work with (targetting zero-copy)

   5. at the beginning of the pipe plumbing both producer and consumer will try to find the most efficient way to exchange data

   This is to avoid any need for conversion commands in between while at the same time offer run-time detection of the most efficient way to produce & consume data.

   Btw. having separate filtering commands as the primary recommended measure to filter data in pipelines might sound very tempting but it actually makes things less composable in practice considering that non-filtering commands will be extended over time which will result in new filtering needs and these will at some point be impossible to satisfy in the generic context of a "filtering command" and will be satisfiable only in the contexts of non-filtering commands.

   Another reason is that filtering commands will never be able to efficiently support the special data types and thus everything would in practice just collapse to using the generic format significantly harming performance. Actually this whole idea might be implemented as an extension subcommand for "generic" filtering commands but the "main" implementation would be rooted in the non-filtering command.

[iandol]

which would have a standardized name/API and which would ensure compatibility on demand ... "each command has to accept pipe data in the most generic format"

I think this would be great. Although the concept of a 'pipeline' is the first one greeting us on the elvish home page, most elvish commands are not pipeline aware (why there is a rather cryptic all (one) to feed an each): all and nested each tare required to get pipelines working across commands. Being more pipeline friendly would make elvish more expressive, see for example nushell which has really expressive and semantically obvious pipelines: df | detect columns | drop column | into filesize 1K-blocks Used Available — @xiaq — you don't mention it above so it seems you prefer for the elvish design to be a pipeline-optional approach rather than a pipeline-first approach?

[ilius]

Python provides a few "standard" exception types like ValueError and TypeError. However, they are way too vague and don't provide nearly enough signal when it comes to fine-grained error handling.

I was surprised and disappointed when I learned that ValueError doesn't even have an attribute/property (passed as optional argument) for the value that caused the ValueError!