RFC: A Path Forward for Ammonite REPL and Scripts in 2023 and Beyond

[lihaoyi]

Ammonite's a pretty old project, coming up on 10 years now (first commit Jan 2015, with work starting in 2014). It's accumulated a lot of cruft and features that aren't useful, design decisions that proved to be dead ends, and things that have become redundant now that the Scala ecosystem has moved on. So it's worth asking the question: what purpose does Ammonite serve at all?

What Didn't Work Out

1. Ammonite is no longer truly necessary as a "non-SBT" Scala runtime environment. We now have Scala-CLI for small scripts, and Mill for larger projects. Mill now also bootstraps itself and no longer depends on Ammonite to run it's build.sc, allowing it to avoid many of the pitfalls below.

2. Ammonite-Shell is a dead end. Nobody ended up replacing their Bash shell with Ammonite, myself included. This has already been removed. Ammonite-Ops was removed and has been largely replaced by OS-Lib

3. Ammonite's "Dynamic First" script running strategy is a dead end. Previously it fit well into the Ammonite-Ops/Ammonite-Shell "Ammonite as a system shell" approach, allowing you to mutate the script runtime environment (compiler flags, classpath, etc.) while they were already running. This has proved to be a dead end, and the architecture prevents improvements like Refactor script-import system to allow pre-compilation without execution #437

4. Ammonite multi-stage scripts proved to be very confusing. Even if the functionality is needed, the current syntax is unacceptable.

5. The Scala 3 REPL is no longer as crappy as it used to be in Scala 2, and even the Scala 2 REPL has improved: JLine now supports multi-line editing in all Scala versions, Scala 3 comes with syntax highlighted input, etc.

6. Ammonite scripts not being integrate-able into larger Scala projects built using SBT or Mill is a problem. You cannot easily share code between an Ammonite script and an SBT or Mill subproject

7. Ammonite's classpath has grown bloated with all it's functionality, e.g. pulling in SemanticDB with its own version of protobuf and other things, which makes including it on the classpath more conflict-prone than before

8. Saving/loading sessions proved to be not useful, despite being a cool tech demo for what you can do with classloaders

9. Magic imports are great for limited use cases like import $file or import $ivy, but are too inflexible for more complex configuration: setting compiler flags, JVM system properties, assembly/packaging instructions, basically anything that requires configuration more complex than a single string. Scala-CLI came up with their own Directives syntax, which I feel is more a sideways change from magic imports rather than a strict improvement

10. Most of Ammonite's scripting functionality is duplicated in Mill, and Mill generally does a better job: incremental compilation, Scala-JS/Native support, IDE support, etc.

11. Scripts are no longer so unique in Scala 3, which allows some degree of top-level definitions

12. Most people who embed Ammonite as a REPL do not want a huge script-running apparatus attached to it; to them, the ability to run scripts and cache them globally etc. is a downside rather than a feature

What Did Out, and is Great

1. Ammonite scripts as "one file" to "a few files" programs with built-in dependency definitions, without needing a separate build file

2. Ammonite @main methods are great, and have been extracted into MainArgs, are widely used by Mill and other projects, and were shamelessly copied for Scala3's main methods

3. Embedding Ammonite to be in the middle of a larger program for debugging or interactivity

4. Ammonite web notebooks e.g. via Almond

5. Pretty-printed output, which is still missing in the Scala 2/3 REPLs

6. The pre-bundled com.lihaoyi libraries are great: requests, upickle, os-lib, etc. make the Ammonite REPL a much more usable out-of-the-box experience than a vanilla Scala REPL with a blank classpath.

Notes

1. It seems likely that the gap between Ammonite and the built-in Scala REPL will shrink over time. It's unclear if they'll eventually implement everything that Ammonite implements, but we can probably expect them to implement more of it over time

2. Scala-CLI's directives syntax is pretty nascent: in discussions on scala-contributors, they explicitly called it out as not something they want to standardize at this point. It also does not follow any broader standard. That makes me reluctant to try and follow suit, as it may result in us chasing a moving target

The Path Forward

1. Remove the bespoke script-running functionality from Ammonite, and move it to Mill. Mill already needs script running for build.sc, and generalizing it slightly would allow it to work as a script runner for arbitrary small Scala scripts with some reasonable defaults. This should be a trivial amount of code - probably <1000 lines - on top what Mill already has

2. Split Ammonite into REPL-only and REPL+script distributions, keeping the REPL lightweight for easy embedding

3. Embed Mill into Ammonite to provide script running functionality, for backwards compatibility. This basically reverses the old "Mill depends on Ammonite" strategy, but by putting it in the REPL+script Ammonite distribution we can keep the REPL-only Ammonite distribution small and easily embeddable. This would remove some of the more dynamic behaviors that Ammonite Scripts support, but that's probably for the better.

4. Replace the magic-imports syntax with a slightly more general structured header format, similar to e.g. The YAML Header format. This should be a thin syntactic layer on top a Mill build (which is already serialized to a hierarchical JSON structure) and would allow configuration and overriding of anything Mill allows you to configure or override. We would keep the original syntax for backwards compatibility.

Outcomes

1. A much smaller and simpler Ammonite codebase overall: Ammonite would only really have logic around the REPL, with a thin shim wrapping Mill to provide scripting functionality.

2. Both Ammonite and Mill would be able to focus on what they do well - respectively providing a great REPL and compiling Scala projects - without duplication between them.

3. Ammonite's REPL would be far smaller a dependency than it is today, making it easier to embed without worrying about classpath conflicts. We could even shade any com.lihaoyi libraries we use to make it entirely self contained, reducing the chance of classpath version conflicts to zero

4. "Ammonite Scripts" would become thin wrappers around a Mill build - similar to how Mill's own build.sc is a thin wrapper around a Mill build - avoiding duplication, improving cross-learnability, and making it much easier to port from one to the other.

5. The new Ammonite/Mill scripts would provide all features of Mill, including publishing to Maven Central, packaging assemblies, Scala.js, Scala-Native, etc. basically for free. All these are things which are impossible with the current Ammonite Scripts architecture

6. As the built-in Scala 3 REPL improves, we can gradually shrink the Ammonite codebase, as the necessity for custom things on top of the built in REPL recede.

CC @alexarchambault (ammonite) @lefou @lolgab (mill) @smarter @jodersky @odersky (scala3) @lwronski @MaciejG604 (scalacli) for your thoughts on this

[lefou]

I mostly agree with your vision / outline. I never used Ammonite that much, beside being it the foundation of Mill for so long, so I have a rather unemotional opinion about it. (For one client, I have it embedded into a server application. A smaller and simpler codebase / package setup would be great.) There is one topic you mentioned, to which I do have some thoughts: the script configuration. I had my difficulties with the imports, and I also took part in some discussions about the using directives introduced by scala-cli. I even experimented with them in Mill. Here are some thoughts.

1. Configuration options should keep the script files parsable / editable / compilable with normal Scala tooling. This was the greatest issue with the magic $-immports of Ammonite. Any action more that simple syntax highlighting or formatting failed. This is also the reason, why I find the using directives interesting. They are completely transparent to the Scala Language. The YAML header you proposed may look like the better technical solution, but will break the plain Scala syntax one would expect in a .scala or .sc file. Of course we could surround it with Scala comments, but there is the risk that the next formatter messes it up. That's where the using directives in their current form as one-line comment are better, as they are less prone to this issue.

2. Keeping context in the configuration. Both, Ammonite imports as well as using directives, are specific to their interpreting tools, yet they don't communicate this dependence / context. When opening a source file, there is no clear sign whether the file is supposed to be processed by Scalac, Ammonite, Scala-cli, Mill, Almond, or any other workbook editor out there. You need to more or less know or infer it. I'd like to have some clear context, something indicating for which application the configuration is meant. E.g. instead of //> using key value it could be //> scala-cli key value or import $ammonite.file instead of import $file. Then, it is always clear to which tool these instructions are. We could even support multiple contexts, so that the same script may run with different tools. Also Yaml is well suited to embed the context (or multiple of it) in its hierarchical format.

[SethTisue]

I think the core Scala-CLI directive syntax is not in flux and will not change. I'm puzzled that you would characterize it that way. It's been through the SIP process, and Scala-CLI has reached 1.0 status. Some of the more advanced particular directives could still evolve, but the core syntax and core directives are firmly in place.

It seems likely that the gap between Ammonite and the built-in Scala REPL will shrink over time.
As the built-in Scala 3 REPL improves, we can gradually shrink the Ammonite codebase

👍 — and for anyone out there looking for OSS stuff to do, porting successful experiments from Ammonite to the stock REPL is highly encouraged. (Even the Scala 2 REPL can still be improved, if someone is motivated to do so.)

[urbanchr]

What I like of amm-repl very much is that you can specify more than one @main and then select at the command line which to run. I believe this feature is not supported by Scala nor Scala-cli.

[SethTisue]

@urbanchr Scala-CLI has that with -M:

% cat S.scala                  
@main def main1() = println("one")
@main def main2() = println("two")
% scala-cli S.scala -M main1 --quiet
one
% scala-cli S.scala -M main2 --quiet
two
%