Make "final product" less(/not) reliant on Boot

[daveyarwood]

I added boot.properties before Clojure 1.7.0 became the default Clojure version in Boot 2.2.0, as a way of enforcing that Clojure 1.7.0 be used when running Alda boot tasks, which used to be how you would run alda's play, parse and repl tasks, pre-alda.cli (i.e. instead of using the alda executable, you would cd into the Alda project directory and run boot play --file /some/file.alda, or boot alda-repl, etc.)

Since then, the latest versions of Boot (since 2.2.0) have made Clojure 1.7.0 the default BOOT_CLOJURE_VERSION, and the usage instructions for Alda has moved from cloning the repo and running Boot tasks to copying a provided executable Boot script (bin/alda) to your $PATH and using the alda command as an entrypoint into the tasks defined in the alda.cli namespace.

Bearing this in mind, does it make sense for us to still have a boot.properties file? With Boot steadily evolving and improving, I'm not sure I like the idea of pinning the Alda project to a specific version of Boot, when really all we want to enforce is Clojure 1.7.0 as a dependency.

[crisptrutski]

In my opinion repeatability is worth it.

Running boot -V > boot.properties every so often isn't too onerous, and if you're worried about forgetting and being left behind you could add a CI test comparing asserting that boot.properties is up-to-date.

[daveyarwood]

I guess my question is more about whether we still need boot.properties, since we have moved away from using Boot tasks as the way to run Alda in the wild. For the "end product" version of Alda (i.e. the executable script), it shouldn't matter what version of Boot you have or what you might have BOOT_CLOJURE_VERSION set to -- only that the JVM (I guess the class loader?) in which Alda is running has Clojure 1.7.0 available to it.

[crisptrutski]

Perhaps it's worth exploring a bin/alda that doesn't even require boot on the client (it just pulls the JAR and runs it). Flipside, since 2.0.0-rc13 boot has supported launching pods with specific version of Clojure (so move alda task into one)

From the perspective of developers collaborating, and being able to cut long term release branches one day, I think the properties file should stay in the repo, even if it plays no role in the distributed artifact.

[daveyarwood]

OK -- I'm convinced that boot.properties can stay -- it makes sense from a development perspective :)

I think having a bin/alda that doesn't require Boot would be a major win! It would definitely simplify things for Alda end users. We could AOT-compile the project with alda.cli/-main as the entrypoint, and then rewrite bin/alda as maybe a bash script(?) that would handle getting the latest version of Alda (either automatically, or on command when you run alda update) and passing the arguments to its main method. (This is probably a subject for a separate issue)

[daveyarwood]

I mulled this over for a while tonight, and another possibility occurred to me:

I think dependency management via Maven would require Boot still, so maybe it would be better if we created a self-contained binary executable wrapping an uberjar -- that way, users would only need to have Java installed. We could generate a new executable each time we push a new release, and make it available on GitHub.

I was able to create a standalone uberjar of Alda without AOT using the experimental boot-ubermain task, by running the following command:

boot -d boot/base:2.2.0 -d boot/core:2.2.0 -d org.clojure/clojure:1.7.0 -d alda:LATEST -d adzerk/boot-ubermain:1.0.0-SNAPSHOT ubermain -m alda.cli/-main

java -jar target/project.jar

One caveat is that the ubermain task runs things in a limited Boot environment where you can't handle dependencies dynamically, which means we would have to include the 125 MB FluidR3 soundfont as an unavoidable dependency (whereas we are currently only downloading it once the user runs alda play or alda repl without the --stock flag). I think this would be OK, though -- I feel like most people would prefer to use FluidR3 over the stock JVM soundfont -- and this would also nicely solve issue #34.

It would be nice to be able to generate a binary executable (preferably via a Boot task) from the uberjar, à la lein-bin. We could also make things right in Windows-land by using Launch4j to create a .exe file from the uberjar (this is how Boot provides a boot.exe for Windows users, in fact).

An open question is what the update process would be like if we went with the self-contained executable idea. I guess we could implement an alda update task which would download the latest Alda executable from GitHub and install it to the correct place?

[daveyarwood]

We should take care not to store the uber-jarred binary file in this git repo -- with all the dependencies included (especially FluidR3, which is 125 MB), the size of the git repo would quickly grow out of hand.

This might help: https://github.com/jedbrown/git-fat

[jeluard]

Another relevant point here is time to boot. On my machine (2011 MacBook Air) time alda -v gives:

alda v0.7.0
alda -v  39,81s user 1,93s system 165% cpu 25,182 total

which is really prohibitive.

My experience is that a real aoted uber jar helps a lot and is the best you can have. Also the smaller the final Jar is, the better. Can FluidR3 be loaded externally somehow?

[daveyarwood]

@jeluard Yikes! I totally agree, the startup time should really not be that high.

On my mid-2014 MacBook Pro, it's considerably shorter, but still not great:

alda v0.7.0
        7.81 real         9.62 user         0.80 sys

A command-line utility like Alda really should not take that long to load. Boot-clj start-up time is a factor, as is JVM load-up time and loading the Clojure run-time. Removing the Boot requirement for the end user will help, not to mention AOT-compiling as much as we can and running Alda from an uberjar.

Unfortunately, packaging everything into a jar makes it hard to load dependencies dynamically like that. Granted, we could always go outside of the Maven dependency infrastructure and just have the Alda program manually download the jar from Maven, unzip it and stick the soundfont in ~/.soundfont or something.

My initial thought was, it wouldn't be too annoying to just include FluidR3 in the jar. But then, the problem is that every time Alda is updated (which is quite frequently), you would need to download a new 100+ MB binary file, which would be obnoxious.

[jeluard]

I did some first tests and I now have a final uberjar of 7M that starts in 8s (3 times faster). Loading FluidR3 bumps load time to 21s. Loading FluidR3 directly (i.e. not wrapped in a Jar) removes a couple seconds.

Much of the remaining code is clojure. timbre and its dependencies might be removed and replaced by regular java logging (I'm not of fan of dependencies in general ;))

Some random questions:

• defclifn usage from alda.cli should be replaced by something, maybe clojure.tools/cli or we must package boot
• when is alda.core/-main used?
• is the reply dependency needed?
• unless I am missing something, only overtone/at-at is needed not the whole overtone ?

About FluidR3 an option could be to have bin/alda just augment the classpath when needed and not bundle it. This does not add startup latency and allows to upgrade FluidR3 only when needed.

This is assuming the update process is done externally, maybe taking inspiration on what leiningen does?

[jeluard]

Added bonus when removing dependencies: it will be easier to port to ClojureScript ;)

[daveyarwood]

This is awesome -- thanks for weighing in on this! I think we're definitely headed in the right direction here.

Loading FluidR3 bumps load time to 21s. Loading FluidR3 directly (i.e. not wrapped in a Jar) removes a couple seconds.

That's surprising!

About FluidR3 an option could be to have bin/alda just augment the classpath when needed and not bundle it. This does not add startup latency and allows to upgrade FluidR3 only when needed.

This is assuming the update process is done externally, maybe taking inspiration on what leiningen does?

Admittedly I don't know terribly much about Java and the JVM, aside from what I've learned through working with Clojure. I haven't dealt too much with things like classpaths. Would this be like adding an extra argument to the call to java -jar in the binary executable? Or is there a more dynamic way to do it in a Clojure program (without being able to use Boot's merge-env!)?

Right now Alda's updates are automatic, via the Boot script using the LATEST version of Alda deployed to clojars. But I think it would be better to have a manual update process by running alda update, which would replace your alda executable with the latest release. Is that what you mean by an external update process?

By the way, there is another, more permanent way to load FluidR3 into the JVM, which I wrote about here. I'm not sure if this is much better than loading FluidR3 from the sf2 file during the execution of Alda, but if it is, then we could consider automating the process of installing FluidR3 as the default JVM soundfont on the user's system, and then people could opt into it when they first get Alda by running alda install fluid-r3, or something. That would even make it easier for users to use custom soundfonts, if there is one they prefer over FluidR3.

Much of the remaining code is clojure. timbre and its dependencies might be removed and replaced by regular java logging (I'm not of fan of dependencies in general ;))

I'm on-board with cutting down on our dependencies, especially in light of all the performance issues people are having. I say the faster we can make Alda, the better!

• defclifn usage from alda.cli should be replaced by something, maybe clojure.tools/cli or we must package boot

Boot's defclifn is nice, but I'm not entirely opposed to switching to clojure.tools/cli.

• when is alda.core/-main used?

The alda.core namespace is not currently in use... it's an artifact of when I first started the project. There are some ideas there that could potentially be merged into the CLI functions in alda.cli. The real entrypoint to Alda is alda.cli/-main. We should probably just remove alda.core and document the ideas somewhere, maybe as an Issue for discussion.

• is the reply dependency needed?

Yes - we use it for Alda's REPL.

• unless I am missing something, only overtone/at-at is needed not the whole overtone ?

You're right. Initially I wanted to use Overtone for generating sound (building synths, using them to play scores), but I've been rethinking that idea due to its dependency on Supercollider. I think I'd rather use something Java-powered, like Minim. At any rate, all we really need from Overtone at this point is overtone/at-at for event scheduling.

[refi64]

Have you guys ever considered using another, non-JVM-dependent Lisp variant, like Hy?

[daveyarwood]

@kirbyfan64 Sorry -- Hy is cool, but I'm committed to Clojure :)

[jeluard]

Great! I'll work on a patch to reduce the startup time then.

Is that what you mean by an external update process?

Yes, I think it makes more sense to have a manual command to upgrade. It also makes the process more package manager friendly, if you ever feel like.

.. then we could consider automating the process of installing FluidR3 as the default JVM soundfont

I didn't know that soundfont where directly supported at the JVM level. I will make some more tests to see how it impacts performance. From user perspective it sounds simpler to have a dedicated command for this too. From dev perspective also as it removes the soundfont management from the main path.

.. reply .. Yes - we use it for Alda's REPL.

Ah ok. Is it magically used because it's on the classpath?

[daveyarwood]

Oh, whoops! Turns out we're not using reply! I think I was playing around with it at first, but then switched to using a JLine ConsoleReader via Java inter-op. We can totally scrap that dependency.

A PR would be awesome! Let me know if there's anything I can help with.

[daveyarwood]

Things are getting interesting in the land of Boot -- the usual entrypoint, boot.sh, which previously needed to be installed and managed by the user, is now just a thin wrapper script which fetches and runs Boot. In other words, Boot can now build and run itself.

This is awesome because it means we can package boot.sh with Alda, and our users will no longer need to have Boot installed (they'll just need Java, which most computers have already).

A major benefit of this approach is that we won't have to re-think the parts of Alda that currently rely on Boot (namely the bin/alda script) -- we can just package Boot in the Alda binary and have it fetch and run Boot to run the script. 💣 💥

[jeluard]

Sorry didn't manage to find time to push things further. This boot improvement sounds an interresting option.
Startup time wise it looks like the biggest change came from removing useless dependencies and not programmatically loading instruments.

[daveyarwood]

That's OK, I've been busy too :)

From what I understand from talking to @micha, I don't think Boot adds too much overhead to the startup time, even with starting up pod pools, etc. Startup time will also become practically moot once we implement #49.

[micha]

@daveyarwood packaging boot.sh in Alda as say bin/alda and bin/alda.exe might be a good way to go. You can also fix the BOOT_VERSION and BOOT_CLOJURE_VERSION by setting Java system properties in the script header and Launch4j configuration XML.

[daveyarwood]

@micha That's a good idea. I've been working a little more on boot-jar2bin lately -- I was thinking of adding task options to set Java system properties in the script header.

[daveyarwood]

Currently planning 1.0.0-rc1, featuring a totally revised build pipeline. The entire project will be packaged into an uberjar and (although it's built via Boot) will not include Boot dependencies in the project itself. We will use boot-jar2bin to export Unix and Windows executables.