This document describes the software architecture of ReaLearn. It’s a work in progress.
Technically, ReaLearn is a third-party native VST plug-in for REAPER which makes heavy use of the REAPER extension API.
… Woah! That is a lot to take in! Let’s see what this means in detail:
-
First and foremost, ReaLearn is a plug-in for REAPER. That means ReaLearn is a software module which is loaded by REAPER dynamically (at runtime). It comes in the form of a DLL on Windows, a DYLIB on macOS and a SO on Linux. Communication goes both ways: ReaLearn calls REAPER functions and REAPER calls ReaLearn functions ("callbacks").
-
It makes heavy use of the REAPER extension API. That means ReaLearn uses functions and data structures of REAPER which are specific to REAPER, so they are not part of a DAW-agnostic plug-in standard. In theory, other DAWs could implement this API as well, but in practice only REAPER itself does it. The reason is that this API is huge and that many parts of it only make sense for REAPER. As a consequence, ReaLearn runs in REAPER only!
-
At the same time, it is a VST plug-in. That means …
-
… it’s not loaded immediately at REAPER startup time. It gets loaded as soon as the user adds the first ReaLearn plug-in instance (in the form of a REAPER FX) or loads a project or FX chain containing a ReaLearn instance.
-
… once loaded, there can be arbitrary many instances of ReaLearn (a big difference to REAPER extensions, which can "exist" only once). Adding more instances is rather cheap because the dynamic library is already loaded at that time.
-
… plug-in data is saved per instance (as a VST chunk). As a consequence, plug-in data is typically saved along with a specific project (but it doesn’t have to because you can put ReaLearn on the monitoring FX chain as well).
-
… it can receive MIDI and audio data from the FX input pins (using functions and data structures defined by the VST plug-in standard).
-
… it can send MIDI and audio data to the FX output pins (using functions and data structures defined by the VST plug-in standard).
-
-
It’s a native plug-in. That means ReaLearn is not a script! It’s full-blown native machine code that operates on eye level with REAPER. Once ReaLearn is loaded, it essentially becomes a part of REAPER itself. It runs in the same process, can open its own threads and is not sandboxed or restrained by a virtual machine (unlike Lua/EEL-based ReaScript or JSFX). That’s great because it means ReaLearn is a "first-class citizen" and doesn’t have any inherent disadvantage in terms of speed and possibilities. Also, it’s written in a language that can leverage all of the power and performance potential that comes with running on "bare metal": Rust. Rust is a non-garbage-collected system programming language, just like C or C++ but more modern.
-
It’s a third-party product. That mean’s it’s not made by Cockos, the developer of REAPER. It’s made by Helgoboss and must be installed separately.
ReaLearn is built in a modular fashion. The following diagram shows ReaLearn’s most important modules (excluding 3rd-party modules):
-
main: The main module of ReaLearn which contains most of its code. We will learn more about it in the following sections.
-
api: This contains the data structures for ReaLearn presets. Its main use case is ReaLearn Script, a way to build mappings with the Lua scripting language.
-
swell-ui: A tiny custom-written GUI framework based on the Win32 API (Windows) and Cockos SWELL (macOS, Linux) respectively. SWELL makes it possible to write the GUI code only once, using a subset of the Windows-specific Win32 API, but making it work on macOS and Linux as well. Basically by translating the Win32 API calls to OS-native GUI framework calls (Cocoa on macOS, GTK on Linux).
-
reaper-rs: Rust bindings to the REAPER API (which itself is based on C and partially C++).
-
helgoboss-learn: A library which contains reusable and DAW-agnostic code related to MIDI/OSC-learn functionality. Some of ReaLearn’s basic notions such as Source, Glue (still called Mode in most parts of the codebase) and Target are defined in this DAW-neutral module.
-
helgoboss-midi: A general-purpose and carefully designed library for dealing with MIDI messages according to the MIDI 1.0 specification.
The main module of ReaLearn is roughly built around an architectural pattern sometimes called Onion Architecture. This means it’s divided into multiple "onion" layers:
These layers follow a simple but strict rule:
Outer layers use its own code and code of inner layers, but inner layers are not allowed to use code of outer layers!
This rule prevents "Spaghetti code" between the different layers and makes sure there’s a clean separation between different responsibilities.
|
Important
|
This rule doesn’t restrict control and data flow! Both control and data can still flow in both directions. In practice they also do because e.g. data from the processing layer needs to be "sent up" to the user interface in order to be displayed! The rule is concerned with the visibility of code symbols only. |
The contents and responsibilities of each layer are described in the following.
-
Contains very generic and reusable utility code that is not specific to ReaLearn and at the same time not substantial enough to put it into a dedicated library module.
-
Also, this layer can be considered as the layer that contains the Rust standard library and other Rust crates that provide utility code (although those are obviously not part of ReaLearn’s own codebase).
-
In addition, it makes very much sense to think of the base layer as the layer that contains REAPER itself. ReaLearn is built around REAPER, it’s not designed to be usable without it.
-
As a direct consequence, all layers within ReaLearn are allowed to use the REAPER API!
-
This doesn’t mean that ReaLearn couldn’t be ported to other DAWs. It could, provided the other DAW is substantially extensible via native modules. On ReaLearn’s side, some effort in this direction has already been done: As mentioned before, the module helgoboss-learn is designed to contain those parts of ReaLearn’s logic that are DAW-agnostic.
-
-
This layer contains the essence of ReaLearn: Its processing logic. This includes the complete control and feedback logic.
-
If you would take away ReaLearn’s graphical user interface, its projection feature, its plug-in nature, its capability to memorize its settings (= persistence) … in short, all the stuff that is more "facade" than "central", then what’s left is the processing layer. The processing layer alone would still be capable of doing ReaLearn’s main job: Routing incoming MIDI or OSC messages through the mapping list and controlling the targets accordingly as well as handling feedback.
-
Because the processing layer is very independent and doesn’t dictate things like user interface and persistence, It would be quite easy to factor it out into a separate module and use it in other ways, e.g. in order to build a totally different user interface on top of it!
-
All the data structures in this layer are custom-tailored and optimized with one primary goal in mind: Performance. ReaLearn should do its main job very fast and efficiently!
-
This layer contains everything related to managing ReaLearn’s objects: Mappings, groups, parameters and all that stuff.
-
All the data structures in this layer (usually called models) are tailored to this purpose. If you think that there’s a lot of duplication between this layer and the processing layer, look twice. Yes, the data structures look similar at times, but often they are completely different. That’s because they are designed for different purposes. This strict separation of concerns ensures that no compromises need to be made between performance (processing layer) and managing/GUI (management/infrastructure layers).
-
Even though this layer still doesn’t dictate a particular user interface, it is user-interface-aware and provides functions and data structures that are typically used by user interfaces. It also allows user interfaces to register hooks in order to be notified whenever the state of ReaLearn’s objects change. The management layer is built with a reactive GUI in mind which reflects all changes immediately.
-
This layer is basically responsible for connecting ReaLearn to the outside world: The user (user interface), the storage (data), the DAW (plug-in), the scripting feature (api) and the Projection server (server).
-
Contains the VST plug-in implementation of ReaLearn.
-
This is the main entry point, the place where ReaLearn’s global initialization happens as well as the initialization per instance.
-
Contains the implementation of ReaLearn’s main graphical user interface.
-
It’s based on the swell-ui module. That means it uses platform-native user interface widgets - which gives ReaLearn the somewhat old-school but extremely professional look ;)
-
Contains data structures for the serialization/deserialization of all ReaLearn objects (mappings, groups, etc.).
-
The data structures in this layer are similar to the corresponding data structures in the management layer but they serve a quite different purpose: Serialization and deserialization of ReaLearn’s state. This is necessary for persistence and features such as copy&paste.
-
One could wonder about the code duplication here, but again: The data structures in this layer serve different purposes than the ones in the management layer. Serialization/deserialization for persistence purposes absolutely needs to be concerned with backward compatibility, which makes these data structures very hard to change. Keeping things separate ensures that the management data structures can develop freely, without being constrained by backward compatibility considerations. Again: No compromises.
-
Contains the data structures that make up ReaLearn Script.
-
The focus of these data structures is to provide an expressive API with a wording that’s straight to the point.
-
These data structures are complete in that they can represent and express all valid ReaLearn object states, much like the structures in data.
-
They were written much later (end of 2021) than the data structures in data (2016) and therefore reflect ReaLearn’s current wording and structure much better.
-
In future, the API data structures might actually be used for persistence and copy&paste as well, eventually replacing data.
-
ReaLearn is made up by a plethora of data structures that resemble components, which can be considered as ReaLearn’s main pillars.
-
Some of these components are part of each instance, others exist globally only once.
ReaLearn supports scripting in various places:
-
MIDI scripts: Lua or EEL (Lua support more powerful)
-
Feedback scripts (for MIDI or OSC): Lua only
-
Control transformations: EEL only (because must be real-time capable)
-
Import/export: JSON or Lua (Lua obviously more powerful)
-
Dynamic conditional activation: Expression language or EEL
-
Target-based conditional activation: Expression language
-
Dynamic expressions: Expression language
As can be seen, Lua is our main scripting language for stuff that doesn’t need to run in real-time. Here’s why:
-
Lua is easily embeddable (this is a must, and it rules out most other mainstream languages)
-
Lua is popular and widely-used (important, rules out exotic or new languages such as Rhai or Gluon)
-
Lua has nice features that make it very suitable for building data structures and even DSLs (our main use case):
-
Operator overloading (rules out JavaScript/TypeScript)
-
Ability to skip parentheses when passing function argument (rules out JavaScript/TypeScript)
-
Really usable multi-line strings
-
-
Doesn’t increase the size of the binary very much (nice to have)
-
It can be quite fast (not important for import/export but very helpful when it comes to e.g. MIDI or feedback scripts)
-
REAPER power users already know Lua because it’s also REAPER’s primary scripting language (not strictly necessary but certainly helps because REAPER users are usually not developers, so switching languages might be a big effort for them)
Some languages that were considered but ruled out. Here they are, along with the most important reasons why there were ruled out:
-
JavaScript/TypeScript: no operator overloading, not possible to skip parentheses when passing function argument, also much harder to embed
-
In general, I was very much in favor of JavaScript/TypeScript because it’s so widespread and the tooling is perfect. But turns out Lua is actually better suited for our main use case of creating large data structures for import/export. Surprise!
-
Assembling mappings is an example where operator overloading is really nice:
name("Scroll up") + shift_or_sustain + button("col1/stop") + feedback_disabled() + turbo() + scroll_vertically(-1) -
Also, embedding it is very hard. Yes, there is TypeScriptToLua, but the TypeScriptToLua compiler is also written in JavaScript. We need a solution that runs 100% in ReaLearn without external pre-processing.
-
-
Python: too heavy-weight, also harder to embed and slower
-
Wren: doesn’t seem to be active anymore, maybe a bit too exotic (looks exciting though)
-
Gluon: too exotic
-
Dyon: too exotic
-
Rhai: too exotic
-
Mun: hard/impossible to embed, not mature enough
-
Rust: hard/impossible to embed, not easy enough
-
WASM: just embedding a WASM runtime wouldn’t help because it’s just for running WASM bytecode but not producing the bytecode, which would require a language-to-WASM compiler, which again brings up the question of which scripting language
-
AssemblyScript: interesting because TypeScript-like and operator overloading, but ultimately looks too hard to embed because it needs a WASM runtime and I need to make it itself run within that runtime, also things like operator overloading don’t have IDE support
-
Haxe: direct embedding doesn’t seem to be possible, so would only be interesting for transpiling outside Helgobox
-
Despite its many advantages, there are also a few really annoying things about Lua:
-
No static typing
-
String concatenation is ugly
-
No distinction between maps an arrays (just tables)
-
Really spartan standard library
-
No strong conventions
There are some interesting projects out there that seek to address Lua’s pain points.
Teal is a statically typed Lua dialect transpiled to Lua.
Pros:
-
Compiles to Lua
Cons:
-
Language server can’t auto-complete fields in table literals. Not at all.
-
The type system is less elaborate than that of Luau. Unions are not powerful enough, e.g. two tables can’t be part of a union, which also excludes tagged unions. We have a lot of tagged unions.
-
Types are required, no structural typing
-
Needs an additional compilation step to be loadable into the VM
LuaLS is a Lua language server with type checking capabilities.
Pros:
-
Types are optional, structural typing
-
Can auto-complete not-yet typed fields in table literals
-
Works without needing another language
-
Seems to allow dots in type names. Which would be nice for something like
Target.TrackVolumeinstead ofTarget_TrackVolume.
Cons:
-
No type deduction based on tagged union discriminator
-
Typing in comments only, feels like patchwork and is not really intuitive
Luau is a Lua fork with optional static typing.
Pros:
-
Types are optional, structural typing
-
Type refinement based on tagged union discriminator
-
Looks like the most elaborate type system of all candidates
-
Promise of backward compatible changes
-
Really nice usability improvements over normal Lua, such as string interpolation and easier iterating
Cons:
-
Type system still has its flaws, especially when it comes to intersections. On the other hand, the other candidates don’t even try something as advanced as intersections.
-
Not sure about using a fork
-
Typing too structural? It doesn’t spit out the type name anymore after assignment.