Proposal: code refactor towards cleaner architecture

[KoltesDigital]

Hi team. Quick intro: I'm in an art collective with @Bubobubobubobubo but I'm more on the technical side, working as a software architect and using Rust. He's invited me to have a look at the code. I'll talk in English as new contributors may not speak French.

So I had a glimpse to the repo, and in accordance with @Bubobubobubobubo I've started a branch to make some structural changes. The goal is to keep the functional code, and the compiled binary should be roughly the same. So changes will mainly deal with putting the code in the right structs and setting their relationships. I'm particularly interested in making the live-coding part testable, so that the tool behavior is deterministic. And tests shall not involve actual delays :)

However I have to say that the code felt more like optimized C++, which is normal from audio software engineers, but kind of annoy me. I believe the code complexity hinders its maintenance and evolution, and this is particularly sad at the beginning of a project, where design is not yet final, so agility is mandatory to make it grow. Onboarding new contributors is a plus, and they have to quickly understand the code. Besides, one goal of this project is for @Bubobubobubobubo to learn Rust, but I don't feel the code is in a proper state to do so. There are lots of Arcs and string clones, which may be the quickest adaptation from his Python background. But this obviously counteracts any micro-optimizations, e.g. the step's struct of arrays. In addition, OSes are not real-time and the scheduler could be off by a few milliseconds, so I believe making the code harder to work on in order to gain some microseconds is not worth it. In particular for the struct of arrays, they are mostly accessed together, so I think there's no gain. I think the data fit in a few pages at most anyway.

In short, I'm not against optimizations, but I prefer to have a clean design, make some benchmarks, then optimize, and see if the benchmark improvements, if any, are worth it. And that's what I'm proposing. I may be able to present my work in a few days/weeks. It shall not be a take-it-all-or-leave-it-all, it shall be open to discussion. I'm fine if you decide to only take a part of it, or even none at all.

Last night I started the refactor, and had a few questions:

1. If I understand correctly, a script could be replaced while it's being executed, but the ScriptExecution's instruction index is not reset. Is it by design? I haven't run any test to confirm the hypothesis.
2. A variable can hold a Program value, which only seems useful for one case: the ASM instruction to change it. Apart from that, it messes with all math operations. Do you really think you'll use this feature? If yes, what about having a library of scripts in the World, so that the variable would only contain its name as a string, and not the whole program?
3. There are instructions to push / pop value from a stack. It looks redundant with the Instance variable storage. Moreover, the language is not at all stack-based, as instructions have their own parameters. So, is this stack actually useful?

[loig]

Hey. Thanks for this message. I agree on the idea to make things as simple as possible in order to be able to change the design quickly and evaluate the impact of design changes. I am looking forward to your refactor. If it can help me better understand Rust it will be a nice side effect (this is also something new for me).

Regarding your questions:

1. A script execution is supposed to be the execution of the script as it was at the beginning of the execution. So, if the script latter changes, this does not affect executions that are currently running. The idea is that the "normal" use of the tool is that a script will roughly have a duration corresponding to the duration of the step to which it corresponds. However, some user may define a script for which the execution is much longer (or even "infinite"). In this case, we chose to keep the execution running as it is, event if the script changes.
2. There is a CallFunction instruction that allows to call a function. What is important to realize about this internal language is that it is not supposed to be used for Live Coding but for compiling scripts in this language. So, the philosophy behind it is to make a lot of things available for compiler design to be possible with various approaches. Having these functions allows for simpler function calls (one does not need to handle the position of the different functions in a program, which would make jump computations a bit more difficult).
3. These push/pop are useful for functions/procedure calls. Also, they simplify compilation as one does not have to bother not messing with variable names. So, this is the same philosophy than for functions: allow for different ways of doing the same thing in order to make compiler building easier.

To summarize: the internal language looks like some assembly language, but we use the fact that we are not actually at instruction-set level in order to help doing complex things easily. For example, the stack for return addresses is separated from the stack for storing other values (the one with push/pop).

If you want to talk about that a bit more in details, I would be happy to call you in the next days.

[KoltesDigital]

Great! If you happen to be around Paris, we could meet at some local hackerspace too.

1. Agreed. I thought that would be the purpose on script (arc). It actually simplifies things that the program is cloned when the exec is spawned, let's keep that.
2. Agreed, although if your goal is to make a new agnostic language, several ones should already exist, and you'd better off using them, especially if they have a JIT compiler. Maybe Lua could do the job.
3. Agreed. One nice thing in actual CPUs is that the stack is both for calls and values, so that leave "pops out" all values pushed by the function. Which is not the case here. Other possibility: Lua's call function takes the number of arguments in and the number of values out, so that the stack after the call is in a known state.

What do you think about a new Variable variant to combine stack and usual operations, like ControlASM::Add(Variable::Stack(2.into()), Variable::Constant(1.into()), Variable::Stack(0.into())) for taking the 2nd topmost value from the stack, adding 1, and pushing the result onto the stack. Other possibility: Ibniz has stack manipulation operations, like rotating/swapping values, because math operations always operate with the top values.

[loig]

2. The goal is to have a language that is easy to execute (think of Java Bytecode for example) and easy to compile into. Using an already existing language would mean that we would have to write a scheduler that can understand this language and execute it step by step. This notion of step by step execution is central in Bubocore, this is what allows us to avoid concurrency between the different scripts that execute at the same time (or rather, cleanly deal with it). Also, I think that compiling to something like LUA may not be easy for all languages that someone would like to define. Finally, by designing our own intern language we have the possibility to handle errors caused by wrong use of things by ourselves easily (for example, we can decide what happens when an illegal operation is performed).
3. In processors (at least intel processors), things are not popped automatically at the end of a function. When compiling for some architecture you have to handle the stack by yourself: move the stack pointer and the base pointer to create the frame of your function at the beginning of this function, move them back at the end so that the return address is at the top of the stack. CALL only pushes the return address and jumps to the start of the function. RET only pops the top of the stack and jump to the corresponding address. That is why I say that it is easier to split the stack in two. Also, I think one of the main reason for having values in the stack is that the number of registers is limited (usually, in compiled programs, a functions with not many arguments and return values will use registers).

I think it is a good idea to have the possibility to directly use values from the top of the stack as "variables" and to directly put results of operations on top of the stack, it would make shorter programs and thus more efficient execution. I would be in favour of taking values one by one, though. In particular because, if for an addition the order of the arguments is not fundamental, for a subtraction it is (and so, one could easily have difficult to find bugs when popping several values from the stack directly in an arithmetic operation)

Here is a document I wrote to try to explain the model of execution of Bubocore and the instructions we have. This is a work in progress but it may give you a better idea of what we want to achieve:
define-languages.pdf

In am not in Paris, and I will not be there in the next few weeks. I think the best solution would be to have a call. Are you available in the next days?