PSA: Moving away from JIT

[AlexDenisov]

Just wanted to share the plans for the nearest future: after lots of discussions and considerations, we decided to move away from JIT to a more classical model.
Just a few words on how everything works right now: Mull extract LLVM Bitcode from an executable, mutates some LLVM instructions, compiles the bitcode into machine code, and feeds the machine code to a JIT engine for further execution.
This approach works pretty well, sometimes. But often it just fails for some odd reasons.
During the years JIT was the biggest driving force behind the development, but it was also the major source of frustration:

• simple cases work very well, while the real-world applications don't
• LLVM's JIT engine is the biggest moving target: we had to put a lot of effort to maintain the same API across different LLVM versions
• constraints imposed by the JIT engine also limit Mull's applicability: we cannot support as many OSes and language runtimes as we want (think of Windows and Objective-C)
• debugging is nearly impossible: if something crashes inside of JITted code there is no way to attach a debugger or obtain a reasonable stack trace.
• etc etc etc...

I'm only slightly exaggerating: the issues I mentioned can be solved, but it becomes a question of trade-offs. In the end, we are not building a JIT engine, we are building a tool for mutation testing.

Short summary of the pros and cons of moving away from JIT.
What we lose:

• explicit test frameworks support: Mull would still support any test framework, but the granularity and "accuracy" will be lower
• performance: it needs to be measured first, but I expect slow-down of mutated code under certain conditions
• custom code coverage: current approach allows us to mutate less code and therefore decrease execution time

What we gain:

• multithreaded app support
• support for integration tests: I can certainly relate to what @ligurio asked for in Support tests written on interpreted languages #778
• better support for various platforms: think of Windows, Linux distros besides Ubuntu, maybe we can get FreeBSD back?
• better debugging: you get a normal, standalone executable
• potential to support more languages besides C and C++
• 32bit apps support: there is still an area where 32bit apps are in use
• (my favorite part) remove lots of half-baked semi-working code I wrote
• (another favorite of mine) we can close lots of JIT related issues (and perhaps open many new ones 😄)

---

The intention so far is not to rewrite Mull from scratch, but gradually migrate to a new model while preserving APIs where possible.

There is no ETA, and there is no clear and detailed plan at the moment. I do have a rough idea of what needs to be done, but it is too rough to be shared in a productive way.

Do not hesitate to drop any questions or thoughts below.

[AlexDenisov]

The JIT was dropped here #808, now Mull emits a standalone executable and runs it as a normal process.
This allowed us to simplify implementation (#811, #813, #814).

What's left is to update documentation/tutorials, and we are ready to release the new version.

[AlexDenisov]

With the next release being on its way #825 I consider migration from JIT to be complete.