Unicorn 2 - looking for sponsors

[aquynh]

Since its born in 2015, Unicorn has been inspiring so many cool research & tools, that we believe it deserves serious upgrade. Based on Qemu 2.1.2, there is a lot to catch up with the latest version of Qemu (Qemu is at v4.2 at the moment). Unfortunately, the code base of Qemu changes so much since 2015, that we believe it is faster to rewrite Unicorn from scratch.

For this reason, next major update of Unicorn (lets call it Unicorn 2) requires huge effort & lots of time. So we call for the support from Unicorn users!

If you are using Unicorn for your work, and wish to sponsor the development of Unicorn 2, please contact us: http://www.unicorn-engine.org/contact/

[korniltsev]

are you going to keep in sync with upsream after rewrite?

[aquynh]

That is not easy to keep syncing with the upstream. This means we must trade off with some sacrifices in features. Keep posting here if you have any ideas.

[Waterman178]

Support for the next version!

[XVilka]

Related #1190
Regarding the keeping in sync - it might be more useful to create a series of semantic patches (with spatch tool) to generate the Unicorn on the fly. There is a mature tool to do this called Coccinelle, see their documentation for more details. There is a good introductory article on how to create semantic patches. Coccinelle used for the Linux kernel sources refactoring, so it's stable and featureful enough.
See the example of this semantic patch:

@@
expression lock, flags;
expression urb;
@@

 spin_lock_irqsave(lock, flags);
 <...
- usb_submit_urb(urb)
+ usb_submit_urb(urb, GFP_ATOMIC)
 ...>
 spin_unlock_irqrestore(lock, flags);

@@
expression urb;
@@

- usb_submit_urb(urb)
+ usb_submit_urb(urb, GFP_KERNEL)

[minexew]

@ XVilka Due to how Unicorn is created (ripping out the heart of QEMU and fixing it up to make it nicely embeddable), it unfortunately seems that any new branch of Unicorn is doomed to ultimately fall out of sync with QEMU upstream again.

[raphaelthegreat]

Is there any update on Unicorn 2?

[kzsnow]

My two cents: I did a bit of "Unicorn 2" prototyping with Qemu 4.2. The two biggests "lifts" I encountered, in terms of source code modifications, were compartmentalizing qemu globals (e.g. when multiple emu instances are used), and dealing with the memory implementation.

Like others, I'm cautiously optimistic there is a way to minimize Qemu source code mods to make it easier to keep up with development there, and greatly lessen the burden for updates.

A quick hack to get around the issue of qemu globals without source code mods is to load unique instances of the shared library for each "emu" instance. Of course the trade-off is increased memory consumption (however big the qemu lib is, for each instance). I've used Unicorn quite a bit, and in all my use cases (personally) this would be an acceptable trade-off.

Regarding memory, there are two options: softmmu or using the host system's mmu. To use the host system's mmu, I didn't need to modify qemu source code at all (basically just linux-user mode without syscall passthrough). Besides that, performance is 5x better than Unicorn/softmmu in some rudimentary tests. It would be really useful to be able to run "Unicorn 2" in this mode for the performance boost, when needed (i.e. fuzzing, etc.).

There are massive downsides to using the host mmu though: emulated code can no longer be securely sandboxed (not that I can tell, at least), and it restricts where regions can be mapped in the emulation (compatibility). Hence, ultimately we'd want the ability to use softmmu. This is where I ran into the biggest issue. The qemu "system vs. host" and "softmmu vs. hostmmu" #defines are not separated well, and source code mods started building up. At this point, the time investment outweighed the benefit.

So I'll put this out there for discussion. The way forward might be to start with pull requests on Qemu towards the goal of adding a "softmmu vs. hostmmu" option for the linux-user build of qemu. I vaguely recall that someone was attempting this years ago and Qemu devs seemed receptive, but it didn't materialize. If we had this option in Qemu, I'd wager that Unicorn-like functionality would be much easier to build out.

[Announcement]

I'm not a typical contributor and not too involved in this project but if QEMU is really that active, why not ask them to publish all the apis and such that you need so we can basically use an unmodified qemu and just plug into it like a dependency? That probably sounds stupid to those who have been working on a unicorn but, what kind of modifications are being made?

[wtdcode]

I'm not a typical contributor and not too involved in this project but if QEMU is really that active, why not ask them to publish all the apis and such that you need so we can basically use an unmodified qemu and just plug into it like a dependency? That probably sounds stupid to those who have been working on a unicorn but, what kind of modifications are being made?

Modifications include (not all of them!):

• Do instrumentation in the translation stage and memory operations to support various hooks.
• Various checks (and even modify tcg IR) to make sure the execution can be stopped anywhere and resumed cleanly.
• Modify some qemu helper functions to handle interrupts and syscalls.

Qemu code is clean indeed, but not all these modifications can be easily done without some really dirty hack.

[aquynh]

We are redesigning the logo of Unicorn for Unicorn2, with the target of having a more modern logo, that can be used better on Tshirt, mug, etc.

If anybody wants to sponsor the new logo design (so your support will be there forever in the project history ;-), please contact.

[aquynh]

towards Unicorn2, we redesigned the logo - for this major version.

https://twitter.com/unicorn_engine/status/1400014142098997249

more news is coming soon.

[Invincibl-e]

Refactoring is a very good choice, now unicorn feels devastated.

[JeffreyO]

Will this allow direct loading and emulation of binary files without modification or wrappers? For an example, a rip from an EPROM that is only machine code without a bootloader or kernel image.

[wtdcode]

Will this allow direct loading and emulation of binary files without modification or wrappers? For an example, a rip from an EPROM that is only machine code without a bootloader or kernel image.

You would like to have a look at https://github.com/qilingframework/qiling

[aquynh]

super excited to released Unicorn2 to the public.

check it out at https://www.unicorn-engine.org/Unicorn2-beta

[raphaelthegreat]

super excited to released Unicorn2 to the public.

check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

[wtdcode]

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

[raphaelthegreat]

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

Is it possible then to add multi-threading support to the Unicorn APIs? I mean in the features list it states that Unicorn is thread safe by design. And now that QEMU is updated as well it should be easier to do that? But anyways I don't think multi-threading is super important, just something that's nice to have.

[wtdcode]

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

Is it possible then to add multi-threading support to the Unicorn APIs? I mean in the features list it states that Unicorn is thread safe by design. And now that QEMU is updated as well it should be easier to do that? But anyways I don't think multi-threading is super important, just something that's nice to have.

From the top of the view, we are leaving this part for users. We would consider adding the support if it's really the case for example by some benchmark.

[dmknght]

Hello! I created the binding for Nim using nimterod generator at https://github.com/dmknght/unimcorn. But the testing code (converted to Nim from example on unicorn's homepage) in readme gave me error Failed on emu start, error: Invalid memory read (UC_ERR_READ_UNMAPPED). I guess the error was wrong variable's types. I hope somebody can help me check the bug so I'm able to fix the binding and create a pull request.