Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicorn 2 - looking for sponsors #1217

Open
aquynh opened this issue Feb 28, 2020 · 19 comments
Open

Unicorn 2 - looking for sponsors #1217

aquynh opened this issue Feb 28, 2020 · 19 comments
Labels

Comments

@aquynh
Copy link
Member

aquynh commented Feb 28, 2020

Since its born in 2015, Unicorn has been inspiring so many cool research & tools, that we believe it deserves serious upgrade. Based on Qemu 2.1.2, there is a lot to catch up with the latest version of Qemu (Qemu is at v4.2 at the moment). Unfortunately, the code base of Qemu changes so much since 2015, that we believe it is faster to rewrite Unicorn from scratch.

For this reason, next major update of Unicorn (lets call it Unicorn 2) requires huge effort & lots of time. So we call for the support from Unicorn users!

If you are using Unicorn for your work, and wish to sponsor the development of Unicorn 2, please contact us: http://www.unicorn-engine.org/contact/

@aquynh aquynh pinned this issue Feb 28, 2020
@korniltsev
Copy link

korniltsev commented Feb 28, 2020

are you going to keep in sync with upsream after rewrite?

@aquynh
Copy link
Member Author

aquynh commented Feb 29, 2020

@Waterman178
Copy link
Contributor

Waterman178 commented Mar 1, 2020

Support for the next version!

@XVilka
Copy link

XVilka commented Mar 4, 2020

Related #1190
Regarding the keeping in sync - it might be more useful to create a series of semantic patches (with spatch tool) to generate the Unicorn on the fly. There is a mature tool to do this called Coccinelle, see their documentation for more details. There is a good introductory article on how to create semantic patches. Coccinelle used for the Linux kernel sources refactoring, so it's stable and featureful enough.
See the example of this semantic patch:

@@
expression lock, flags;
expression urb;
@@

 spin_lock_irqsave(lock, flags);
 <...
- usb_submit_urb(urb)
+ usb_submit_urb(urb, GFP_ATOMIC)
 ...>
 spin_unlock_irqrestore(lock, flags);

@@
expression urb;
@@

- usb_submit_urb(urb)
+ usb_submit_urb(urb, GFP_KERNEL)

@minexew
Copy link

minexew commented May 6, 2020

@ XVilka Due to how Unicorn is created (ripping out the heart of QEMU and fixing it up to make it nicely embeddable), it unfortunately seems that any new branch of Unicorn is doomed to ultimately fall out of sync with QEMU upstream again.

@GPUCode
Copy link

GPUCode commented May 23, 2020

Is there any update on Unicorn 2?

@kzsnow
Copy link

kzsnow commented Aug 3, 2020

My two cents: I did a bit of "Unicorn 2" prototyping with Qemu 4.2. The two biggests "lifts" I encountered, in terms of source code modifications, were compartmentalizing qemu globals (e.g. when multiple emu instances are used), and dealing with the memory implementation.

Like others, I'm cautiously optimistic there is a way to minimize Qemu source code mods to make it easier to keep up with development there, and greatly lessen the burden for updates.

A quick hack to get around the issue of qemu globals without source code mods is to load unique instances of the shared library for each "emu" instance. Of course the trade-off is increased memory consumption (however big the qemu lib is, for each instance). I've used Unicorn quite a bit, and in all my use cases (personally) this would be an acceptable trade-off.

Regarding memory, there are two options: softmmu or using the host system's mmu. To use the host system's mmu, I didn't need to modify qemu source code at all (basically just linux-user mode without syscall passthrough). Besides that, performance is 5x better than Unicorn/softmmu in some rudimentary tests. It would be really useful to be able to run "Unicorn 2" in this mode for the performance boost, when needed (i.e. fuzzing, etc.).

There are massive downsides to using the host mmu though: emulated code can no longer be securely sandboxed (not that I can tell, at least), and it restricts where regions can be mapped in the emulation (compatibility). Hence, ultimately we'd want the ability to use softmmu. This is where I ran into the biggest issue. The qemu "system vs. host" and "softmmu vs. hostmmu" #defines are not separated well, and source code mods started building up. At this point, the time investment outweighed the benefit.

So I'll put this out there for discussion. The way forward might be to start with pull requests on Qemu towards the goal of adding a "softmmu vs. hostmmu" option for the linux-user build of qemu. I vaguely recall that someone was attempting this years ago and Qemu devs seemed receptive, but it didn't materialize. If we had this option in Qemu, I'd wager that Unicorn-like functionality would be much easier to build out.

@Announcement
Copy link

Announcement commented Apr 20, 2021

I'm not a typical contributor and not too involved in this project but if QEMU is really that active, why not ask them to publish all the apis and such that you need so we can basically use an unmodified qemu and just plug into it like a dependency? That probably sounds stupid to those who have been working on a unicorn but, what kind of modifications are being made?

@wtdcode
Copy link
Member

wtdcode commented Apr 21, 2021

I'm not a typical contributor and not too involved in this project but if QEMU is really that active, why not ask them to publish all the apis and such that you need so we can basically use an unmodified qemu and just plug into it like a dependency? That probably sounds stupid to those who have been working on a unicorn but, what kind of modifications are being made?

Modifications include (not all of them!):

  • Do instrumentation in the translation stage and memory operations to support various hooks.
  • Various checks (and even modify tcg IR) to make sure the execution can be stopped anywhere and resumed cleanly.
  • Modify some qemu helper functions to handle interrupts and syscalls.

Qemu code is clean indeed, but not all these modifications can be easily done without some really dirty hack.

@aquynh
Copy link
Member Author

aquynh commented Apr 30, 2021

We are redesigning the logo of Unicorn for Unicorn2, with the target of having a more modern logo, that can be used better on Tshirt, mug, etc.

If anybody wants to sponsor the new logo design (so your support will be there forever in the project history ;-), please contact.

@aquynh
Copy link
Member Author

aquynh commented Jun 6, 2021

towards Unicorn2, we redesigned the logo - for this major version.

https://twitter.com/unicorn_engine/status/1400014142098997249

more news is coming soon.

@Invincibl-e
Copy link

Invincibl-e commented Jun 11, 2021

Refactoring is a very good choice, now unicorn feels devastated.

@JeffreyO
Copy link

JeffreyO commented Aug 28, 2021

Will this allow direct loading and emulation of binary files without modification or wrappers? For an example, a rip from an EPROM that is only machine code without a bootloader or kernel image.

@wtdcode
Copy link
Member

wtdcode commented Oct 1, 2021

Will this allow direct loading and emulation of binary files without modification or wrappers? For an example, a rip from an EPROM that is only machine code without a bootloader or kernel image.

You would like to have a look at https://github.com/qilingframework/qiling

@wtdcode wtdcode mentioned this issue Oct 3, 2021
@aquynh
Copy link
Member Author

aquynh commented Oct 3, 2021

super excited to released Unicorn2 to the public.

check it out at https://www.unicorn-engine.org/Unicorn2-beta

@GPUCode
Copy link

GPUCode commented Oct 3, 2021

super excited to released Unicorn2 to the public.

check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

@wtdcode
Copy link
Member

wtdcode commented Oct 3, 2021

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

@GPUCode
Copy link

GPUCode commented Oct 3, 2021

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

Is it possible then to add multi-threading support to the Unicorn APIs? I mean in the features list it states that Unicorn is thread safe by design. And now that QEMU is updated as well it should be easier to do that? But anyways I don't think multi-threading is super important, just something that's nice to have.

@wtdcode
Copy link
Member

wtdcode commented Oct 3, 2021

super excited to released Unicorn2 to the public.
check it out at https://www.unicorn-engine.org/Unicorn2-beta

Thanks a ton @aquynh for this wonderful release! One question though, does Unicorn 2 incorporate the multi-threaded TCG? I'm so excited for this release as Unicorn is ideal for projects like emulators and adding multi-threading will make it even more powerful.

No. Unicorn internally is single-threaded.

Is it possible then to add multi-threading support to the Unicorn APIs? I mean in the features list it states that Unicorn is thread safe by design. And now that QEMU is updated as well it should be easier to do that? But anyways I don't think multi-threading is super important, just something that's nice to have.

From the top of the view, we are leaving this part for users. We would consider adding the support if it's really the case for example by some benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests