add QEMU mode

[tomirgang]

This PR allows to use a local QEMU VM without the need to use libvirt.

This change makes use of elbe in containers much more easy, since libvirtd is hard to containerize and comes with a long list of dependencies and requirements.

[t-8ch]

Thanks!

FYI it is a general goal of me to simplify the initvm (and maybe also remove it somehow)
Running in containers is also a general goal.

[jogness]

FYI it is a general goal of me to simplify the initvm (and maybe also remove it somehow) Running in containers is also a general goal.

Note that ELBE originally did not use a virtual machine. This change was in 2012 (unfortunately pre-github and pre-mailing list). The project started in 2009. The main reasons why the ELBE team decided to change the build environment to be in a virtual machine:

• Minimizing dependencies so that ELBE can run on different Linux distributions and operating systems (even native Windows is supported).
• Having full control over dependencies so that ELBE can build reliably on any host system.
• Running foreign-arch software requires binfmt support.
• Avoid pulling in build host details from /proc /sys /dev.
• Use root for many tasks without risks to host.

Perhaps some of these requirements have changed in the last decade. The available technologies certainly have changed. I just wanted to point out these reasons because they tend to get lost when a project switches maintainers a few times.

BTW, I still have many interesting emails about ELBE design requirements that were posted to the elbe-devel mailing list (before the list was archiving). It may be of use to look them over to avoid repeating the same mistakes we made 15 years ago. It may also be of value to create some sort of manifesto so that the goals/requirements are not lost 10 years from now.

[tomirgang]

FYI it is a general goal of me to simplify the initvm (and maybe also remove it somehow) Running in containers is also a general goal.

Note that ELBE originally did not use a virtual machine. This change was in 2012 (unfortunately pre-github and pre-mailing list). The project started in 2009. The main reasons why the ELBE team decided to change the build environment to be in a virtual machine:

• Minimizing dependencies so that ELBE can run on different Linux distributions and operating systems (even native Windows is supported).
• Having full control over dependencies so that ELBE can build reliably on any host system.
• Running foreign-arch software requires binfmt support.
• Avoid pulling in build host details from /proc /sys /dev.
• Use root for many tasks without risks to host.

Perhaps some of these requirements have changed in the last decade. The available technologies certainly have changed. I just wanted to point out these reasons because they tend to get lost when a project switches maintainers a few times.

BTW, I still have many interesting emails about ELBE design requirements that were posted to the elbe-devel mailing list (before the list was archiving). It may be of use to look them over to avoid repeating the same mistakes we made 15 years ago. It may also be of value to create some sort of manifesto so that the goals/requirements are not lost 10 years from now.

The initvm is from my point of view a good thing, and wit the QEMU kvm and user-mode combination the resulting build performance is quite good.

My challenge is that libvirt is hard to use in a VS Code devcontainer (but it's possible, I have a working PoC), and QEMU kvm without libvirt is very easy to use (only a privileged container is needed). Without kvm not even a priviledged container is needed, but then the build performance is horrible slow

[tomirgang]

@t-8ch Your review findings are now fixed form my point of view. Please tell me if you don't agree.

[t-8ch]

The initvm is from my point of view a good thing, and wit the QEMU kvm and user-mode combination the resulting build performance is quite good.

It is indeed a good thing. My hope is to reach all of its goals but avoid some of the problems the current implementation incurs.

Please tell me if you don't agree.

A few things still need some work, please see my comments.

[tomirgang]

@t-8ch Rebase and rework done. Please review again.

[tomirgang]

@t-8ch Most of your remarks are fixed, but I don't think it should be part of this PR to implement a shell client.

[t-8ch]

Thanks! Just some more tiny nitpicks.
Could you also clean up the commit history?

As for the shellclient:
I'll implement one myself after your changes are merged.
It would be great if you can send the patch to you for testing then.

[tomirgang]

Thanks! Just some more tiny nitpicks. Could you also clean up the commit history?

As for the shellclient: I'll implement one myself after your changes are merged. It would be great if you can send the patch to you for testing then.

Done.

I can test the shell client if you want to implement one.
Form my point of view, it would be a better way forward to drop the attach command or only print a help text how to attach, instead of implementing unrelated features. If you want to have a robust shellclient, which is handling all edge cases and special characters correctly (e.g. Ctrl + C), I guess it will become quite complex.

[t-8ch]

Done.

Thanks!

If you want to have a robust shellclient, which is handling all edge cases and special characters correctly

My idea was that all characters are blindly forwarded without any special handling. Ctrl-C should not be a character at all but an exception that gets generated by python itself.

[t-8ch]

Please also check the code style with flake8 elbepack/ --max-line-length 100

[tomirgang]

Done.

Thanks!

If you want to have a robust shellclient, which is handling all edge cases and special characters correctly

My idea was that all characters are blindly forwarded without any special handling. Ctrl-C should not be a character at all but an exception that gets generated by python itself.

From my point of view, this is not a good idea. Not being able to cancel or sleep shell processes in the VM is a huge restriction.

[tomirgang]

flake8 elbepack/ --max-line-length 100

Done.

[t-8ch]

Toplevel note (to myself):
This is used for devcontainer: https://github.com/tomirgang/elbe_dev_container/tree/main/.devcontainer

[t-8ch]

Thanks! I picked this up manually to add my review tag.