Change the OS used in dom0

[mfc]

We have discussed this numerous times but don't have an issue to track these discussions. It would be worth understanding what would be needed to change dom0 from Fedora to Debian (say Debian 8). Benefits include:

• increased hardware compatibility
• incorporate serious work taken towards reproducible builds
• better firstboot installer
• better (slower) release cycle than Fedora with longer-term support
• other things?

This ticket does not encompass modifying the desktop environment.

[TNTBOMBOM]

the problem is , why anyone would take fedora which is very known for its issues (from about 2000) and their enterprise lockin on the code (redhat sucking stuff...) ?

is there anyone can give me good reasons why taking fedora at the first place ? why not debian ? why not even gentoo (which is regarding of code easiness/freedomcy better than fedora)?

im afraid the one who took the first (lets call it "bad") decision is still continuing to take these decisions which will make qubes a disaster even if u shifted to debian or X distro.

so i think consultations on these decisions is very important. and fedora to be the core of qubes or any other linux distro is a mistaken step.

[woju]

Fedora is currently hosting the graphical environment and, like it or not, it has the best compatibility with graphic cards. Debian generally has old X drivers and older kernel. This is especialy problematic with Qubes, because the hardware selection is already constrained by virtualisation extension. We will surely consider switching the distro when we implement GUI domain. It is on out roadmap and depends on GPU passthrough, which does not currently work, at least not without patching Xen and libvirt. We this it will be hard enough, and the effort should be directed there. When switching, it will be neither Fedora nor Debian or any general purpose distro, but something very minimal, like Alpine.

[Jeeppler]

What exactly is the issue with GPU passthrough and Xen?

[marmarek]

Take a look at this thread - it's a good summary of GPU passthrough and PCI passthrough to HVM at all: https://groups.google.com/d/topic/qubes-devel/mvZBIUuyjv0/discussion

[minad]

It would be really nice to have a choice here. Like @woju I think it would be best to have a minimal distribution. However it might be nice to try something more declarative like NixOS.

I guess you wouldn't want to maintain different distributions, at least not for the official distribution? Do you have an idea how many Fedora specific configuations are in the qubes builder and the applications?

[marmarek]

I guess you wouldn't want to maintain different distributions, at least not for the official distribution?

Exactly.

Do you have an idea how many Fedora specific configuations are in the qubes builder and the applications?

During the current Fedora 20 -> 23 upgrade I'm trying to collect that list. Actually two lists:

• customization of GUI domain (which currently is the same as dom0) - this list is almost completed based on KDE5 decoration plugin #1784
• things customized in dom0 in general

I think the only really Fedora specific thing is update mechanism, other parts are just some system configuration which may need some adjustment when porting to other distribution (like config file locations etc).

Yes, we're considering NixOS here, but no decision has been made yet.

[unman]

I think the only really Fedora specific thing is update mechanism, other parts are just some system configuration which may need some adjustment when porting to other distribution (like config file locations etc).

I've got some experience here as I've been running Debian in dom0 for a while. It's an unholy mix of packages, alien and hand crafted stuff,(hand crafted like the Borja Jesus), and not full Qubes. But it works well enough for my slight requirements. I'm hoping that the upgrade to 3.1 will be somewhat closer to full Qubes.
So far I would say there is almost nothing Fedora specific that comes outwith the normal repackaging process.

It's interesting that @mfc suggests that Debian provides increased hardware compatibility, while @woju says Fedora has best compatibility with graphic cards. Is there any evidence either way?

[minad]

The Borja jesus 😂 That still sounds somehow encouraging! What are your reasons for prefering Debian on the dom0? I just started using qubes and I had much more experience with Debian, Arch, ... but basically none with Fedora. It took a short while to get accustomed - but in the end it feels all the same with some minor package manager differences. Since qubes takes a more radical approach in general, I think it would also make sense to consider a change to something conceptually "better" or more advanced like nix.

[h01ger]

unman, could you please share more details about your setup, maybe even commits? ;-) Are you running this on a jessie based dom0 or stretch? How did you build+install the qubes specific stuff?

I'm still pondering doing the same, and looking at the cubes git repos I've identified these as being needed for dom0:

• qubes-core-admin
• qubes-core-admin-linux
• qubes-core-agent-linux
• qubes-core-libvirt
• qubes-core-vchan-xen

Is that correct? Which are missing? Cause once dom0 is working one should be able to deploy VMs from a backup… ;-)

[marmarek]

On Wed, May 18, 2016 at 09:03:43AM -0700, Holger Levsen wrote:

I'm still pondering doing the same, and looking at the cubes git repos I've identified these as being needed for dom0:

• qubes-core-admin
• qubes-core-admin-linux
• qubes-core-agent-linux
• qubes-core-libvirt
• qubes-core-vchan-xen

Is that correct? Which are missing?

In addition to above:

• qubes-vmm-xen (yes, you need this, not native Debian packages because
  of Qubes-specific patches, and parts in Debian packages - libxenvchan,
  stubdomains)
• qubes-gui-common (required to build gui-daemon)
• qubes-gui-daemon
• qubes-core-qubesdb
• qubes-manager

And mgmt-salt stuff (everything with "base" and "dom0").

Some of them already have debian packaging.

Packaging Xen for dom0 would be tricky because of stubdomains. It
consists of some external libraries/tools built as part of Xen build and
statically linked. Something that AFAIR is against Debian policy (for
example there are multiple source tarballs, from different projects).

And template packages needs to be converted to deb. Or not packaged at
all and installed some other way - which I think would be even better
option (installing templates as packages have a lot of side effects...).

Cause once dom0 is working one should be able to deploy VMs from a backup… ;-)

Yes, it should be possible.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[h01ger]

Marek, thanks for the hints!

For a start I don't mind violating Debian policy if needed. That said, while statically linking and embedded code copies are discouraged in general, there are also exceptions to these rules.

But really, at first I just want to be able to use a Debian based system as dom0.

[unman]

@h01ger
Worked through the build logs from fedora and the spec files. I followed the same build order and worked through each package in turn. I started with a base jessie install, and built up from there.
I tried cherry picking Xen packages from sid and applying the Qubes patches, and that worked reasonably but I found difficult to incorporate stubdomains, and got pretty lost. In the end I basically built everything from source. It was relatively easy to identify Debian equivalents for the fedora requirements, and referencing the native Debian packages also helped, particularly for Xen.
I had some success using alien for simpler stuff, although some paths and scripts needed to be bodged by hand.
I didn't build manager or salt stack, but I cant imagine they would be difficult.
Some stuff just doesn't work at all - can't boot from iso in HVM, for example. And nothing is really packaged so it's probably fragile on updates.

I'm currently working on 3.1, and trying to do the right thing, packaging properly and (sometimes) using qubes-builder framework, but I have limited time before my next trip. When I have anything sensible to share I will certainly commit, but I'm a way off that at present.

When I started there was a clear thought that Qubes would be moving to Debian, and I wanted to see what would be involved. Now, of course, that's no longer the case, so I suspect time is better spent working on Debian issues on the vm side.

[Jeeppler]

I thought one of the main reasons to switch from Fedora to Debian are reproducible builds. Debian made and make a lot of progress in this area.

The other question which was discussed before is moving from KDE to GTK (Gnome) for the desktop.

@woju proposed above a switch to Alpine Linux. I understand that Alpine Linux has nice security features, but it seems to be a radical switch. A switch to Alpine would be less radical if the management GUI and dom0 would be separated. In the case the separation between GUI and dom0 would be possible and implemented, dom0 would be Alpine, but what about the GUI domain?

Why not going step by step. First moving from Fedora to Debian, because of the ability to create reproducible builds. Then moving from KDE to GTK (Gnome). The final step would be to separate Dom0 into separate domains. The management domain would be Alpine whereas the graphical engagement domain would be still Debian.

Another good point of @woju is the problem with drivers, older kernel and drivers in Debian. This is true for Debian stable, but Debian has a testing, unstable and experimental repository which contain more up to date packages.

Debian and Linux in general has many distribution derivatives. Why not using one of the derivatives which fulfills the requirements:

• based on Debian
• static release model
• up-to-date driver

Tanglu is a distribution which fulfills all the requirements. Tanglu is based on Debian, but uses software packages from testing, unstable and sometimes experimental to provide the newest drivers and other software packages, but is still based on Debian stable. Tanglu has three main editions KDE, GNOME and core. In addition Tanglu uses Calamares the independent system installer.

[adrelanos]

For reference:

• https://wiki.debian.org/Qubes
• https://wiki.debian.org/Qubes/Devel

[h01ger]

https://wiki.debian.org/Qubes/Devel is the relevant URL for this issue.

https://wiki.debian.org/Qubes just describe how to use Qubes from the point of view of someone familar with Debian - and that wiki page is also a bit outdated.

[adrelanos]

@unman in #1781 (comment)

I don't think that #1919 is still a target.

How did you come to that conclusion?

In any case it seems to me that most users should be kept well away from dom0 and so what is running there is irrelevant. If anyone has the nous to tinker in dom0 then they should be able to handle the differences.

I don't think Fedora can be as transparent as that. Once one wants to do certain things, one example is #1375 and there are others, one is back to learning about Fedora. Same when issues happen or when debug info is requested. And that probably is not going to go away. Also higher effort for developers to know two systems. For example makes getting grsecurity in Qubes harder since an rpm package / repository is required.

[tasket]

If the user needs to use this information to decide whether the update is safe, then this is going to end up being ignored or misunderstood by the vast majority of users. Does not look like a very nice UI. :)

I think this is also falling into the trap of reinforcing the pre-Qubes status quo, which is "hide context from end users", instead of thinking strategically about what role the user can play. Displaying a timestamp gives us a better chance to defend ourselves.

What's concerning is the not-a-nice-UI argument logically leads to this question..... Why use Qubes at all when the user has to use VM context info to decide what is safe? "this is going to end up being ignored or misunderstood by the vast majority of users", right?

So I don't think the assumptions used for that argument are compatible with Qubes (because they negate Qubes) and in general I would hope that Debian, Fedora, etc. update their threat models to flesh-out the scenarios being discussed here.

Andrew stated:

If this is a correct description of the problem, then it seems like the fundamental problem is a philosophy that's too trusting of the infrastructure.

Yes, these big distros have not fleshed-out their threat models for system updates. That is understandable because the thinking is so old, practically from a different generation when certain classes of threats (or threat actors) were not taken seriously.

Another suggestion:

Don't cling to the idea that sneakernet and cherry-picked installs have to provide the smoothest verification path. Best practices should not be bent around that expectation. It would be healthier to show a warning and/or require a --force option in the least-secure situations (when no signed metadata for that package-version is available), and give users tools and guidance to acquire the needed metadata in those corner cases. For me, that would answer the question over signed metadata vs signed packages, with the former being preferred.

[andrewdavidwong]

@DonKult: Thank you very much for the detailed response. I apologize that you were summoned here. You're right that I should have asked my questions on the appropriate Debian mailing list instead, but thank you for indulging my curiosity here nonetheless.

Apologies to others, as well, if I took us too far afield from the original topic. As many of you have noted, the high-level questions about trust requirements are relevant to this issue, but the fine-grained details about specific package management systems are probably best hashed out elsewhere. 🙂

[DemiMarie]

@DonKult From what I can tell, the real problem is that the main process puts too much trust in the helper. If the helper is trusted, there is no real point in confining it. If it is not trusted, then the hashes it sends back also cannot be trusted. Instead, the parent process needs to copy the file (since the helper might still have access to the file it just downloaded) and calculate the hash of the copy.

[indolering]

Yes, these big distros have not fleshed-out their threat models for system updates. That is understandable because the thinking is so old, practically from a different generation when certain classes of threats (or threat actors) were not taken seriously.

Don't discount economics: there is no commercial motivation to sign these packages. Red Hat doesn't provide GPG signatures for RHEL ISOs, you verify the ISO checksum by signing into their licensing portal. Red Hat, Ubuntu, and Suse respond to questions about reproducible builds with marketing speak about their secure build process.

I'm not proposing some evil conspiracy, I think it is possible to get Fedora and Debian to start signing everything. But either be prepared to volunteer a lot of personal time and effort or find some company that would benefit from this financially and get them to pay for it.

That isn't decided yet, but at the point when we'll be minimizing dom0, more likely option would be something capable of building light system images, for example Yocto.

Why would you switch to a distro designed for embedded hardware or building containers? Both distros and hardware manufacturers put in a ton of work into keeping consumer hardware working. You are going to cut off access to a large percentage of hardware and make updates much more risky for end users.

[tasket]

Red Hat doesn't provide GPG signatures for RHEL ISOs, you verify the ISO checksum by signing into their licensing portal.

Indeed. But that just throws Fedora's practices into high relief. Its the odd one out, and the other distros manage just fine signing their metadata.

[tabbyrobin]

Hope I don't crowd this already crowded topic too much. Thought I'd pull up this overview from a while back:
#1919 (comment)

I'll repeat part of it here: competing+conflicting criteria for dom0. (In parentheses, distro(s) that do(es) each thing well.):

1. Unchanging/slowly changing base on which Qubes can be built. (debian, centos)
2. Small TCB in terms of minimal installed size. (yocto, fedoraCoreOS)
3. Good driver/HW support. (fedora)
4. Large external support community. (debian, fedora)
5. Reproducible builds. (debian, ?yocto?)
6. Safe upgrades. (fedora SB)
7. Up to date/recent software. (fedora)
8. Be the same as one of the default templates, don't proliferate systems. (fedora, debian)

It's been a year and sys-gui seems to be coming together. Because of that, fedora is much less attractive for dom0 than it used to be. Dom0 won't need recent software or much hardware support. That can all be handled by fedora in sys-gui, sys-* and domUs.

Debian seems like one obvious choice. The only major downsides seem to be the difficulty in making the switch from Fedora (retooling Qubes); and the fact that transactional upgrades (ostree) won't be coming.

Yocto was also mentioned but this seems like it might be a more long-term goal? Seems like ad-hoc configurability in dom0 is still going to be useful for the near future. (Until maybe dom0 is further atomized/decomposed.)

Either Debian or CentOS + some kind of ostree layer (even a Qubes-custom layer) might be interesting. Not sure how feasible coding an ostree layer for Debian would be. EndlessOS has coded a very rudimentary ostree layer for Debian. (But it doesn't include package layering like Fedora Silverblue.) CentOS used to have rpm-ostree integration but it has been dropped for future releases. Also, personally I don't know much about CentOS/its community, though others here have spoken favorably about it.

[DemiMarie]

I think that CentOS would be a good choice for dom0, as it can have a recent kernel, yet is extremely stable and has signed packages and signed metadata.

I've already tried to build Qubes with dom0 having CentOS 8. Like for VM, there is a lot of missing packages that we need. For example, here the ones needed for VM that I needed to adapt/build: https://copr.fedorainfracloud.org/coprs/fepitre/epel-8-qubes/packages/ and https://copr.fedorainfracloud.org/coprs/fepitre/epel-8-python38/packages/.

@fepitre Most of those seem to be related to documentation, which we don’t need to build in dom0.

[marmarek]

While sphinx indeed is about documentation, it is build dependencies of some packages. So yes, we do need those packages in dom0 (build) environment, even if not installed in the target system. I don't think excluding (for example) qvm-* tools man pages from dom0 packages is a good idea - if anything, we have too little documentation available in the system, not to much.

But also, it doesn't seem to me that "most" is related to documentation. While some of the packages in "epel-8-python38" repo are about sphinx, it doesn't look to be even half of it. And in "epel-8-qubes" repository I see at most one package related to the documentation.

[DemiMarie]

How difficult would it be for us to build the remaining packages ourselves?

[fepitre]

How difficult would it be for us to build the remaining packages ourselves?

I've started to wrote some tool to ease Fedora/CentOS src package -> Qubes src package. According to CentOS git, there is packages that are now in CentOS 8 beta which would help us a little bit but still, it would remain something like one month of work to have dom0 under CentOS8 which is not in our possibilities currently.

[DiagonalArg]

... why not debian ? why not even gentoo (which is regarding of code easiness/freedomcy better than fedora)?

Except for this quick mention, no one took up the question of Gentoo. I may not be the most educated in regard to security matters, but I do see that Gentoo is compatible with Xen, it has a Hardened version, and it being compiled from sources, would be optimized for the platform. As a rolling release, it would make Qubes much easier to upgrade, as it would only be the VM's with point releases.

This is a bit OT, but I might also ask why sys-firewall isn't something like pfsense/OPNsense (BSD based) or IPFire (Linux based). Or if you want more full-blown options, there are the CentOS based NethServer or Koozali. These are built to purpose and offer all kinds of relevant software from bonding multiple WANs to IDS's like Snort or Suricata.

[adrelanos]

It might not be great for productivity to mix up discussions on:

• technical progress, actual development, how to port Qubes dom0 to Debian mixed up with
• general discussion about Qubes dom0 going for Debian or not
• considering suitability of other Linux distributions for dom0
• having a general Linux distribution wars debate (there are tons of endless of these on the internet already)
• advocating one Linux distribution over another for Qubes dom0 (that should be the final discussion. Included should only be distributions which where previously considered suitable.)

---

A good question to ask for any proposed Linux distribution in my opinion is:

Does it pass the TUF (The Update Framework) threat model?

• https://theupdateframework.io/security/
• https://web.archive.org/web/20201127155237/https://theupdateframework.io/security/

And if the answer is no: that distribution is unstable.

---

May also also suggest to develop a list of generic criteria before reviewing specific Linux distributions? For inspiration, see Criteria for Choosing a Base Distribution.

---

On Gentoo / Hardened Gentoo...

Quote https://www.whonix.org/wiki/Dev/Operating_System#Gentoo_.2F_Hardened_Gentoo

Insecure package manager. Back then bug reports got closed down without much regard.

• unsigned Gentoo latest files Whonix/Gentoo-Port#19 - https://bugs.gentoo.org/show_bug.cgi?id=539954
• Gentoo Updater: Find out if it passes the TUF Threat Model Whonix/Gentoo-Port#10 - https://archives.gentoo.org/gentoo-portage-dev/message/94425239fcaedcee6c49ef398f12aa85
• https://phabricator.whonix.org/T212#5691
• [gentoo-portage-dev] Security and Comparison of Portage with other Package Managers: https://archives.gentoo.org/gentoo-portage-dev/message/bda425ee6c676ec7a6b3c9500a9b00bf
• [gentoo-portage-dev] Portage and Update Security: https://archives.gentoo.org/gentoo-portage-dev/message/94425239fcaedcee6c49ef398f12aa85

In this regard, Hardened Gentoo does not differ from Gentoo.

Due to the way these bug reports were handled, Gentoo was removed from the candidates of secure base operating systems.

[DiagonalArg]

@adrelanos - Thank you. That explains the situation with Gentoo. I appreciate your patience. At risk of taxing it, I will ask if any consideration have been given to the use of a BSD for Dom0? I see at least two are compatible with Xen, but there has been no mention at all on this thread.

[andrewdavidwong]

Please, let's try to limit further comments on this issue to only what is essential for developers to do their work. qubes-issues is not intended to be a discussion forum, and this issue is getting too big to be helpful to the developers. If anyone wants to ask questions for their own understanding or have a more general discussion about these topics, please take it to the appropriate mailing list.

[indolering]

Could @mfc or one of the admins edit the OP to link to the Alt Distro in dom0 forum post and close|lock this ticket? I think it's outlived its usefulness.

[0spinboson]

Marek (2016):

Yes, we're considering NixOS here, but no decision has been made yet.

out of curiosity, I assume this was rejected at the time, but could you say why?