don't really boot TemplateVM, only chroot into those

[adrelanos]

I think booting a TemplateVM directly is bad. During boot, various daemons start [custom installed packages] and create their data folders, random seeds are created and so forth. All that date is then shared with AppVMs, so it could happen, that those all share the same random seeds.

So I am suggesting to not "really" boot into the TemplateVM. From the point of view of the user nothing should change. But when one boots a TemplateVM, what really should happen under the hood, is creating a temporary AppVM based on that Template. And then chrooting into the TemplateVMs image. When one would update or install new packages inside the TemplateVM, these services would not start up, hence not make the image "dirty".

[marmarek]

I think it is a bad idea, for many reasons:

1. Creating initial data is in many cases desirable. Remember that rootfs changes are not persistent in AppVMs, so when some service needs some more work at first startup, it will be done at each VM startup. Results in bad performance, long VM startup, waste of CPU power and high IO (especially important on laptops, with SSD). Some example of such service is dnf-makecache - it creates cache of metadata from every installed and available package, takes a lot of time (minutes, or even tens minutes, 100% CPU). This one we have specifically disabled, but this isn't the only such service. And BTW we'll probably need to reenable dnf in Fedora 22 template...
2. Starting TemplateVM is not always only about installing packages. Some users want to make system-wide changes. And some of them, especially when using GUI tools, require running (dbus) services which applies the changes. This is also about GUI updaters (Graphical update of Fedora 21 template reports no updates available #982), which we want to resurrect one day.
3. Such approach would make the whole template update/installation actions really fragile. Need to track upstream changes, probably apply some hacks/workarounds for things not working in chroot.
4. If you fear some initial random seeds creation, this approach doesn't help fully - because some packages creates them in post installation scripts. So it would be just half-solution, which isn't generic enough, but introduce a lot of potential problems (now and in the future).
5. Qubes OS is not only for anonymity. In some cases (especially when one use a lot of VMs) higher priority is performance (VM startup time) over anonymity. This of course should be done in a way not compromising security (think - cryptography).

So, instead, I would suggest preparing some "blacklist" of packages known of creating unique initial data, which should really be unique for every VM. This list would be probably different for Whonix and non-Whonix templates. And then clean such data (it should be simple to create some service running at TemplateVM shutdown, which would remove/sanitize listed files). This isn't generic solution (neither is yours), but IMHO have much less potential for breaking random things (there are over 2k packages in template and probably only few of them require such treatment).

Another solution would be to move such (selected) data to /rw partition (either using symlinks or bind-mounts). Then each VM would have its own, persistent copy. But this would be probably much more service-specific and require more work.

[adrelanos]

Marek Marczykowski-Górecki:

I think it is a bad idea, for many reasons:

Sure. The final decision is yours. Just trying to brainstorm here. I am
looking for the most generic, best maintainable, least code, you name
it, solution here. I try to specific some more. If my idea still doesn't
fly, just close this one.

1. Creating initial data is in many cases desirable. Remember that
   rootfs changes are not persistent in AppVMs, so when some service
   needs some more work at first startup, it will be done at each VM
   startup. Results in bad performance, long VM startup, waste of CPU
   power and high IO (especially important on laptops, with SSD). Some
   example of such service is dnf-makecache - it creates cache of
   metadata from every installed and available package, takes a lot of
   time (minutes, or even tens minutes, 100% CPU). This one we have
   specifically
   disabled,
   but this isn't the only such service. And BTW we'll probably need to
   reenable dnf in Fedora 22 template...

Maybe there is a contradiction here or lack of understanding on my side.
On one hand, officially distributed image builds are created in chroot
and never booted [systemd services never run] before they get deployed.
That is considered fine by both of us.

If the user boots the template and installs packages, you prefer the
services (systemd...) to actually start. On the other hand, I suggest
these should be handled same as if in chroot.

But imagine a user just downloaded a fresh TemplateVM and created an
AppVM based on it. Then just starts the AppVM without ever starting the
TemplateVM beforehand. Doesn't that AppVM then miss everything you
mentioned?

1. Starting TemplateVM is not
   always only about installing packages. Some users want to make
   system-wide changes. And some of them, especially when using GUI
   tools, require running (dbus) services which applies the changes.
   This is also about GUI updaters (Graphical update of Fedora 21 template reports no updates available #982), which we want to resurrect
   one day.

For system-wide changes:
What I had in mind would run the terminal, file manager, etc. already
chrooted into the template image.

GUI tools, dbus in chroot:
I don't know if there would be a sane solution to just selectively and
dynamically start what the user requires. A quick search implies, that
GUI and dbus can be used within chroot.

What I am suggesting is basically just to start in the TemplateVM what
the user wishes, not all the system default. I.e. run in chroot,
apt-get, Apper with its dbus dependency, nautilus etc. As opposed to
starting the whole system default. That would for example exclude
openssh-server from starting and creating its private keys.

1. Such approach would make the whole template
   update/installation actions really fragile. Need to track upstream
   changes, probably apply some hacks/workarounds for things not working
   in chroot.

I don't think so. This is same as 1. If we consider package installation
during build in chroot fine, then package installation by the user in
chroot also should be fine? I've never came across a package that is not
installable inside chroot. I guess that would be considered as a release
critical bug by upstream.

1. If you fear some initial random seeds creation, this
   approach doesn't help fully - because some packages creates them in
   post installation scripts. So it would be just half-solution, which
   isn't generic enough, but introduce a lot of potential problems (now
   and in the future).

I see.

1. Qubes OS is not only for anonymity. In some
   cases (especially when one use a lot of VMs) higher priority is
   performance (VM startup time) over anonymity. This of course should
   be done in a way not compromising security (think - cryptography).

Ok.

So, instead, I would suggest preparing some "blacklist" of packages
known of creating unique initial data, which should really be unique
for every VM.

This list would be probably different for Whonix and
non-Whonix templates.

On first thought, I think if anything, Whonix would just extend that
list. Not use fewer. But even then, for now I cannot imagine a use case
where a service better runs in non-Whonix but not Whonix VMs or vice versa.

And then clean such data (it should be simple
to create some service running at TemplateVM shutdown, which would
remove/sanitize listed files).

Hm. Cleaning sounds difficult, package specific, maintenance burdensome,
test intensive, fragile. But maybe there is no other way.

If the unwanted files aren't created by post installation scripts, we
are better off adding a systemd drop-in file to prevent startup of the
service when run withing a TemplateVM.

This isn't generic solution (neither
is yours), but IMHO have much less potential for breaking random
things (there are over 2k packages in template and probably only few
of them require such treatment).

But there are X how many k packages available from installation from
upstream (Debian/Fedora) repositories. And the user is likely to install
problematic ones in the template. For example, openssh-server seems like
a very problematic case.

Another solution would be to move such (selected) data to /rw
partition (either using symlinks or bind-mounts). Then each VM would
have its own, persistent copy. But this would be probably much more
service-specific and require more work.

Yes.

Btw, there is such a bind-mount script. @nrgaway wrote it.
https://github.com/Whonix/qubes-whonix/blob/master/usr/lib/qubes-whonix/bind-directories
Not sure why it was committed to qubes-whonix rather than Qubes. I had
in mind to make it configurable and to to upstream it to Qubes.

[marmarek]

1. Creating initial data is in many cases desirable. Remember that
   rootfs changes are not persistent in AppVMs, so when some service
   needs some more work at first startup, it will be done at each VM
   startup. Results in bad performance, long VM startup, waste of CPU
   power and high IO (especially important on laptops, with SSD). Some
   example of such service is dnf-makecache - it creates cache of
   metadata from every installed and available package, takes a lot of
   time (minutes, or even tens minutes, 100% CPU). This one we have
   specifically
   disabled,
   but this isn't the only such service. And BTW we'll probably need to
   reenable dnf in Fedora 22 template...

Maybe there is a contradiction here or lack of understanding on my side.
On one hand, officially distributed image builds are created in chroot
and never booted [systemd services never run] before they get deployed.
That is considered fine by both of us.

We have our template builder which can apply some post-installation
fixes. Granted, currently there is almost no such fixes. AFAIR only
setting locale, because by default it isn't set at all (defaults to
"C"), which prevent gnome-terminal from working.

If the user boots the template and installs packages, you prefer the
services (systemd...) to actually start. On the other hand, I suggest
these should be handled same as if in chroot.

But imagine a user just downloaded a fresh TemplateVM and created an
AppVM based on it. Then just starts the AppVM without ever starting the
TemplateVM beforehand. Doesn't that AppVM then miss everything you
mentioned?

Not really - because the template is started during package installation

• mostly to retrieve applications list and icons, but also to create
  initial data (to speed up later AppVMs boot).
  And that's also the reason why some distributions provide some
  post-installation configuration tools (firstboot/initial-setup in case
  of Fedora) - this tool is specifically run in just installed system, not
  during installation (using chroot).

1. Starting TemplateVM is not
   always only about installing packages. Some users want to make
   system-wide changes. And some of them, especially when using GUI
   tools, require running (dbus) services which applies the changes.
   This is also about GUI updaters (Graphical update of Fedora 21 template reports no updates available #982), which we want to resurrect
   one day.

For system-wide changes:
What I had in mind would run the terminal, file manager, etc. already
chrooted into the template image.

GUI tools, dbus in chroot:
I don't know if there would be a sane solution to just selectively and
dynamically start what the user requires. A quick search implies, that
GUI and dbus can be used within chroot.

Not sure if that's fully true. Calling dbus service requires that the
service is started (or already running). In case of system services
(like PackageKit, or systemd-timedated) this means that dbus system
daemon (first required service) forward a start request to systemd
(second running service), which starts the service (third service).

What I am suggesting is basically just to start in the TemplateVM what
the user wishes, not all the system default. I.e. run in chroot,
apt-get, Apper with its dbus dependency, nautilus etc. As opposed to
starting the whole system default. That would for example exclude
openssh-server from starting and creating its private keys.

Hmm, I don't think even that would be enough. During template build
process we specifically disable starting services:
https://github.com/QubesOS/qubes-builder-debian/blob/master/template_debian/distribution.sh#L91-L120
Without this code, during template build a bunch of services were
started, for example preventing unmounting the image.

This is mostly Debian-specific, because Debian policy is to start the
service (make it ready to use) just after installation. In Fedora
services are not started at installation (idea: user first may/should
configure it).

1. Such approach would make the whole template
   update/installation actions really fragile. Need to track upstream
   changes, probably apply some hacks/workarounds for things not working
   in chroot.

I don't think so. This is same as 1. If we consider package installation
during build in chroot fine, then package installation by the user in
chroot also should be fine? I've never came across a package that is not
installable inside chroot. I guess that would be considered as a release
critical bug by upstream.

I have in mind mostly starting/running the process (all the surrounding
things, like proxy configuration, GUI tools (including terminal emulator
itself) etc. Not package installation itself, which indeed should just
work.

So, instead, I would suggest preparing some "blacklist" of packages
known of creating unique initial data, which should really be unique
for every VM.

This list would be probably different for Whonix and
non-Whonix templates.

On first thought, I think if anything, Whonix would just extend that
list.

Yes, that was my thought.

Not use fewer. But even then, for now I cannot imagine a use case
where a service better runs in non-Whonix but not Whonix VMs or vice versa.

I can't think of such example either for now. But I have in mind some
applications with first start wizards etc.

And then clean such data (it should be simple
to create some service running at TemplateVM shutdown, which would
remove/sanitize listed files).

Hm. Cleaning sounds difficult, package specific, maintenance burdensome,
test intensive, fragile. But maybe there is no other way.

If the unwanted files aren't created by post installation scripts, we
are better off adding a systemd drop-in file to prevent startup of the
service when run withing a TemplateVM.

Yes. And consider it either Qubes-wide, or only for Whonix. Case by
case.

This isn't generic solution (neither
is yours), but IMHO have much less potential for breaking random
things (there are over 2k packages in template and probably only few
of them require such treatment).

But there are X how many k packages available from installation from
upstream (Debian/Fedora) repositories. And the user is likely to install
problematic ones in the template. For example, openssh-server seems like
a very problematic case.

Just double checked - currently chroot doesn't prevent host key
generation on openssh-server package install. See here:
https://gist.github.com/marmarek/b227dcff130862fa5b48

Another solution would be to move such (selected) data to /rw
partition (either using symlinks or bind-mounts). Then each VM would
have its own, persistent copy. But this would be probably much more
service-specific and require more work.

Yes.

Btw, there is such a bind-mount script. @nrgaway wrote it.
https://github.com/Whonix/qubes-whonix/blob/master/usr/lib/qubes-whonix/bind-directories
Not sure why it was committed to qubes-whonix rather than Qubes. I had
in mind to make it configurable and to to upstream it to Qubes.

Good idea. Making it configurable seems to be quite easy, as in the
current shape it already have "configuration", just embedded instead of
separate file. Then Whonix could just insert configuration for it (using
drop-ins approach).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Good idea. Making it configurable seems to be quite easy, as in the current shape it already have "configuration", just embedded instead of separate file. Then Whonix could just insert configuration for it (using drop-ins approach).

Yes. (Added this to my plate - https://phabricator.whonix.org/T414 - but I not mind about anyone jumping in. A good, easy task for contributors to get started btw.)

[adrelanos]

Closing this as wontfix then.

Thanks a lot for all the explanation and your consideration!