Create `qubes-upgrade-vm` package for Debian templates

[marmarek]

Qubes version upgrade requires upgrading all the templates, which require switching package repository (for example R3.0->R3.1). For Fedora we have a package qubes-upgrade-vm, which ships new repository definitions including signing key (package in R2 ships R3.0 repositories, package in R3.0 - R3.1 repositories). This way template upgrade require installing this package, then perform standard template update. At the end of this process, qubes-upgrade-vm will be automatically removed.

In Fedora that automatic package removal at the end of upgrade is done using Obsoletes: rpm dependency, from core-agent package. For example

1. In R3.0 repository, qubes-upgrade-vm (with contains repositories of R3.1) has version 3.0
2. In R3.1, package qubes-core-vm (qubes-core-agent in Debian) contains Obsoletes: qubes-upgrade-vm < 3.1

This way, qubes-upgrade-vm-3.0 will be automatically removed, but later, when upgrading R3.1 to the next major release it would be possible to install qubes-upgrade-vm-3.1 (which is uploaded to the repository at the time of first release candidate).

It would be nice to have the same for Debian template, to ease and unify its upgrade.
Does Debian packages allows something equivalent to this Obsoletes: mechanism, to automatically remove the package, but allow later installing newer its version? @adrelanos

[adrelanos]

Does Debian packages allows something equivalent to this Obsoletes: mechanism, to automatically remove the package, but allow later installing newer its version? @adrelanos

I haven't seen packages automatically removed from Debian ever. As soon as no other package depends on that package anymore and if the package is not marked as manually installed, it will be removed during the next manual run of sudo apt-get autoremove.

The closest thing coming to my mind is Replaces: in debian/control. Debian usually transforms the to-be-obsoleted package into an empty transitional package and uploads a new version to the repository. During the next release [in terms jessie -> stretch] that transitional package is removed. That cruft than would not hurt anyone still not having it uninstalled.

Ubuntu has do-release-upgrade, but Debian has to my knowledge no such tool. (What is Debian's equivalent of do-release-upgrade?)

• do-release-upgrade source: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/ubuntu-release-upgrader/wily/files
• do-release-upgrade rationale: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/ubuntu-release-upgrader/wily/view/head:/README

[adrelanos]

Does Debian packages allows something equivalent to this Obsoletes: mechanism, to automatically remove the package, but allow later installing newer its version? @adrelanos

I haven't seen packages automatically removed from Debian ever. As soon as no other package depends on that package anymore and if the package is not marked as manually installed, it will be removed during the next manual run of sudo apt-get autoremove.

The closest thing coming to my mind is Replaces: in debian/control. Debian usually transforms the to-be-obsoleted package into an empty transitional package and uploads a new version to the repository. During the next release [in terms jessie -> stretch] that transitional package is removed. That cruft than would not hurt anyone still not having it uninstalled.

Ubuntu has do-release-upgrade, but Debian has to my knowledge no such tool. (What is Debian's equivalent of do-release-upgrade?)

• do-release-upgrade source: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/ubuntu-release-upgrader/wily/files
• do-release-upgrade rationale: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/ubuntu-release-upgrader/wily/view/head:/README

[marmarek]

Ok, it may be also a tool like do-release-upgrade, maybe called qubes-template-upgrade? Then it would be called like this:

# qubes-template-upgrade
Please choose target release version, available options:
3.0
3.1
4.0
# qubes-template-upgrade 3.1
Performing template upgrade to Qubes 3.1...
Setting up new sources.list...
Running apt-get update
(...)
Running apt-get -V dist-upgrade
(...)
Cleaning up sources.list...
Done.

What do you think?

[marmarek]

Ok, it may be also a tool like do-release-upgrade, maybe called qubes-template-upgrade? Then it would be called like this:

# qubes-template-upgrade
Please choose target release version, available options:
3.0
3.1
4.0
# qubes-template-upgrade 3.1
Performing template upgrade to Qubes 3.1...
Setting up new sources.list...
Running apt-get update
(...)
Running apt-get -V dist-upgrade
(...)
Cleaning up sources.list...
Done.

What do you think?

[adrelanos]

Yes. A tool. Good idea. I am just worried such a tool would easily
expand into loads of new issues that cannot be foreseen at this point.

• One thing. For example documentation would recommend to apt-get update
  and dist-upgrade first. So qubes-template-upgrade gets an up to date
  list about what other Qubes release there are. Then running
  qubes-template-upgrade. dist-upgrade before release upgrade is a good
  practice for stability anyhow.
• qubes-template-upgrade should not confuse users "oh 4.0 sounds best"
  and then accidentally upgrade to an alpha version. So
  qubes-template-upgrade needs to somehow explain which versions are
  stable, alpha, beta, w/e.
• By default qubes-template-upgrade should only show the latest
  recommended stable version. No RC, beta, etc. For simplicity. Otherwise
  have some message "no stable Qubes release upgrade available".
  qubes-template-upgrade --some-switch could show RC, beta, etc. versions.
• Would we need any other features of the command line
  do-release-upgrade version or any other of its features?

I wouldn't make this a separate package. And not automated installing
and uninstalling a package since this is more complicated.

Not sure about the name. qubes-template-upgrade is not bad. But we are
doing a release upgrade here. Users should not be confused about apt-get
update vs apt-get dist-ugprade vs qubes-template-upgrade.

/cc @bnvk

[adrelanos]

Yes. A tool. Good idea. I am just worried such a tool would easily
expand into loads of new issues that cannot be foreseen at this point.

• One thing. For example documentation would recommend to apt-get update
  and dist-upgrade first. So qubes-template-upgrade gets an up to date
  list about what other Qubes release there are. Then running
  qubes-template-upgrade. dist-upgrade before release upgrade is a good
  practice for stability anyhow.
• qubes-template-upgrade should not confuse users "oh 4.0 sounds best"
  and then accidentally upgrade to an alpha version. So
  qubes-template-upgrade needs to somehow explain which versions are
  stable, alpha, beta, w/e.
• By default qubes-template-upgrade should only show the latest
  recommended stable version. No RC, beta, etc. For simplicity. Otherwise
  have some message "no stable Qubes release upgrade available".
  qubes-template-upgrade --some-switch could show RC, beta, etc. versions.
• Would we need any other features of the command line
  do-release-upgrade version or any other of its features?

I wouldn't make this a separate package. And not automated installing
and uninstalling a package since this is more complicated.

Not sure about the name. qubes-template-upgrade is not bad. But we are
doing a release upgrade here. Users should not be confused about apt-get
update vs apt-get dist-ugprade vs qubes-template-upgrade.

/cc @bnvk

[unman]

Why is it necessary to remove the package once it's been installed?

A simple method could be:
R3.1 ships with qubes-upgrade-vm-3.1 (defs for 3.1) or no deb at all.
When R3.2 ready, qubes-upgrade-vm-3.2 added to repo, containing new definitions. (Warnings and caveats built in to package with pre- and post-inst.)
User who wants to upgrade, upgrades that package, then follows normal Debian update/upgrade procedures.
User who doesn't want to upgrade, holds package at existing version.

Then qubes-core-agent-3.2 has Conflicts: qubes-upgrade-vm<3.2, or perhaps Pre-Depends:qubes-upgrade-vm=3.2.

Instead of a new tool, this would fit more naturally with Debian procedures. On the other hand, there's undoubtedly a good reason why there isn't such a package already: it's pretty difficult to get it right.

[unman]

Why is it necessary to remove the package once it's been installed?

A simple method could be:
R3.1 ships with qubes-upgrade-vm-3.1 (defs for 3.1) or no deb at all.
When R3.2 ready, qubes-upgrade-vm-3.2 added to repo, containing new definitions. (Warnings and caveats built in to package with pre- and post-inst.)
User who wants to upgrade, upgrades that package, then follows normal Debian update/upgrade procedures.
User who doesn't want to upgrade, holds package at existing version.

Then qubes-core-agent-3.2 has Conflicts: qubes-upgrade-vm<3.2, or perhaps Pre-Depends:qubes-upgrade-vm=3.2.

Instead of a new tool, this would fit more naturally with Debian procedures. On the other hand, there's undoubtedly a good reason why there isn't such a package already: it's pretty difficult to get it right.

[marmarek]

Why is it necessary to remove the package once it's been installed?

To not have duplicated repository definitions.

[marmarek]

Why is it necessary to remove the package once it's been installed?

To not have duplicated repository definitions.

[unman]

I wasn't clear. My proposal is to move the qubes repo stuff out of qubes-core-agent in to a separate package, installing keys and repo definitions.
When R3.2 is ready, an updated package is placed in the repository. If selected, user will be prompted as to whether they want to upgrade tools. If not, installation aborted and the old package can be placed on hold.
Make qubes-core-agent-3.2 Pre-Depends:qubes-upgrade-vm=3.2. This ensures that new version cant be installed unless the appropriate repositories and keys have been installed.

There's no question of duplicated definitions because each package ships it's own, and the usual Debian versioning controls will apply, and can enforce relevant progression.

If you think that pushing out a new version on an update might be problematic, just make varieties available from the upgrade web-page. User downloads relevant deb, installs and update/upgrade.
This has advantage of fitting the natural upgrade path.

[unman]

I wasn't clear. My proposal is to move the qubes repo stuff out of qubes-core-agent in to a separate package, installing keys and repo definitions.
When R3.2 is ready, an updated package is placed in the repository. If selected, user will be prompted as to whether they want to upgrade tools. If not, installation aborted and the old package can be placed on hold.
Make qubes-core-agent-3.2 Pre-Depends:qubes-upgrade-vm=3.2. This ensures that new version cant be installed unless the appropriate repositories and keys have been installed.

There's no question of duplicated definitions because each package ships it's own, and the usual Debian versioning controls will apply, and can enforce relevant progression.

If you think that pushing out a new version on an update might be problematic, just make varieties available from the upgrade web-page. User downloads relevant deb, installs and update/upgrade.
This has advantage of fitting the natural upgrade path.

[marmarek]

My proposal is to move the qubes repo stuff out of qubes-core-agent in to a separate package, installing keys and repo definitions.

Ah, I see.

When R3.2 is ready, an updated package is placed in the repository. If selected, user will be prompted as to whether they want to upgrade tools. If not, installation aborted and the old package can be placed on hold.

This can be risky. Qubes update consists of both dom0 and VMs update, at the same time. So, if VM update is proposed automatically (even with a separate confirmation), surely someone will do it, without updating dom0, which will most likely lead some some problems - in extreme case unbootable system.

Make qubes-core-agent-3.2 Pre-Depends:qubes-upgrade-vm=3.2. This ensures that new version cant be installed unless the appropriate repositories and keys have been installed.

Not a problem, as qubes-core-agent-3.2 will not be available without 3.2 repository definition (it isn't present in 3.1 repo).

Can you clarify how exactly user would trigger template upgrade?

[marmarek]

My proposal is to move the qubes repo stuff out of qubes-core-agent in to a separate package, installing keys and repo definitions.

Ah, I see.

When R3.2 is ready, an updated package is placed in the repository. If selected, user will be prompted as to whether they want to upgrade tools. If not, installation aborted and the old package can be placed on hold.

This can be risky. Qubes update consists of both dom0 and VMs update, at the same time. So, if VM update is proposed automatically (even with a separate confirmation), surely someone will do it, without updating dom0, which will most likely lead some some problems - in extreme case unbootable system.

Make qubes-core-agent-3.2 Pre-Depends:qubes-upgrade-vm=3.2. This ensures that new version cant be installed unless the appropriate repositories and keys have been installed.

Not a problem, as qubes-core-agent-3.2 will not be available without 3.2 repository definition (it isn't present in 3.1 repo).

Can you clarify how exactly user would trigger template upgrade?

[unman]

Not a problem, as qubes-core-agent-3.2 will not be available without 3.2 repository definition (it isn't present in 3.1 repo).

The ability of users to confound expectations should not be underestimated. It would, of course, be possible to download directly from the repositories.

Can you clarify how exactly user would trigger template upgrade?

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

dpkg -i qubes-upgrade-vm-3.2
apt-get update
apt-get upgrade
apt-get dist-upgrade

[unman]

Not a problem, as qubes-core-agent-3.2 will not be available without 3.2 repository definition (it isn't present in 3.1 repo).

The ability of users to confound expectations should not be underestimated. It would, of course, be possible to download directly from the repositories.

Can you clarify how exactly user would trigger template upgrade?

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

dpkg -i qubes-upgrade-vm-3.2
apt-get update
apt-get upgrade
apt-get dist-upgrade

[marmarek]

On Tue, Jun 07, 2016 at 04:26:57PM -0700, unman wrote:

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

This means user would need to manually verify package signature (which
by default isn't even present in deb packages). So not a good idea,
because some of them would not do that.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Tue, Jun 07, 2016 at 04:26:57PM -0700, unman wrote:

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

This means user would need to manually verify package signature (which
by default isn't even present in deb packages). So not a good idea,
because some of them would not do that.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Marek Marczykowski-Górecki:

On Tue, Jun 07, 2016 at 04:26:57PM -0700, unman wrote:

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

This means user would need to manually verify package signature (which
by default isn't even present in deb packages). So not a good idea,
because some of them would not do that.

If the package is added to the repository the user is currently using, I
see no reason why the package could not be downloaded using apt / yum.
The website could advice the command to download it using apt / yum.

[ Just saying. I however have no opinion on your recent discussion above. ]

[adrelanos]

Marek Marczykowski-Górecki:

On Tue, Jun 07, 2016 at 04:26:57PM -0700, unman wrote:

If not available via a repository, to avoid the issue you identify, the qubes-upgrade-vm packages are perhaps linked on a web page identifying flavours.
User downloads one, transfers to template. Installs that package and then performs normal upgrade process.

This means user would need to manually verify package signature (which
by default isn't even present in deb packages). So not a good idea,
because some of them would not do that.

If the package is added to the repository the user is currently using, I
see no reason why the package could not be downloaded using apt / yum.
The website could advice the command to download it using apt / yum.

[ Just saying. I however have no opinion on your recent discussion above. ]

[unman]

@marmarek
Here's an alternative approach:
Put the new definitions in qubes-update-vm AND qubes-core-agent.
So, qubes-update-vm-3.1.1 is rc1 for 3.2 (for example)
qubes-core-agent.-3.2 has Replaces:qubes-update-vm(<=3.1.2) and Conflicts:qubes-update-vm(<=3.1.2)
This will overwrite the sources on install of qubes-update-vm and remove the file on install of qubes-core-agent, so there's no risk of the update file being accidentally updated.

The only problem with this is that because of the conflict, the flow has to be:

apt-get -o Dpkg::Options::="--force-overwrite" install qubes-update-vm-3.1.1
apt-get update
apt-get upgrade
apt-get dist-upgrade

I doubt anyone is going to do this in error.
The available versions can be kept in repo and referenced from a web page.
This is closest to your original question.

[unman]

@marmarek
Here's an alternative approach:
Put the new definitions in qubes-update-vm AND qubes-core-agent.
So, qubes-update-vm-3.1.1 is rc1 for 3.2 (for example)
qubes-core-agent.-3.2 has Replaces:qubes-update-vm(<=3.1.2) and Conflicts:qubes-update-vm(<=3.1.2)
This will overwrite the sources on install of qubes-update-vm and remove the file on install of qubes-core-agent, so there's no risk of the update file being accidentally updated.

The only problem with this is that because of the conflict, the flow has to be:

apt-get -o Dpkg::Options::="--force-overwrite" install qubes-update-vm-3.1.1
apt-get update
apt-get upgrade
apt-get dist-upgrade

I doubt anyone is going to do this in error.
The available versions can be kept in repo and referenced from a web page.
This is closest to your original question.

[unman]

@marmarek Do you want to see that approach implemented, or do you prefer update tool?

[unman]

@marmarek Do you want to see that approach implemented, or do you prefer update tool?

[marmarek]

Using -o Dpkg::Options:: ... looks scary. So, update tool would be better.

[marmarek]

Using -o Dpkg::Options:: ... looks scary. So, update tool would be better.