Mechanism for triggering template build

[marmarek]

Right now template build require direct access to build machine. This is bad for various reasons, including:

• lack of transparency
• inability to delegate it

We've using a better mechanism for individual packages for some time. It consists of:

• a script that build and upload to current-testing a package that got new signed version tag
• a script that process signed comment on github to migrate package from current-testing to current (#2573)

This works well for normal packages, lets do something similar for templates.
The problem here is there is no natural place for template version tag - template consists of more than just linux-template-builder - in many cases we need to build new template after just updating individual packages (which doesn't involve any change in linux-template-builder).

I propose using something similar to #2573 for triggering the build too. The work flow would be:

• template maintainer open new issue (in updates-status repository? or here?) with inline-signed command to build the template
• appropriate script verify the signature and proceed to build the template (using make build-template-in-dispvm)
• when build succeed, result is reported to https://github.com/QubesOS/updates-status/issues repository and template is uploaded to online repository
• build failure would be also reported, in https://github.com/QubesOS/build-issues/issues

Open questions:

1. How build trigger command should look like (what we need there)? My current idea:

    build-template QUBES_VERSION TEMPLATE TIMESTAMP
   

   For example: build-template r4.0 fc28+minimal 201805171636

2. Do we need templates-itl-testing and templates-community-testing repositories and upload templates there first? And the use similar workflow to migrate them later to final repositories?

3. Replay protection - in the example above I've added TIMESTAMP (which will be used as %release part of template version) exactly for this reason. The script would verify if it isn't too far in the past. Any better idea?

4. Template package version is pretty meaningless right now (it's template builder version (but not builder plugin - containing most actual building scripts) + timestamp). Maybe we could use this occasion to change this too? For example specify template version manually too? Or build it based on qubes version (3.2.x for templates for Qubes 3.2 etc)?

This all would for example allow @adrelanos to trigger new Whonix template build or @fepitre for Fedora and CentOS, or @ptitdoc for Archlinux, or @unman for Debian.

cc @andrewdavidwong

[andrewdavidwong]

Do we need templates-itl-testing and templates-community-testing repositories and upload templates there first? And the use similar workflow to migrate them later to final repositories?

This would probably be a good idea.

[andrewdavidwong]

Do we need templates-itl-testing and templates-community-testing repositories and upload templates there first? And the use similar workflow to migrate them later to final repositories?

This would probably be a good idea.

[marmarek]

@adrelanos see above questions, any opinion? Most of required changes are already done, I'd like to use it for Whonix 14 build (#4016).

[marmarek]

@adrelanos see above questions, any opinion? Most of required changes are already done, I'd like to use it for Whonix 14 build (#4016).

[marmarek]

The command to create the timestamp is...?

date --utc +%Y%m%d%H%M

Yes.

Only one command per message?

For now yes.

Is this safe against attacks, i.e. someone copying and pasting any comments? I.e. a replay attack. The signature is valid regardless how often it is posted.

This is why timestamp is included in the command. It's valid for one hour. There is also protection refusing to build template which already exists (exact version, including requested timestamp), so it's only possible to replay command (for one hour) for a failed build. Which I consider a feature (retry a build after intermittent network problem caused by Tor).

[marmarek]

The command to create the timestamp is...?

date --utc +%Y%m%d%H%M

Yes.

Only one command per message?

For now yes.

Is this safe against attacks, i.e. someone copying and pasting any comments? I.e. a replay attack. The signature is valid regardless how often it is posted.

This is why timestamp is included in the command. It's valid for one hour. There is also protection refusing to build template which already exists (exact version, including requested timestamp), so it's only possible to replay command (for one hour) for a failed build. Which I consider a feature (retry a build after intermittent network problem caused by Tor).

[marmarek]

BTW I see gpg2 supports expiration time for signatures (--default-sig-expire). But I haven't tried it and not sure if that's a good idea.

[marmarek]

BTW I see gpg2 supports expiration time for signatures (--default-sig-expire). But I haven't tried it and not sure if that's a good idea.

[adrelanos]

QubesOS/updates-status#566 (comment) - Did I do something wrong?

[adrelanos]

QubesOS/updates-status#566 (comment) - Did I do something wrong?

[marmarek]

Looks fine, let me check the logs...

[marmarek]

Looks fine, let me check the logs...

[marmarek]

Ok, I found the problem - I'm using subkey ID not primary key ID for access control, but actually listed primary key ID in configuration. Will fix in a moment

[marmarek]

Ok, I found the problem - I'm using subkey ID not primary key ID for access control, but actually listed primary key ID in configuration. Will fix in a moment

[marmarek]

Ok, retry now (copy and paste the same comment again should be ok).

[marmarek]

Ok, retry now (copy and paste the same comment again should be ok).

[adrelanos]

Issue for the build was created this time.

---

https://github.com/marmarek/qubes-builder-github/blob/master/lib/functions.sh

        eval "$local_signer"="$(grep -Po \
            '^\[GNUPG:\] VALIDSIG (([0-9A-F-]+ ){9}|)\K([0-9A-F]*)' \
            "$tmpdir/gpg-status")"

Not exactly same as #4070 but couldn't we better avoid any eval here?

Could that be a bash script? Would a bash array help avoid the eval?

[adrelanos]

Issue for the build was created this time.

---

https://github.com/marmarek/qubes-builder-github/blob/master/lib/functions.sh

        eval "$local_signer"="$(grep -Po \
            '^\[GNUPG:\] VALIDSIG (([0-9A-F-]+ ){9}|)\K([0-9A-F]*)' \
            "$tmpdir/gpg-status")"

Not exactly same as #4070 but couldn't we better avoid any eval here?

Could that be a bash script? Would a bash array help avoid the eval?

[marmarek]

eval here is is to assign to a variable named by $local_signer, how to do that? Alternatively it could write keyid to a file named by that variable... The point here is we need to return two things from that function - verified command and a key ID used to sign it.

[marmarek]

eval here is is to assign to a variable named by $local_signer, how to do that? Alternatively it could write keyid to a file named by that variable... The point here is we need to return two things from that function - verified command and a key ID used to sign it.

[adrelanos]

File seems better. Otherwise the eval risks to evaluate anything form gpg status file output.

---

QubesOS/updates-status#576

Minor issue - duplicate comment:

Package for fc23 was built (build log) and uploaded to templates-community-testing repository

[adrelanos]

File seems better. Otherwise the eval risks to evaluate anything form gpg status file output.

---

QubesOS/updates-status#576

Minor issue - duplicate comment:

Package for fc23 was built (build log) and uploaded to templates-community-testing repository

[marmarek]

File seems better. Otherwise the eval risks to evaluate anything form gpg status file output.

The function strictly grep for [0-9A-F]*, but will convert to a file anyway.

Minor issue - duplicate comment:

That was me.

[marmarek]

File seems better. Otherwise the eval risks to evaluate anything form gpg status file output.

The function strictly grep for [0-9A-F]*, but will convert to a file anyway.

Minor issue - duplicate comment:

That was me.

[adrelanos]

I've installed from qubes-dom0-unstable and wondered about stale package versions.

So qubes-dom0-unstable is different from templates-community-testing?

Package for fc25 was built (build log) and uploaded to templates-community-testing repository

dom0 installation command would be helpful to encourage testing.

[adrelanos]

I've installed from qubes-dom0-unstable and wondered about stale package versions.

So qubes-dom0-unstable is different from templates-community-testing?

Package for fc25 was built (build log) and uploaded to templates-community-testing repository

dom0 installation command would be helpful to encourage testing.

[adrelanos]

For example for QubesOS/updates-status#581 what is the installation command?

sudo qubes-dom0-update --enablerepo=templates-community-testing qubes-template-whonix-gw-14 says repository not found.

[adrelanos]

For example for QubesOS/updates-status#581 what is the installation command?

sudo qubes-dom0-update --enablerepo=templates-community-testing qubes-template-whonix-gw-14 says repository not found.

[marmarek]

The command is correct, but template definition isn't there yet. See this:

The template will be uploaded to appropriate templates testing repository. For now you need to create repository definition manually (copy existing one and append "-testing" to the name and URL).

[marmarek]

The command is correct, but template definition isn't there yet. See this:

The template will be uploaded to appropriate templates testing repository. For now you need to create repository definition manually (copy existing one and append "-testing" to the name and URL).

[marmarek]

Oh and salt have hardcoded templates-community repo name: https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/qvm/template-whonix-gw.sls#L19
So, testing this with salt require changing that line (it's /srv/formulas/base/virtual-machines-formula/qvm/template-whonix-*.sls).

[marmarek]

Oh and salt have hardcoded templates-community repo name: https://github.com/QubesOS/qubes-mgmt-salt-dom0-virtual-machines/blob/master/qvm/template-whonix-gw.sls#L19
So, testing this with salt require changing that line (it's /srv/formulas/base/virtual-machines-formula/qvm/template-whonix-*.sls).

[marmarek]

Ok, qubes-release-4.0-4 contains templates testing repositories. Note I've also added metalinks there (for automatic mirror selection) - please let me know if you notice any problems with that.

[marmarek]

Ok, qubes-release-4.0-4 contains templates testing repositories. Note I've also added metalinks there (for automatic mirror selection) - please let me know if you notice any problems with that.

[marmarek]

For 3.2, it's already in qubes-release-3.2-2

[marmarek]

For 3.2, it's already in qubes-release-3.2-2

[adrelanos]

Due to #4095 Whonix templates need to be rebuild. Is it possible to build from Qubes testing rather than Qubes stable repository?

[adrelanos]

Due to #4095 Whonix templates need to be rebuild. Is it possible to build from Qubes testing rather than Qubes stable repository?

[marmarek]

Is it possible to build from Qubes testing rather than Qubes stable repository?

Yes, add USE_QUBES_REPO_TESTING=1 to https://github.com/QubesOS/qubes-template-configs/blob/master/R4.0/templates-community/whonix-14.conf

[marmarek]

Is it possible to build from Qubes testing rather than Qubes stable repository?

Yes, add USE_QUBES_REPO_TESTING=1 to https://github.com/QubesOS/qubes-template-configs/blob/master/R4.0/templates-community/whonix-14.conf

[adrelanos]

QubesOS/updates-status#583 says it's in current-testing. But is it also in unstable? Does it go into unstable beforehand and stay there?

[adrelanos]

QubesOS/updates-status#583 says it's in current-testing. But is it also in unstable? Does it go into unstable beforehand and stay there?

[marmarek]

Hm? I see only version "0.1" in unstable.

[marmarek]

Hm? I see only version "0.1" in unstable.

[adrelanos]

Ok, I assume packages are usually not uploaded to unstable. Good to know.

May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"?

[adrelanos]

Ok, I assume packages are usually not uploaded to unstable. Good to know.

May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"?

[marmarek]

Ok, I assume packages are usually not uploaded to unstable. Good to know.

Yes, unstable repo is only for early testing versions, not suitable for broader testing. I'd say "developers only" in your repository layout.

May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"?

Any specific reason for that? Package manager picks newest available version. Also, unstable repository is disabled by default and no one really should have it enabled permanently (unless want a really unstable system) - this repository may include potentially known-broken packages.
Generally we do not remove packages from repositories.

[marmarek]

Ok, I assume packages are usually not uploaded to unstable. Good to know.

Yes, unstable repo is only for early testing versions, not suitable for broader testing. I'd say "developers only" in your repository layout.

May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"?

Any specific reason for that? Package manager picks newest available version. Also, unstable repository is disabled by default and no one really should have it enabled permanently (unless want a really unstable system) - this repository may include potentially known-broken packages.
Generally we do not remove packages from repositories.

[adrelanos]

Marek Marczykowski-Górecki:

> Ok, I assume packages are usually not uploaded to unstable. Good to know. Yes, unstable repo is only for early testing versions, not suitable for broader testing. I'd say "developers only" in your repository layout.

I see.

> May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"? Any specific reason for that?

Not not sure if a strong enough reason. When even developers / advanced users get confused, maybe it's too difficult. :) https://www.whonix.org/wiki/Qubes/Install/Testing still had: sudo qubes-dom0-update --enablerepo=qubes-dom0-unstable qubes-core-admin-addon-whonix Since I expected all packages go through unstable -> testing -> stable. Another reason: in case instructions once contain "--enablerepo=qubes-dom0-unstable" and later packages are uploaded straight to testing while skipping unstable, following documentation can certainly lead to strange results, this can create confusion which is difficult to sort out. It's how Debian works. sid (unstable) -> testing -> stable. Qubes unstable (maybe) | testing -> stable I find unexpected, confusing.

[adrelanos]

Marek Marczykowski-Górecki:

> Ok, I assume packages are usually not uploaded to unstable. Good to know. Yes, unstable repo is only for early testing versions, not suitable for broader testing. I'd say "developers only" in your repository layout.

I see.

> May I recommend to remove the unstable version since it's lower than testing version? Perhaps automated "if version in unstable lower than in testing or stable, remove from unstable"? Any specific reason for that?

Not not sure if a strong enough reason. When even developers / advanced users get confused, maybe it's too difficult. :) https://www.whonix.org/wiki/Qubes/Install/Testing still had: sudo qubes-dom0-update --enablerepo=qubes-dom0-unstable qubes-core-admin-addon-whonix Since I expected all packages go through unstable -> testing -> stable. Another reason: in case instructions once contain "--enablerepo=qubes-dom0-unstable" and later packages are uploaded straight to testing while skipping unstable, following documentation can certainly lead to strange results, this can create confusion which is difficult to sort out. It's how Debian works. sid (unstable) -> testing -> stable. Qubes unstable (maybe) | testing -> stable I find unexpected, confusing.

[adrelanos]

Me and testers also confused installing and old version of testers-only Qubes-Whonix from unstable rather than a newer one from Qubes testing earlier. That resulted in a lot not quickly accountable confusion and lost time.

[adrelanos]

Me and testers also confused installing and old version of testers-only Qubes-Whonix from unstable rather than a newer one from Qubes testing earlier. That resulted in a lot not quickly accountable confusion and lost time.

[marmarek]

Here is canonical documentation about testing: https://www.qubes-os.org/doc/testing/
Both dom0 and VM pages contains explanation of testing repositories:

Testing repositories

There are three Qubes dom0 testing repositories:

    qubes-dom0-current-testing – testing packages that will eventually land in the stable (current) repository
    qubes-dom0-security-testing – a subset of qubes-dom0-current-testing that contains packages that qualify as security fixes
    qubes-dom0-unstable – packages that are not intended to land in the stable (qubes-dom0-current) repository; mostly experimental debugging packages

Since we have templates testing repositories now, unstable repository is no longer used against its purpose.

[marmarek]

Here is canonical documentation about testing: https://www.qubes-os.org/doc/testing/
Both dom0 and VM pages contains explanation of testing repositories:

Testing repositories

There are three Qubes dom0 testing repositories:

    qubes-dom0-current-testing – testing packages that will eventually land in the stable (current) repository
    qubes-dom0-security-testing – a subset of qubes-dom0-current-testing that contains packages that qualify as security fixes
    qubes-dom0-unstable – packages that are not intended to land in the stable (qubes-dom0-current) repository; mostly experimental debugging packages

Since we have templates testing repositories now, unstable repository is no longer used against its purpose.

[adrelanos]

Due to frequent flaky network related build failures, could you add please an auto-retry loop of 3 to 5 attempts or so?

[adrelanos]

Due to frequent flaky network related build failures, could you add please an auto-retry loop of 3 to 5 attempts or so?

[adrelanos]

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

qubes-template-whonix-gw-14 4.0.1-201807190624 is being downloaded.

Upload failure not detected?

Out of disk space?

https://yum.qubes-os.org/r4.0/templates-community-testing/rpm/ has:

• qubes-template-whonix-gw-14-4.0.1-2018 07130946.noarch.rpm
• qubes-template-whonix-gw-14-4.0.1-2018 07171801.noarch.rpm
• qubes-template-whonix-ws-14-4.0.1-2018 07131452.noarch.rpm
• qubes-template-whonix-ws-14-4.0.1-2018 07171801.noarch.rpm

By the way, all older templates can all be purged to save disk space.

[adrelanos]

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

qubes-template-whonix-gw-14 4.0.1-201807190624 is being downloaded.

Upload failure not detected?

Out of disk space?

https://yum.qubes-os.org/r4.0/templates-community-testing/rpm/ has:

• qubes-template-whonix-gw-14-4.0.1-2018 07130946.noarch.rpm
• qubes-template-whonix-gw-14-4.0.1-2018 07171801.noarch.rpm
• qubes-template-whonix-ws-14-4.0.1-2018 07131452.noarch.rpm
• qubes-template-whonix-ws-14-4.0.1-2018 07171801.noarch.rpm

By the way, all older templates can all be purged to save disk space.

[marmarek]

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

It is, for 3.2 (not 4.0), here: https://yum.qubes-os.org/r3.2/templates-community-testing/rpm/

[marmarek]

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

It is, for 3.2 (not 4.0), here: https://yum.qubes-os.org/r3.2/templates-community-testing/rpm/

[adrelanos]

QubesOS/updates-status#572 (comment)

Upload core-agent-linux ec251da5d8e6ed5544e91d92695a41f4d86d43d5 r4.0 current repo

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

I guess such a notification if upload actually happened could save a some time in future in case it fails for some reason.

[adrelanos]

QubesOS/updates-status#572 (comment)

Upload core-agent-linux ec251da5d8e6ed5544e91d92695a41f4d86d43d5 r4.0 current repo

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

I guess such a notification if upload actually happened could save a some time in future in case it fails for some reason.

[marmarek]

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

Sort of. This is because it wasn't uploaded. Because there was already newer package: QubesOS/updates-status#593

[marmarek]

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

Sort of. This is because it wasn't uploaded. Because there was already newer package: QubesOS/updates-status#593