builder: longer timeouts and more retries for APT to avoid aborted builds due to network issues

[adrelanos]

Qubes OS release

R4.2

Brief summary

Quote @marmarek in QubesOS/updates-status#5605 (comment)

Maybe there should be some (better) retry mechanism? Documentation suggests Acquire::Retries setting defaults to 3, and indeed I see 3 attempts in the log. But maybe it's not enough? Or maybe it fails due to the current tor circuit being broken and retry should use a different one to have a chance (not sure how to do that, or if that's even possible)? Or maybe when it fails on name resolution (log has Something wicked happened resolving 'deb.qubes-os.org:https' (-4 - Non-recoverable failure in name resolution).) it doesn't really retry? I'm quite confused here, because it stops working in the middle of template build, for example installing qubes-core-agent-dom0-updates some lines above worked. And also installing a lot of Debian and Whonix packages earlier worked too...

Steps to reproduce

Build a Debian based Qubes Template.

Expected behavior

Long timeouts and many retries.

Actual behavior

Bails out due to network issues.

• qubes-template-whonix-gateway-17 4.2.0-202504261018 (r4.2) updates-status#5593
• qubes-template-whonix-gateway-17 4.2.0-202504270856 (r4.2) updates-status#5595
• qubes-template-whonix-gateway-17 4.2.0-202504290958 (r4.2) updates-status#5605
• qubes-template-whonix-gateway-17 4.3.0-202504291137 (r4.3) updates-status#5607
• (Corresponding gateway build also failed.)

[adrelanos]

Recommended:

• -o Acquire::Retries=5

At least 5 or higher.

---

Does Qubes Builder v2 support any APT cache? I think that is pretty much required to avoid repeat build failures. How about apt-cacher-ng (or any other apt cache of your choice)?

• -o Acquire::http::Proxy=http://127.0.0.1:3142
• -o Acquire::https::Proxy=http://127.0.0.1:3142
• -o Acquire::tor::Proxy=http://127.0.0.1:3142

Related:

• qubes-builder make template-github curl broken - wrong environment variable handling #6737

---

Generally recommended for security and reliability:

• -o APT::Update::Error-Mode=any

References:

• https://forums.whonix.org/t/debian-bullseye-apt-get-error-on-any/12068
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594813
• https://salsa.debian.org/apt-team/apt/-/commit/c7123bea6a8dc2c9e327ce41ddfc25e29f1bb145

---

Reducing download of unneeded files during build, improve build speed and reduce risk from network failures:

• -o Acquire::Languages=none
• -o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false

References:

• https://forums.whonix.org/t/speeding-up-apt-update-with-acquire-languages-none-and-contents-deb-defaultenabled-false-its-so-much-faster/8894/2

[marmarek]

Build a Debian based Qubes Template.

Interestingly, it appears the issue affects only Whonix template builds. Kicksecure template built just fine in similar time, yet none of 8 Whonix builds did.

Recommended:

* `-o Acquire::Retries=5`

At least 5 or higher.

Currently there is explicit -o Acquire::Retries=3

Does Qubes Builder v2 support any APT cache?

No, it doesn't, kinda intentionally (to not add cross-template attack vector - for example one template build compromising the cache service to feed another template build wrong/old packages).

Generally recommended for security and reliability:

* `-o APT::Update::Error-Mode=any`

Would that improve reliability of retrying?

Reducing download of unneeded files during build, improve build speed and reduce risk from network failures:

* `-o Acquire::Languages=none`

* `-o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false`

What is the impact of those options on the target system? Will it have some packages skipped?

[adrelanos]

Build a Debian based Qubes Template.

Interestingly, it appears the issue affects only Whonix template builds. Kicksecure template built just fine in similar time, yet none of 8 Whonix builds did.

That is very strange indeed.

Does Qubes Builder v2 support any APT cache?

No, it doesn't, kinda intentionally (to not add cross-template attack vector - for example one template build compromising the cache service to feed another template build wrong/old packages).

Generally recommended for security and reliability:

* `-o APT::Update::Error-Mode=any`

Would that improve reliability of retrying?

No. But it might prevent reproducibility issues in corner cases and/or prevent security upgrades from being installed during the build process. (Without this option, apt-get update will exit 0, failing repositories such as security.debian.org are ignored.) Highly recommended but arguably unrelated.

Reducing download of unneeded files during build, improve build speed and reduce risk from network failures:

* `-o Acquire::Languages=none`

* `-o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false`

What is the impact of those options on the target system?

I didn't observe any differences. There might be differences in theory if building with a non-English locale inside the chroot, which I think should be unsupported.

APT sources are thrown away at the end of the build process anyhow as in apt-get clean / apt-get autoclean?

• They should be for reproducibility, to save space, and avoid shipping Templates with outdated APT package lists.
• If yes: No differences.
• If no:
  • The difference would be translations for non-English locales for Debian package information (such as apt-cache show package-name) to be only available in English, not other languages. This "issue" will be remedied after the running apt-get update.
  • apt-file list package cache would be empty. This cache will be populated after the user runs apt-get update.

These command line options will not have any effect after starting to use the template. Because it's just apt-get command line options during the build process and not an APT config file. Since the user starts with a default APT configuration file, language files and package content files will be downloaded as per Debian default.

Setting -o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false permanently using an APT configuration file might be considered in a different ticket. It's a default inside Kicksecure. Advantage: Faster apt-get update and less traffic. Disadvantage: Those wishing to use apt-file need to apply Kicksecure apt-file instructions. (Almost the same for command-not-found.)

As for -o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false, no users have asked about this ever.

Will it have some packages skipped?

No. This option only influences apt-get update. Does not influence apt-get install. Any missing dependency will still error out as they should.

[adrelanos]

Does Qubes Builder v2 support any APT cache?

No, it doesn't, kinda intentionally (to not add cross-template attack vector - for example one template build compromising the cache service to feed another template build wrong/old packages).

I think apt-cacher-ng would be OK in the threat model. The network is untrusted. Operating system (Debian) mirrors are untrusted. APT is secured by apt-secure (the usual digital software signature verification mechanism) for a long time already.

Anything received by APT, directly from the mirror or apt-cacher-ng would be subject to apt-secure.

From perspective of the Template build, it should not matter if packages are coming directly form a:

• remote mirror (untrusted) or;
• local apt-cacher-ng mirror (untrusted)?