Make qubes-builder working on Debian host

[marmarek]

Currently qubes-builder works only on Fedora based system. The idea of using chroot for actual package build should make it easy for running it on any distribution.
This is required for adding support for Qubes in Travis-CI, or Debian Reproducible Builds testing rig.

[akuckartz]

👍

[akuckartz]

👍

[adrelanos]

Marek Marczykowski-Górecki:

Currently qubes-builder works only on Fedora based system. The idea of using chroot for actual package build should make it easy for running it on any distribution.
This is required for adding support for Qubes in Travis-CI, or Debian Reproducible Builds testing rig.

---

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#1907

Working on Debian host may be a good.

However, Travis-CI may be very difficult, since Ubuntu based. I had once
working for Whonix. Too much if Debian else Ubuntu code. Too many
package version differences. Another problem was that Travis-CI did not
support certain system near operations. Since it is already running
inside a VM container it is very limited. In OpenVZ, which makes it even
more limited. Stuff like key generation was very low or even failed,
since there is no entropy. So you need workarounds for that also. I
highly recommend staying away from Ubuntu anything and use a Debian
based CI.

[adrelanos]

Marek Marczykowski-Górecki:

Currently qubes-builder works only on Fedora based system. The idea of using chroot for actual package build should make it easy for running it on any distribution.
This is required for adding support for Qubes in Travis-CI, or Debian Reproducible Builds testing rig.

---

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#1907

Working on Debian host may be a good.

However, Travis-CI may be very difficult, since Ubuntu based. I had once
working for Whonix. Too much if Debian else Ubuntu code. Too many
package version differences. Another problem was that Travis-CI did not
support certain system near operations. Since it is already running
inside a VM container it is very limited. In OpenVZ, which makes it even
more limited. Stuff like key generation was very low or even failed,
since there is no entropy. So you need workarounds for that also. I
highly recommend staying away from Ubuntu anything and use a Debian
based CI.

[marmarek]

However, Travis-CI may be very difficult, since Ubuntu based. I had once working for Whonix. Too much if Debian else Ubuntu code. Too many package version differences.

That shouldn't be a problem, since most of the build is happening inside chroot (with debian/fedora/whatever).

Anyway, that was just an example of use cases.

[marmarek]

However, Travis-CI may be very difficult, since Ubuntu based. I had once working for Whonix. Too much if Debian else Ubuntu code. Too many package version differences.

That shouldn't be a problem, since most of the build is happening inside chroot (with debian/fedora/whatever).

Anyway, that was just an example of use cases.

[marmarek]

Major progress:

1. Building a single component (at least core-agent-linux) on Debian host does work
2. Building a single component (at least core-agent-linux) on Travis-CI does work: https://travis-ci.org/marmarek/qubes-core-agent-linux/builds/123709594

To do:

• check other components
• make template build working on Debian host
• make ISO build working on Debian host
• consider enabling Travis-CI for all the Qubes OS components, for pull requests and/or all pushes.

The last one may be interesting :) But it is only about building, not full test run, unfortunately.

cc @woju @adrelanos

[marmarek]

Major progress:

1. Building a single component (at least core-agent-linux) on Debian host does work
2. Building a single component (at least core-agent-linux) on Travis-CI does work: https://travis-ci.org/marmarek/qubes-core-agent-linux/builds/123709594

To do:

• check other components
• make template build working on Debian host
• make ISO build working on Debian host
• consider enabling Travis-CI for all the Qubes OS components, for pull requests and/or all pushes.

The last one may be interesting :) But it is only about building, not full test run, unfortunately.

cc @woju @adrelanos

[adrelanos]

Cool stuff! Does travis CI support chroot? Do you think it can also do full template builds? Would be interesting to have changes like #1174 build new templates, isos in "all" combinations to see if the change is rock solid breaking nothing.

[adrelanos]

Cool stuff! Does travis CI support chroot? Do you think it can also do full template builds? Would be interesting to have changes like #1174 build new templates, isos in "all" combinations to see if the change is rock solid breaking nothing.

[marmarek]

Regarding chroot - yes, they have images with sudo access, so chroot can be used. Take a look at linked test result - there are two builds: for jessie and fc23. Both done on Ubuntu 14.04 host ;)

Regarding templates and ISO - I'm currently working on it. It should be possible, the only potential problem would be hitting some limit (time, space, etc).

[marmarek]

Regarding chroot - yes, they have images with sudo access, so chroot can be used. Take a look at linked test result - there are two builds: for jessie and fc23. Both done on Ubuntu 14.04 host ;)

Regarding templates and ISO - I'm currently working on it. It should be possible, the only potential problem would be hitting some limit (time, space, etc).

[marmarek]

Ok, for template build it looks we're hitting time limit:
https://travis-ci.org/marmarek/qubes-builder-fedora/builds/124719727

While theoretically time limit is 10 min of no output, build is interrupted exactly after 5 minutes...

[marmarek]

Ok, for template build it looks we're hitting time limit:
https://travis-ci.org/marmarek/qubes-builder-fedora/builds/124719727

While theoretically time limit is 10 min of no output, build is interrupted exactly after 5 minutes...

[adrelanos]

There is a travis_wait command. Does that help?

https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

[adrelanos]

There is a travis_wait command. Does that help?

https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

[marmarek]

Haven't tried yet. But in theory it will not change anything, as the build produce a lot of output. For example here it build running now: https://travis-ci.org/marmarek/qubes-builder-fedora/builds/124785753

[marmarek]

Haven't tried yet. But in theory it will not change anything, as the build produce a lot of output. For example here it build running now: https://travis-ci.org/marmarek/qubes-builder-fedora/builds/124785753

[marmarek]

Interesting - setting language: generic (which force usage of "minimal" trusty image) helps somehow. The build linked above is still running, for over 20 minutes. Hopefully will finish shortly.

There is another problem - build output (log) is too long. This is a minor problem, as it only affects interactive web log view, but it is possible to download full raw log. But we may consider lowering verbosity. @adrelanos what do you think?

[marmarek]

Interesting - setting language: generic (which force usage of "minimal" trusty image) helps somehow. The build linked above is still running, for over 20 minutes. Hopefully will finish shortly.

There is another problem - build output (log) is too long. This is a minor problem, as it only affects interactive web log view, but it is possible to download full raw log. But we may consider lowering verbosity. @adrelanos what do you think?

[marmarek]

This went offtopic. Qubes builder on Debian is done. Travis CI support is a separate task - #1926

[marmarek]

This went offtopic. Qubes builder on Debian is done. Travis CI support is a separate task - #1926

[adrelanos]

I personally prefer the full log and would open the raw log. Actually I always open the raw log anyhow since that works faster in the browser. Please have a look in the build log and search for: declare -p TEMPLATE_FLAVOR_PREFIX Then you see there is some code which feels like accounting for 2/3 of the build log. It's very repetitive. Could this be somehow refactored, run earlier to reduce the log length?

[adrelanos]

I personally prefer the full log and would open the raw log. Actually I always open the raw log anyhow since that works faster in the browser. Please have a look in the build log and search for: declare -p TEMPLATE_FLAVOR_PREFIX Then you see there is some code which feels like accounting for 2/3 of the build log. It's very repetitive. Could this be somehow refactored, run earlier to reduce the log length?