Move Debian tests from unstable to testing #5726

martinpitt · 2017-01-11T15:29:48Z

This will unblock issues like in PR #5714.

I'm currently building a debian-testing image locally and will run tests against it and check that storaged works against debian testing too (I don't expect any difficulties, though).

martinpitt · 2017-01-11T15:38:33Z

Unfortunately building an image for Debian testing doesn't currently work either:

Package docker.io is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'docker.io' has no installation candidate

It got kicked out of testing three months ago because it has too many release-critical bugs.

martinpitt · 2017-01-11T15:58:46Z

I worked around the docker.io issue by adding jessie-backports, pushed. I also took the liberty to change the mirror to the automatic/GeoIP aware redirector for faster local builds. Please let me know if ftk.uk.d.o. must be used because of some firewall rules or similar.

dperpeet · 2017-01-11T16:00:15Z

I also took the liberty to change the mirror to the automatic/GeoIP aware redirector for faster local builds. Please let me know if ftk.uk.d.o. must be used because of some firewall rules or similar.

This is good and maybe deserves its own commit.

martinpitt · 2017-01-11T16:05:22Z

Note to self: move jessie-backports enabling to test/images/scripts/debian-testing.setup, alongside enabling experimental. Fits better there.

martinpitt · 2017-01-11T17:04:43Z

Pushed again, image building gets further now. Now it fails with the pbuilder pre-caching:

The following packages have unmet dependencies:
 libssh-dev : Depends: libssl1.0-dev but it is not going to be installed
 npm : Depends: node-gyp (>= 0.10.9) but it is not going to be installed

In testing libssh-dev still depends on libssl1.0-dev while nodejs-dev (dependency of npm) already moved to OpenSSL 1.0 (libssl-dev), so they are not co-installable. This should hopefully clear up in 6 days (https://tracker.debian.org/news/830414).

mvollmer · 2017-01-12T07:33:21Z

I also took the liberty to change the mirror to the automatic/GeoIP aware redirector for faster local builds. Please let me know if ftk.uk.d.o. must be used because of some firewall rules or similar.

This is good and maybe deserves its own commit.

Didn't we try this and had to revert to ftp.uk.d.o because the mirrors were frequently unreliable? But I am all for trying again.

fsateler · 2017-01-12T18:50:40Z

I also took the liberty to change the mirror to the automatic/GeoIP aware redirector for faster local builds. Please let me know if ftk.uk.d.o. must be used because of some firewall rules or similar.

This is good and maybe deserves its own commit.

Didn't we try this and had to revert to ftp.uk.d.o because the mirrors were frequently unreliable? But I am all for trying again.

I think http://deb.debian.org/ is the new "global" recommendation (it is backed by 2 cdns, fastly and cloudfront).

martinpitt · 2017-01-16T12:10:58Z

I reworked this to drop the pcp dependencies and the cockpit-pcp package when building for debian-testing, as this is under-maintained and rather broken (uninstallable in unstable, not available in testing). The structure of this makes it relatively easy to put it back once it's available again.

Building a "debian-testing" image and preparing the test suite on it (i. e. package build, etc.) now worked locally, so I'll request an image build; I can work on adjusting the pcp test to be conditional while the infra is cranking out the new image.

stefwalter · 2017-01-16T12:10:10Z

test/images/scripts/lib/debian.install

@@ -46,6 +46,9 @@ if [ -n "$do_build" ]; then
    tar -xzf cockpit-*.tar.gz
    ( cd cockpit-*/
      cp -rp tools/debian debian
+      # Hack: Remove PCP build dependencies while pcp is not in testing
+      # (https://tracker.debian.org/pcp)
+      sed -i '/libpcp.*-dev/d' debian/control


It seems this situation would prevent us from doing our weekly releases against Debian Testing? Broader question: Should we only be doing weekly releases against Jessie, and drop everything else?

To the contrary, this PR now would start to allow us doing weekly releases into unstable/testing as a new cockpit can actually propagate. It can't while it build/binary depends on pcp which isn't available in testing, and at the moment we can't even build cockpit with PCP in unstable.

The price is of course to drop the -pcp binary for the time being, but that's still better than dropping cockpit entirely IMHO.

... oh, that of course means that we have to drop the pcp build deps from the actual upload. If we can do this more elegantly someplace else, I'm all for it of course.

A different approach would be to use build dependency alternatives like pcp-dev | aspell-doc so that the builders install the (harmless) aspell-doc if pcp is not available. But this is really quite a hack and not even guaranteed to work on Debian builders (I know that Ubuntu's builders try alternate build dependencies, but Debian's didn't in the past), so I'd rather do the uploads in a way that the uploaded source package has the pcp build deps removed.

The continuous delivery scripts are here:

https://github.com/cockpit-project/cockpituous/blob/master/release/release-dsc

https://github.com/cockpit-project/cockpituous/blob/master/release/release-debian

https://github.com/cockpit-project/cockpituous/blob/master/release/release-ubuntu-ppa

There's very little Cockpit specific in the Debian continuous delivery scripts, and we should be working towards removing the last remaining Cockpit specific bits.

Thanks for the pointer. Indeed there are two seds in there for mangling debian/rules already. If we want to generalize those, how about instead calling a debian/rules prepare-dsc target or a debian/prepare-dsc.sh from cockpituous, and keep the project specific bits in cockpit itself?

This should also get a release or series argument (passed from release-{debian,ubuntu}) so that the mangling can be release specific and release-dsc can actually put it into the changelog too (instead of experimental).

I like that. But lets take it a step further:

if -s or RELEASE_DEBIAN is a directory, then use it directly as a debian directory.

If -s or RELEASE_DEBIAN points to an executable, then execute it to produce a debian directory.

That said, in either case, how would we pass in information about which distribution we're creating an DSC for? Also currently we create one DSC and use it for multiple distributions, and then munge it further in release-debian ... however I'm not against creating multiple DSC files that are distribution appropriate right from release-dsc.

I filed that as cockpit-project/cockpituous#65

quick adjustment is in cockpit-project/cockpituous#66

martinpitt · 2017-01-16T12:11:34Z

bot: Image refresh for debian-testing

martinpitt · 2017-01-16T16:34:55Z

Uploaded manually, as the bot doesn't work for this yet: https://fedorapeople.org/groups/cockpit/images/debian-testing-c4971ea174fe44f14239d555c1fd0c9def9d84a1.qcow2.xz

Thus this should work now.

stefwalter

Hmmm, I think if we're removing the test/images/debian-unstable link then we need to remove "debian-unstable" from test/common and friends.

martinpitt · 2017-01-16T20:03:54Z

OK, I locally removed the remaining traces of "debian-unstable". I'll wait with pushing until the debian-testing test finishes, if the currently linked log is already current there are some failures there which look systematic (old docker.io?).

stefwalter · 2017-01-17T20:06:10Z

You need this diff:

diff --git a/test/verify/check-realms b/test/verify/check-realms
index 3fa972d..882c051 100755
--- a/test/verify/check-realms
+++ b/test/verify/check-realms
@@ -45,7 +45,7 @@ def prepare_resolv(machine, address):
     machine.execute(script=RESOLV_SCRIPT.format(address=address))
 
 @skipImage("No realmd available", "continuous-atomic", "fedora-atomic", "rhel-atomic")
-@skipImage("No freeipa available", "debian-8")
+@skipImage("No freeipa available", "debian-8", "debian-testing")
 class TestRealms(MachineCase):
     additional_machines = {
         'ipa': { 'machine': { 'image': 'ipa' } }
@@ -196,7 +196,7 @@ getent passwd admin@cockpit.lan
 
 # This is here because our test framework can't run ipa VM's twice
 @skipImage("No realmd available", "continuous-atomic", "fedora-atomic", "rhel-atomic")
-@skipImage("No freeipa available", "debian-8")
+@skipImage("No freeipa available", "debian-8", "debian-testing")
 class TestKerberos(MachineCase):
     additional_machines = {
         'ipa': { 'machine': { 'image': 'ipa' } }

Also enable jessie-backports in the bootstrap script until docker.io gets back into testing (https://tracker.debian.org/news/804615). Restart docker after installing everything, so that it starts with the aufs backend (aufs-dkms might be installed after docker.io). Drop freeipa-client as this is only in unstable due to too many release-critical bugs.

Set dpkg's unsafe-io option to avoid fsyncing after unpacking every file, for greatly speaking up package installation. Don't download long package descriptions, we don't need those and they take considerable time on apt-get update.

This is an automatic mirror redirector, see <http://deb.debian.org>.

It will soon get dropped from the actual Debian packaging (cockpit-project#5710) and gets in the way in debian-testing due to some packages using libssl1.0-dev still.

pcp is currently not available in Debian testing and uninstallable/broken in unstable, so drop it from the image build and pbuilder. See https://tracker.debian.org/pkg/pcp for details. Skip the corresponding TestMetrics.testPcp() test case.

Current Debian testing, and maybe future Debian/Ubuntu releases don't have pcp. Do not build the cockpit-pcp package either in that case.

It transitively gets pulled in through pcp, but when building without that it failed with ./tools/tap-driver: /usr/bin/python: bad interpreter: No such file or directory

This avoids getting broken images and broken tests when any of the imports fail for some reason.

We don't install ntpd into our test images, and if ntp.service does not exist the TestSystemInfo.testTime test fails with CalledProcessError: Command 'systemctl stop ntp' returned non-zero exit status 5

Something in the virt-builder/upgrade chain causes sshd_config to have "#PermitRootLogin prohibit-password" (i. e. being commented out). Make the sed accept that. This fixes check-dashboard TestDashboardSetup.testSetup.

martinpitt · 2017-01-17T20:17:27Z

Can reproduce, pushed.

martinpitt · 2017-01-18T08:18:05Z

FTR, all debian-8 tests pass locally for me, production seems to have some connection trouble to github ATM. In yesterday's test attempt there was only one failure on debian-8 which looked random.

Set dpkg's unsafe-io option to avoid fsyncing after unpacking every file, for greatly speaking up package installation. Don't download long package descriptions, we don't need those and they take considerable time on apt-get update. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

This is an automatic mirror redirector, see <http://deb.debian.org>. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

It will soon get dropped from the actual Debian packaging (#5710) and gets in the way in debian-testing due to some packages using libssl1.0-dev still. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

pcp is currently not available in Debian testing and uninstallable/broken in unstable, so drop it from the image build and pbuilder. See https://tracker.debian.org/pkg/pcp for details. Skip the corresponding TestMetrics.testPcp() test case. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

Current Debian testing, and maybe future Debian/Ubuntu releases don't have pcp. Do not build the cockpit-pcp package either in that case. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

It transitively gets pulled in through pcp, but when building without that it failed with ./tools/tap-driver: /usr/bin/python: bad interpreter: No such file or directory Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

This avoids getting broken images and broken tests when any of the imports fail for some reason. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

We don't install ntpd into our test images, and if ntp.service does not exist the TestSystemInfo.testTime test fails with CalledProcessError: Command 'systemctl stop ntp' returned non-zero exit status 5 Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

Something in the virt-builder/upgrade chain causes sshd_config to have "#PermitRootLogin prohibit-password" (i. e. being commented out). Make the sed accept that. This fixes check-dashboard TestDashboardSetup.testSetup. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

…ting pcp has release-critical bugs and fell out of testing, see https://tracker.debian.org/pkg/pcp. Drop the pcp build dependencies when building for unstable so that the package can be used on testing/stretch. This requires cockpit-project/cockpit#5726 so that the package actually builds without these build dependencies. Closes #66 Reviewed-by: Stef Walter <stefw@redhat.com>

stefwalter added bot needswork labels Jan 11, 2017

martinpitt force-pushed the debian-move-to-testing branch 2 times, most recently from 6440b6c to 6be8958 Compare January 11, 2017 15:53

martinpitt force-pushed the debian-move-to-testing branch 3 times, most recently from 60c35d5 to d94b245 Compare January 11, 2017 17:04

martinpitt force-pushed the debian-move-to-testing branch from d94b245 to 7770fd6 Compare January 11, 2017 21:52

martinpitt force-pushed the debian-move-to-testing branch from 7770fd6 to fff7c30 Compare January 15, 2017 21:38

stefwalter reviewed Jan 16, 2017

View reviewed changes

martinpitt force-pushed the debian-move-to-testing branch from 4f53b3a to 3018c78 Compare January 16, 2017 13:56

martinpitt force-pushed the debian-move-to-testing branch from 3018c78 to e714106 Compare January 16, 2017 16:35

stefwalter removed bot needswork labels Jan 16, 2017

stefwalter changed the title ~~WIP: Move Debian tests from unstable to testing~~ Move Debian tests from unstable to testing Jan 16, 2017

martinpitt force-pushed the debian-move-to-testing branch from e714106 to 7025d60 Compare January 16, 2017 16:43

martinpitt mentioned this pull request Jan 16, 2017

Debian: Fix starting/stopping of units on package (un)install #5714

Closed

stefwalter approved these changes Jan 16, 2017

View reviewed changes

stefwalter suggested changes Jan 16, 2017

View reviewed changes

stefwalter added the needswork label Jan 17, 2017

martinpitt and others added 10 commits January 17, 2017 21:17

test: debian/ubuntu VM bootstrap speedups

33c196c

Set dpkg's unsafe-io option to avoid fsyncing after unpacking every file, for greatly speaking up package installation. Don't download long package descriptions, we don't need those and they take considerable time on apt-get update.

test: Move Debian apt sources to deb.debian.org

7d33e30

This is an automatic mirror redirector, see <http://deb.debian.org>.

test: Drop libssl-dev from Debian pbuilders

eb29b3b

It will soon get dropped from the actual Debian packaging (cockpit-project#5710) and gets in the way in debian-testing due to some packages using libssl1.0-dev still.

test: Drop PCP from debian-testing

c277170

pcp is currently not available in Debian testing and uninstallable/broken in unstable, so drop it from the image build and pbuilder. See https://tracker.debian.org/pkg/pcp for details. Skip the corresponding TestMetrics.testPcp() test case.

debian: Only build cockpit-pcp if pcp is available

4db1115

Current Debian testing, and maybe future Debian/Ubuntu releases don't have pcp. Do not build the cockpit-pcp package either in that case.

debian: Add missing python build dependency

c44023c

It transitively gets pulled in through pcp, but when building without that it failed with ./tools/tap-driver: /usr/bin/python: bad interpreter: No such file or directory

test: Make docker-images.setup failure fatal

2743cb2

This avoids getting broken images and broken tests when any of the imports fail for some reason.

test: Drop obsolete ntp stop quirk in check-system-info

a6f49eb

We don't install ntpd into our test images, and if ntp.service does not exist the TestSystemInfo.testTime test fails with CalledProcessError: Command 'systemctl stop ntp' returned non-zero exit status 5

test: Fix setting PermitRootLogin in sshd config

298dbeb

Something in the virt-builder/upgrade chain causes sshd_config to have "#PermitRootLogin prohibit-password" (i. e. being commented out). Make the sed accept that. This fixes check-dashboard TestDashboardSetup.testSetup.

martinpitt force-pushed the debian-move-to-testing branch from 73ad182 to 298dbeb Compare January 17, 2017 20:17

martinpitt removed the needswork label Jan 17, 2017

This was referenced Jan 18, 2017

Move cockpit specific Debian source package mangling from cockpituous into cockpit.git cockpit-project/cockpituous#65

Closed

release-debian: Remove PCP build dependencies while pcp is not in testing cockpit-project/cockpituous#66

Closed

stefwalter closed this in 5c14fa1 Jan 18, 2017

stefwalter pushed a commit that referenced this pull request Jan 18, 2017

test: Move Debian apt sources to deb.debian.org

aa812b1

This is an automatic mirror redirector, see <http://deb.debian.org>. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

stefwalter pushed a commit that referenced this pull request Jan 18, 2017

test: Make docker-images.setup failure fatal

25aa15c

This avoids getting broken images and broken tests when any of the imports fail for some reason. Closes #5726 Reviewed-by: Stef Walter <stefw@redhat.com>

martinpitt deleted the debian-move-to-testing branch January 18, 2017 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move Debian tests from unstable to testing #5726

Move Debian tests from unstable to testing #5726

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017

dperpeet commented Jan 11, 2017

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017 •

edited

Loading

mvollmer commented Jan 12, 2017

fsateler commented Jan 12, 2017

martinpitt commented Jan 16, 2017

stefwalter Jan 16, 2017

martinpitt Jan 16, 2017

martinpitt Jan 16, 2017

stefwalter Jan 16, 2017

martinpitt Jan 16, 2017

stefwalter Jan 16, 2017 •

edited

Loading

martinpitt Jan 18, 2017

martinpitt Jan 18, 2017

martinpitt commented Jan 16, 2017

martinpitt commented Jan 16, 2017

stefwalter left a comment

martinpitt commented Jan 16, 2017

stefwalter commented Jan 17, 2017

martinpitt commented Jan 17, 2017

martinpitt commented Jan 18, 2017

Move Debian tests from unstable to testing #5726

Move Debian tests from unstable to testing #5726

Conversation

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017

dperpeet commented Jan 11, 2017

martinpitt commented Jan 11, 2017

martinpitt commented Jan 11, 2017 • edited Loading

mvollmer commented Jan 12, 2017

fsateler commented Jan 12, 2017

martinpitt commented Jan 16, 2017

stefwalter Jan 16, 2017

Choose a reason for hiding this comment

martinpitt Jan 16, 2017

Choose a reason for hiding this comment

martinpitt Jan 16, 2017

Choose a reason for hiding this comment

stefwalter Jan 16, 2017

Choose a reason for hiding this comment

martinpitt Jan 16, 2017

Choose a reason for hiding this comment

stefwalter Jan 16, 2017 • edited Loading

Choose a reason for hiding this comment

martinpitt Jan 18, 2017

Choose a reason for hiding this comment

martinpitt Jan 18, 2017

Choose a reason for hiding this comment

martinpitt commented Jan 16, 2017

martinpitt commented Jan 16, 2017

stefwalter left a comment

Choose a reason for hiding this comment

martinpitt commented Jan 16, 2017

stefwalter commented Jan 17, 2017

martinpitt commented Jan 17, 2017

martinpitt commented Jan 18, 2017

martinpitt commented Jan 11, 2017 •

edited

Loading

stefwalter Jan 16, 2017 •

edited

Loading