tests: support running spread suite in autopkgtests #2188

kyrofa · 2018-08-02T00:11:31Z

Have you followed the guidelines for contributing?
Have you signed the CLA?
If this is a bugfix. Have you checked that there is a bug report open for the issue you are trying to fix on bug reports?
If this is a new feature. Have you discussed the design on the forum?
Have you successfully run ./runtests.sh static?
Have you successfully run ./runtests.sh unit?

The spread suite should run as part of the release process. Add an autopkgtest suite to run it.

The spread suite should run as part of the release process. Add an autopkgtest suite to run it. Also update testing docs for running autopkgtests on Bionic. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

codecov-io · 2018-08-02T04:25:30Z

Codecov Report

❗ No coverage uploaded for pull request base (master@41724d5). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2188   +/-   ##
=========================================
  Coverage          ?   91.13%           
=========================================
  Files             ?      201           
  Lines             ?    12920           
  Branches          ?     1915           
=========================================
  Hits              ?    11775           
  Misses            ?      777           
  Partials          ?      368

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41724d5...a44bd18. Read the comment docs.

sergiusens · 2018-08-02T12:25:58Z

TESTING.md

+The easiest way is to use a LXC container. From the root of the project, run:
+
+    sudo apt install autopkgtest
+    autopkgtest . -U -- lxd ubuntu:xenial


autopkgtest instead of adt becuase the name changed in the archive?

Indeed, there is no adt-run now.

sergiusens · 2018-08-02T12:28:35Z

debian/tests/integrationtests-spread

+if [ "${http_proxy:-}" != "" ]; then
+	cat <<-EOF | tee /etc/systemd/system/snapd.service.d/proxy.conf
+		[Service]
+		Environment=http_proxy=$http_proxy


would this be a harmless unit if $http_proxy is not defined?

It wouldn't be defined at all (see the if statement above).

duh, I completely missed that; my suggestion was going to be to gate it just like you did!

sergiusens · 2018-08-02T12:30:09Z

debian/tests/integrationtests-spread

+
+# and now run spread against localhost
+# shellcheck disable=SC1091
+. /etc/os-release


maybe "source" instead of "."?

source is a bashism, this is using dash.

sergiusens · 2018-08-02T12:31:48Z

spread.yaml

-  if [ -z "$SNAPCRAFT_CHANNEL" ]; then
-      if stat /snapcraft/*.snap 2>/dev/null; then
-          snap install --classic --dangerous /snapcraft/*.snap
+  if [ "$SNAPCRAFT_PACKAGE_TYPE" = "deb" ]; then


maybe not now, but this might be nice to have in helper scripts to be able to use shellcheck? Or did we discuss this already and am I going on in circles?

I agree 100%, sadly, see this comment for why it has to be this way.

sergiusens

thanks for getting back to me, feel free to merge once the adt run proves successful

kyrofa · 2018-08-02T15:57:19Z

Judging from our other integration tests, I suspect armhf will fail due to an inability to install snaps.

kyrofa · 2018-08-02T21:12:44Z

Haha, spread tests pass, plugin suite fails.

sergiusens · 2018-08-03T00:29:57Z

as expected @kyrofa the one that fails though is the cmake one which is now in master as a spread test, seems like a missing gcc build-packages entry satisfied by the migrated autotools one

…pkgtest

kyrofa · 2018-08-03T16:03:36Z

@sergiusens indeed, it requires g++, which I fixed in the migration to spread, so I've now merged with master. The only other fix is to add openssh-server to the test deps so armhf can actually be used by spread, so we'll see how it is now.

Apparently not all autopkgtest images have it (looking at you, armhf). Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

sergiusens · 2018-08-04T17:37:45Z

@kyrofa maybe just disable on armhf for now, let's run it on the other green arches to make sure it is isolated before making the change though (i386 and arm64), this might be a discrepancy between native run versus container run scenario that we are seeing.

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

This reverts commit 826676b.

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

This will automatically skip armhf tests since it can't provide such isolation. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

…pkgtest

Cheat and use python3-setuptools instead, which exists on both xenial and bionic, and depends upon python3-distutils in bionic. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

…pkgtest

kyrofa · 2018-08-09T14:43:12Z

Jeez, finally. \o/

tests: support running spread suite in autopkgtests

909bef1

The spread suite should run as part of the release process. Add an autopkgtest suite to run it. Also update testing docs for running autopkgtests on Bionic. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

kyrofa force-pushed the feature/spread_autopkgtest branch from 789b685 to 909bef1 Compare August 2, 2018 00:12

sergiusens reviewed Aug 2, 2018

View reviewed changes

sergiusens approved these changes Aug 2, 2018

View reviewed changes

Merge remote-tracking branch 'origin/master' into feature/spread_auto…

16931e8

…pkgtest

debian/tests: add openssh-server to deps

6b17e67

Apparently not all autopkgtest images have it (looking at you, armhf). Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

kyrofa force-pushed the feature/spread_autopkgtest branch from 65cffc8 to 6b17e67 Compare August 3, 2018 17:16

kyrofa added 4 commits August 6, 2018 08:48

Don't try to install snaps in armhf autopkgtest

826676b

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

Revert "Don't try to install snaps in armhf autopkgtest"

0c0e9ec

This reverts commit 826676b.

Add fuse so armhf autopkgtests can install snaps

b1e4904

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

Restrict spread tests to isolation-machine

5fe72c5

This will automatically skip armhf tests since it can't provide such isolation. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

sergiusens approved these changes Aug 7, 2018

View reviewed changes

Add git to spread test deps

05c1389

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

kyrofa force-pushed the feature/spread_autopkgtest branch from 56bb37b to 05c1389 Compare August 8, 2018 02:32

sergiusens mentioned this pull request Aug 8, 2018

spread: stop testing 17.10 #2197

Merged

6 tasks

kyrofa added 4 commits August 8, 2018 07:52

debian/control: add distutils as dependency to snapcraft

7a99733

Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

Merge remote-tracking branch 'origin/master' into feature/spread_auto…

646aedb

…pkgtest

python3-distutils isn't available on xenial (nor is it needed)

10fc5c7

Cheat and use python3-setuptools instead, which exists on both xenial and bionic, and depends upon python3-distutils in bionic. Signed-off-by: Kyle Fazzari <kyrofa@ubuntu.com>

Merge remote-tracking branch 'origin/master' into feature/spread_auto…

a44bd18

…pkgtest

kyrofa merged commit 10139a4 into canonical:master Aug 9, 2018

kyrofa deleted the feature/spread_autopkgtest branch August 9, 2018 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: support running spread suite in autopkgtests #2188

tests: support running spread suite in autopkgtests #2188

kyrofa commented Aug 2, 2018 •

edited

codecov-io commented Aug 2, 2018 •

edited

sergiusens Aug 2, 2018

kyrofa Aug 2, 2018

sergiusens Aug 2, 2018

kyrofa Aug 2, 2018

sergiusens Aug 2, 2018

sergiusens Aug 2, 2018

kyrofa Aug 2, 2018 •

edited

sergiusens Aug 2, 2018 •

edited

kyrofa Aug 2, 2018

sergiusens left a comment

kyrofa commented Aug 2, 2018 •

edited

kyrofa commented Aug 2, 2018

sergiusens commented Aug 3, 2018

kyrofa commented Aug 3, 2018 •

edited

sergiusens commented Aug 4, 2018

kyrofa commented Aug 9, 2018

tests: support running spread suite in autopkgtests #2188

tests: support running spread suite in autopkgtests #2188

Conversation

kyrofa commented Aug 2, 2018 • edited

codecov-io commented Aug 2, 2018 • edited

Codecov Report

sergiusens Aug 2, 2018

Choose a reason for hiding this comment

kyrofa Aug 2, 2018

Choose a reason for hiding this comment

sergiusens Aug 2, 2018

Choose a reason for hiding this comment

kyrofa Aug 2, 2018

Choose a reason for hiding this comment

sergiusens Aug 2, 2018

Choose a reason for hiding this comment

sergiusens Aug 2, 2018

Choose a reason for hiding this comment

kyrofa Aug 2, 2018 • edited

Choose a reason for hiding this comment

sergiusens Aug 2, 2018 • edited

Choose a reason for hiding this comment

kyrofa Aug 2, 2018

Choose a reason for hiding this comment

sergiusens left a comment

Choose a reason for hiding this comment

kyrofa commented Aug 2, 2018 • edited

kyrofa commented Aug 2, 2018

sergiusens commented Aug 3, 2018

kyrofa commented Aug 3, 2018 • edited

sergiusens commented Aug 4, 2018

kyrofa commented Aug 9, 2018

kyrofa commented Aug 2, 2018 •

edited

codecov-io commented Aug 2, 2018 •

edited

kyrofa Aug 2, 2018 •

edited

sergiusens Aug 2, 2018 •

edited

kyrofa commented Aug 2, 2018 •

edited

kyrofa commented Aug 3, 2018 •

edited