tests: dependency packages installed during prepare-project #3483

Merged
merged 15 commits into from Jul 19, 2017

Conversation

Projects
None yet
7 participants
Contributor

sergiocazzolato commented Jun 14, 2017

The objective of this change is to provide an environment for all the
tests as similar as possible, so all the dependencies are installed
during the project setup.

The packages are not removed at the end of the project, so it will help
in case the machine needs to be used to debug.

Openvswitch test is moved to manual due to the dependencies install is
stuck trying to install openvswitch-switch package.

codecov-io commented Jun 15, 2017

Codecov Report

Merging #3483 into master will increase coverage by <.01%.
The diff coverage is 75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3483      +/-   ##
==========================================
+ Coverage    74.9%   74.91%   +<.01%     
==========================================
  Files         380      380              
  Lines       32951    32986      +35     
==========================================
+ Hits        24682    24711      +29     
- Misses       6476     6483       +7     
+ Partials     1793     1792       -1
Impacted Files Coverage Δ
cmd/cmd.go 93.2% <75%> (+4.83%) ⬆️
cmd/snap/cmd_aliases.go 93.33% <0%> (-1.67%) ⬇️
cmd/snap-seccomp/main.go 50.78% <0%> (+0.36%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2c69c3...9566d60. Read the comment docs.

@sergiocazzolato sergiocazzolato changed the title from tests: dependency packages installed during project prepare to tests: dependency packages installed during prepare-project Jun 15, 2017

Collaborator

mvo5 commented Jun 19, 2017

Thanks for doing this! It looks good, I wonder however if dependencies.sh should instead simply be part of pkgdb.sh, this way all pkg releated things can be easily found in the same place.

Thanks! Going in a good direction. Just a few trivial points and questions.

tests/lib/dependencies.sh
@@ -0,0 +1,131 @@
+#!/bin/bash
@niemeyer

niemeyer Jun 19, 2017

Contributor

Agree with @mvo5 that pkgdb.sh seems like a reasonable place for this logic.

@sergiocazzolato

sergiocazzolato Jun 19, 2017

Contributor

sure, make sense to me too

tests/lib/dependencies.sh
+export DISTRO_BUILD_DEPS=
+
+add_pkgs(){
+ DEPENDENCY_PACKAGES="$DEPENDENCY_PACKAGES $@"
@niemeyer

niemeyer Jun 19, 2017

Contributor

We're trying to standardize on four spaces per indent on shell and yaml code. I've seen some other files which are already all over the place, with mixed 2, 3 and 4 spaces on the same file. Let's please fix that as we drive by these files (no need to rush it), and also try not to introduce new inconsistencies meanwhile.

tests/lib/dependencies.sh
+}
+
+get_apt_dependencies_generic(){
+ add_pkgs autoconf
@niemeyer

niemeyer Jun 19, 2017

Contributor

Communicating across shell functions via sticky environment variables is pretty hard to follow. It's a backdoor without a clear flow between the involved parties. Instead, this can simply echo autoconf here, and the other side can use the standard output as usual for its needs.

tests/lib/dependencies.sh
+ esac
+}
+
+get_apt_dependencies_core(){
@niemeyer

niemeyer Jun 19, 2017

Contributor

I suggest s/get_/pkg_/ on all of these functions. This preserves them with the same length and adds a bit of context to it (e.g. "test dependencies" alone could mean many things).

tests/lib/dependencies.sh
+ echo "Opensuse dependencies not ready yet"
+}
+
+get_test_dependencies(){
@niemeyer

niemeyer Jun 19, 2017

Contributor

Is there an advantage in splitting down test and build? Won't we end up with both of these installed anyway?

@sergiocazzolato

sergiocazzolato Jun 19, 2017

Contributor

yes, I already put all of them together

Member

chipaca commented Jul 10, 2017

Looks like the debian failures are related to your changes.

Contributor

sergiocazzolato commented Jul 12, 2017

@niemeyer when you have time, please take a look to this one, all the comments were addressed.

Looks great, thanks! Just a small nitpick

tests/lib/pkgdb.sh
- *)
- ;;
-esac
+pkg_apt_dependencies_generic(){
@fgimenez

fgimenez Jul 17, 2017

Contributor

I find this naming a bit inconsistent with what we have for fedora and opensuse where there's no reference to dnf or zypper, maybe pkg_dependency_ubuntu_{generic,classic} would be better.

A few last comments, and LGTM with those addressed.

Thanks!

tests/lib/pkgdb.sh
-esac
+
+pkg_dependency_ubuntu_generic(){
+ echo autoconf
@niemeyer

niemeyer Jul 17, 2017

Contributor

On all these cases, this would be nicer:

echo "
    foo
    bar
    baz
    "
tests/lib/pkgdb.sh
+ echo rpm-build
+}
+
+pkg_dependency_opensuse(){
@niemeyer

niemeyer Jul 17, 2017

Contributor

s/dependency/dependencies/ on all cases above would read slightly better and also align with the cases below.

tests/lib/pkgdb.sh
+ esac
+}
+
+install_dependencies(){
@niemeyer

niemeyer Jul 17, 2017

Contributor

install_pkg_dependencies perhaps?

tests/lib/pkgdb.sh
+}
+
+install_dependencies(){
+ pkgs=$(pkg_dependencies | tr "\n" " ")
@niemeyer

niemeyer Jul 17, 2017

Contributor

The tr should not be necessary here.

tests/lib/pkgdb.sh
+ pkgs=$(pkg_dependencies | tr "\n" " ")
+
+ # ensure systemd is up-to-date, if there is a mismatch libudev-dev
+ # will fail to install because the poor apt resolver does not get it
@niemeyer

niemeyer Jul 17, 2017

Contributor

As such, this should be inside distro_install_package and should be conditional on libudev-dev being part of the requested set.

tests/lib/pkgdb.sh
+ apt-get install -y --only-upgrade systemd
+ esac
+
+ echo "Installing the following packages: $pkgs"
@niemeyer

niemeyer Jul 17, 2017

Contributor

The logging should also be inside the install function.

Contributor

sergiocazzolato commented Jul 17, 2017

@niemeyer fixes requested already pushed

sergiocazzolato and others added some commits Jun 14, 2017

tests: dependency packages installed during project prepare
The objective of this change is to provide an environment for all the
tests as similar as possible, so all the dependencies are installed
during the project setup.

The packages are not removed at the end of the project, so it will help
in case the machine needs to be used to debug.

Openvswitch test is moved to manual due to the dependencies install is
stuck trying to install openvswitch-switch package.
Removing not needed exit while stalling dependencies
This is to support ubuntu17 on ci
Update dependencies function names
Requested on PR review
cmd,tests: fix classic confinement confusing re-execution code (#3598)
cmd,tests: fix classic confinement confusing re-execution code

When snapd inside the core snap is more recent than the snapd in the
classic distribution and the classic distribution supports snapd
re-execution then certain tools, namely snap and snapd will re-execute
the version of themselves found in the core snap.

To prevent endless loops as a part of this re-execution the variable
SNAP_DID_REEXEC is to 1 and tested in the re-execution mechanism.

Snaps using classic confinement do not get a mount namespace rooted at
the core (or designated base) snap and instead share the classic mount
namespace. As such their /usr/bin/snap is the one from the distribution
package.

When those two features are combined, a snap using classic confinement
will run with SNAP_DID_REEXEC=1 and /usr/bin/snap being the one from the
distribution. If such snap now runs a snap command, it will couple
potentially older /usr/bin/snap (e.g. from 2.25) with the profiles from
potentially newer /usr/bin/snapd (e.g. from 2.26.9). That specific
combination crosses the threshold where snap-seccomp begins to be used
and thus the old snap-confine cannot run new applications.

As a fix, unset SNAP_DID_REEXEC after observing the value 1, this will
ensure that the execution environment will *not* contain that flag and
each subsequent attempt to run /usr/bin/snap will re-consider
re-execution correctly.

Fixes: https://bugs.launchpad.net/snapd/+bug/1704860
Forum: https://forum.snapcraft.io/t/snapd-2-26-9-and-conjure-up-no-longer-work/
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

@sergiocazzolato sergiocazzolato merged commit a462005 into snapcore:master Jul 19, 2017

7 checks passed

artful-amd64 autopkgtest finished (success)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
xenial-amd64 autopkgtest finished (success)
Details
xenial-i386 autopkgtest finished (success)
Details
xenial-ppc64el autopkgtest finished (success)
Details
yakkety-amd64 autopkgtest finished (success)
Details
zesty-amd64 autopkgtest finished (success)
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment