pluginhandler: support more complex stage-packages. (#1059)

Currently snapcraft only supports a flat list of stage packages to be
staged regardless of target architecture. This commit introduces a more
complex grammar that allows one to filter stage packages depending on
various selectors (target arch for right now), as well as specify
optional packages.

The grammar is made up of two statements: `on` and `try`.

- on <selector>[,<selector>...]
  - ...
- else[ fail]:
  - ...

The body of the `on` clause is taken into account if every (AND, not OR)
selector is true for the target build environment. Currently the only
selectors supported are target architectures (e.g. amd64).

If the `on` clause doesn't match and it's immediately followed by an
`else` clause, the `else` clause must be satisfied. An `on` clause
without an `else` clause is considered satisfied even if no selector
matched. The `else fail` form allows erroring out if an `on` clause was
not matched.

- try:
  - ...
- else:
  - ...

The body of the `try` clause is taken into account only when all
packages contained within it are valid. If not, if it's immediately
followed by `else` clauses they are tried in order, and one of them must
be satisfied. A `try` clause with no `else` clause is considered
satisfied even if it contains invalid packages.

LP: #1637282

Signed-off-by: Kyle Fazzari <kyle@canonical.com>

docs/snapcraft-syntax.md

@@ -82,8 +82,43 @@ contain.
       be searched for in [the wiki](https://wiki.ubuntu.com/Snappy/Parts).
       *If a part is supposed to run after another, the prerequisite part will
       be staged before the dependent part starts its lifecycle.*
-    * `stage-packages` (list of strings)
-      A list of Ubuntu packages to use that would support the part creation.
+    * `stage-packages` (list of strings and/or sublists)
+      A set of Ubuntu packages to be downloaded and unpacked to join the part
+      before it's built. Note that these packages are not installed on the host.
+      Like the rest of the part, all files from these packages will make it into
+      the final snap unless filtered out via the `snap` keyword.
+
+      One may simply specify packages in a flat list, in which case the packages
+      will be fetched and unpacked regardless of build environment. In addition,
+      a specific grammar made up of sub-lists is supported here that allows one
+      to filter stage packages depending on various selectors (e.g. the target
+      arch), as well as specify optional packages. The grammar is made up of two
+      nestable statements: 'on' and 'try'.
+
+          - on <selector>[,<selector>...]:
+            - ...
+          - else[ fail]:
+            - ...
+
+      The body of the 'on' clause is taken into account if every (AND, not OR)
+      selector is true for the target build environment. Currently the only
+      selectors supported are target architectures (e.g. amd64).
+
+      If the 'on' clause doesn't match and it's immediately followed by an
+      'else' clause, the 'else' clause must be satisfied. An 'on' clause without
+      an 'else' clause is considered satisfied even if no selector matched. The
+      'else fail' form allows erroring out if an 'on' clause was not matched.
+
+          - try:
+            - ...
+          - else:
+            - ...
+
+      The body of the 'try' clause is taken into account only when all packages
+      contained within it are valid. If not, if it's immediately followed by
+      'else' clauses they are tried in order, and one of them must be satisfied.
+      A 'try' clause with no 'else' clause is considered satisfied even if it
+      contains invalid packages.
     * `build-packages` (list of strings)
       A list of Ubuntu packages to be installed on the host to aid in building
       the part. These packages will not go into the final snap.

integration_tests/snaps/stage-package-grammar/snapcraft.yaml

@@ -0,0 +1,53 @@
+name: stage-package-grammar
+version: '0.1'
+summary: Test the stage package grammar
+description: A few different parts that exercise the grammar differently
+grade: devel
+confinement: strict
+
+parts:
+  simple:
+    plugin: nil
+    stage-packages:
+      - hello
+
+  try:
+    plugin: nil
+    stage-packages:
+      - try:
+        - hello
+
+  try-skipped:
+    plugin: nil
+    stage-packages:
+      - try:
+        - invalid-package
+
+  try-else:
+    plugin: nil
+    stage-packages:
+      - try:
+        - invalid-package
+      - else:
+        - hello
+
+  on-other-arch:
+    plugin: nil
+    stage-packages:
+      - on other-arch:
+        - foo
+
+  on-other-arch-else:
+    plugin: nil
+    stage-packages:
+      - on other-arch:
+        - foo
+      - else:
+        - hello
+
+  on-other-arch-else-fail:
+    plugin: nil
+    stage-packages:
+      - on other-arch:
+        - foo
+      - else fail

integration_tests/test_stage_package_grammar.py

@@ -0,0 +1,93 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2017 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import subprocess
+
+import integration_tests
+from testtools.matchers import (
+    Contains,
+    FileExists,
+    Not
+)
+
+
+class StagePackageGrammarTestCase(integration_tests.TestCase):
+
+    def test_simple(self):
+        """Test that 'simple' fetches stage package."""
+
+        self.run_snapcraft(['prime', 'simple'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            FileExists())
+
+    def test_try(self):
+        """Test that 'try' fetches stage package."""
+
+        self.run_snapcraft(['prime', 'try'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            FileExists())
+
+    def test_try_skipped(self):
+        """Test that 'try-skipped' fetches nothing."""
+
+        self.run_snapcraft(['prime', 'try-skipped'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            Not(FileExists()))
+
+    def test_try_else(self):
+        """Test that 'try-else' fetches stage package."""
+
+        self.run_snapcraft(['prime', 'try-else'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            FileExists())
+
+    def test_on_other_arch(self):
+        """Test that 'on-other-arch' fetches nothing."""
+
+        self.run_snapcraft(['prime', 'on-other-arch'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            Not(FileExists()))
+
+    def test_on_other_arch_else(self):
+        """Test that 'on-other-arch-else' fetches stage package."""
+
+        self.run_snapcraft(
+            ['prime', 'on-other-arch-else'], 'stage-package-grammar')
+
+        self.assertThat(
+            os.path.join('prime', 'usr', 'bin', 'hello'),
+            FileExists())
+
+    def test_on_other_arch_else_fail(self):
+        """Test that 'on-other-arch-else-fail' fails with an error."""
+
+        exception = self.assertRaises(
+            subprocess.CalledProcessError, self.run_snapcraft,
+            ['prime', 'on-other-arch-else-fail'], 'stage-package-grammar')
+
+        self.assertThat(exception.output, Contains(
+            "Unable to satisfy 'on other-arch', failure forced"))

schema/snapcraft.yaml

@@ -1,5 +1,33 @@
 $schema: http://json-schema.org/draft-04/schema#
 
+definitions:
+  stage-packages:
+    type: array
+    minitems: 1
+    uniqueItems: true
+    items:
+      anyOf:
+        - type: string
+          usage: "<string>"
+        - type: object
+          usage: "on <selector>[,<selector>...]:"
+          additionalProperties: false
+          patternProperties:
+            ^on\s+.+$:
+              $ref: "#/definitions/stage-packages"
+        - type: object
+          usage: "try:"
+          additionalProperties: false
+          patternProperties:
+            ^try$:
+              $ref: "#/definitions/stage-packages"
+        - type: object
+          usage: "else:"
+          additionalProperties: false
+          patternProperties:
+            ^else$:
+              $ref: "#/definitions/stage-packages"
+
 title: snapcraft schema
 type: object
 properties:
@@ -220,12 +248,8 @@ properties:
               type: string
             default: []
           stage-packages:
-            type: array
-            minitems: 1
-            uniqueItems: true
-            items:
-              type: string
-            default: []
+            $ref: "#/definitions/stage-packages"
+            default: [] # For some reason this doesn't work if in the ref
           build-packages:
             type: array
             minitems: 1

setup.py

@@ -43,6 +43,7 @@
               'snapcraft.internal.cache',
               'snapcraft.internal.deltas',
               'snapcraft.internal.pluginhandler',
+              'snapcraft.internal.pluginhandler.stage_package_grammar',
               'snapcraft.internal.sources',
               'snapcraft.internal.states',
               'snapcraft.plugins',

snapcraft/__init__.py

@@ -144,13 +144,90 @@
     binaries within the snap (in which case they'll be discovered via `ldd`),
     or they are explicitly described in stage-packages.
 
-  - stage-packages: [deb, deb, deb...]
+  - stage-packages: YAML list
 
-    A list of Ubuntu packages to be downloaded and unpacked to join the part
+    A set of Ubuntu packages to be downloaded and unpacked to join the part
     before it's built. Note that these packages are not installed on the host.
     Like the rest of the part, all files from these packages will make it into
     the final snap unless filtered out via the `snap` keyword.
 
+    One may simply specify packages in a flat list, in which case the packages
+    will be fetched and unpacked regardless of build environment. In addition,
+    a specific grammar made up of sub-lists is supported here that allows one
+    to filter stage packages depending on various selectors (e.g. the target
+    arch), as well as specify optional packages. The grammar is made up of two
+    nestable statements: 'on' and 'try'.
+
+    Let's discuss `on`.
+
+        - on <selector>[,<selector>...]:
+          - ...
+        - else[ fail]:
+          - ...
+
+    The body of the 'on' clause is taken into account if every (AND, not OR)
+    selector is true for the target build environment. Currently the only
+    selectors supported are target architectures (e.g. amd64).
+
+    If the 'on' clause doesn't match and it's immediately followed by an 'else'
+    clause, the 'else' clause must be satisfied. An 'on' clause without an
+    'else' clause is considered satisfied even if no selector matched. The
+    'else fail' form allows erroring out if an 'on' clause was not matched.
+
+    For example, say you only wanted to stage `foo` if building for amd64 (and
+    not stage `foo` if otherwise):
+
+        - on amd64: [foo]
+
+    Building on that, say you wanted to stage `bar` if building on an arch
+    other than amd64:
+
+        - on amd64: [foo]
+        - else: [bar]
+
+    You can nest these for more complex behaviors:
+
+        - on amd64: [foo]
+        - else:
+          - on i386: [bar]
+          - on armhf: [baz]
+
+    If your project requires a package that is only available on amd64, you can
+    fail if you're not building for amd64:
+
+        - on amd64: [foo]
+        - else fail
+
+    Now let's discuss `try`:
+
+        - try:
+          - ...
+        - else:
+          - ...
+
+    The body of the 'try' clause is taken into account only when all packages
+    contained within it are valid. If not, if it's immediately followed by
+    'else' clauses they are tried in order, and one of them must be satisfied.
+    A 'try' clause with no 'else' clause is considered satisfied even if it
+    contains invalid packages.
+
+    For example, say you wanted to stage `foo`, but it wasn't available for all
+    architectures. Assuming your project builds without it, you can make it an
+    optional stage package:
+
+        - try: [foo]
+
+    You can also add alternatives:
+
+        - try: [foo]
+        - else: [bar]
+
+    Again, you can nest these for more complex behaviors:
+
+        - on amd64: [foo]
+        - else:
+          - try: [bar]
+
   - organize: YAML
 
     Snapcraft will rename files according to this YAML sub-section. The

snapcraft/_schema.py

@@ -47,6 +47,12 @@ def part_schema(self):
         properties = sub['^(?!plugins$)[a-z0-9][a-z0-9+-\/]*$']['properties']
         return properties
 
+    @property
+    def definitions_schema(self):
+        """Return sub-schema that describes definitions used within schema."""
+
+        return self._schema['definitions'].copy()
+
     def _load_schema(self):
         schema_file = os.path.abspath(os.path.join(
             common.get_schemadir(), 'snapcraft.yaml'))