Merge discover step documentation and fixes [psss#204]

spec/steps/discover.fmf

@@ -1,44 +1,102 @@
-summary: Discover test relevant for execution
+summary: Discover tests relevant for execution
 
 description: |
-    Gather information about the test cases which are supposed to
-    be run. This includes list of test cases with corresponding
-    L1 metadata. From the test case metadata constraints for test
-    enviroment can be detected:
+    Gather information about tests which are supposed to be run.
+    Provide method ``tests()`` returning a list of discovered
+    tests and ``requires()`` returning a list of all required
+    packages aggregated from the `require`_ attribute of the
+    individual test metadata.
 
-    * Architectures supported
-    * Disk and memory constraints
-    * Product relevancy
-    * Environment variables
+    .. _require: https://tmt.readthedocs.io/en/latest/spec/tests.html#require
 
-    Examples of metadata storage:
-
-    shell
-        Manual list of shell test cases
-    fmf
-        Flexible Metadata Format filter
-
-    For each ``repository`` the discover step should produce a
-    ``repository.yaml`` file with the list of tests in the
-    following format::
+    Store the list of aggregated tests with their corresponding
+    metadata in the ``tests.yaml`` file. The format should be a
+    dictionary of dictionaries structured in the following way::
 
         /test/one:
-            test: ./test.sh
-            path: /test/path
-            duration: 5m
+            summary: Short test summary.
+            description: Long test description.
+            contact: Petr Šplíchal <psplicha@redhat.com>
+            component: [tmt]
+            test: tmt --help
+            path: /test/path/
+            require: [package1, package2]
             environment:
-                X: 1
-                Y: 2
-                Z: 3
+                key1: value1
+                key2: value2
+                key3: value3
+            duration: 5m
+            enabled: true
+            result: respect
+            tag: [tag]
+            tier: 1
+            relevancy: |
+                distro < rhel-8: False
 
         /test/two:
-            test: ./test.sh
-            path: /test/path
-            duration: 10m
+            summary: Short test summary.
+            description: Long test description.
+            ...
+
+/shell:
+    summary: Provide a manual list of shell test cases
+    description:
+        List of test cases to be executed can be defined manually
+        directly in the plan as a list of dictionaries containing
+        test ``name``, actual ``test`` script and optionally a
+        ``path`` to the test.
+    example: |
+        discover:
+            how: shell
+            tests:
+            - name: /help/main
+              test: tmt --help
+            - name: /help/test
+              test: tmt test --help
+            - name: /help/smoke
+              test: ./smoke.sh
+              path: /tests/shell
+    implemented: /tmt/steps/discover/shell
+
+/fmf:
+    summary: Discover available tests using the fmf format
+    description: |
+        Use the `Flexible Metadata Format`_ to explore all
+        available tests in given repository. The following
+        parameters are supported:
+
+        url
+            Git repository containing the metadata tree.
+            Current git repository used by default.
+        ref
+            Branch, tag or commit specifying the desired git
+            revision. Defaults to the ``master`` branch if url
+            given or to the current ``HEAD`` if url not provided.
+        path
+            Path to the metadata tree root. Should be relative to
+            the git repository root if url provided, absolute
+            local filesystem path otherwise. By default ``.`` is
+            used.
+        test
+            List of test names or regular expressions used to
+            select tests by name.
+        filter
+            Apply advanced filter based on test metadata
+            attributes. See ``pydoc fmf.filter`` for more info.
+
+        See also the `fmf identifier`_ documentation.
 
-example: |
-    discover:
-        how: 'fmf'
-        filter: 'tier: 1'
+        .. _fmf identifier: https://fmf.readthedocs.io/en/latest/concept.html#identifiers
+        .. _Flexible Metadata Format: https://fmf.readthedocs.io/
+    example: |
+        discover:
+            how: fmf
 
-implemented: /tmt/steps/discover
+        discover:
+            how: fmf
+            url: https://github.com/psss/tmt
+            ref: master
+            path: /metadata/tree/path
+            test: [regexp]
+            filter: tier:1
+    implemented: /tmt/steps/discover/fmf

tests/discover/data/.fmf/version

@@ -0,0 +1 @@
+1

tests/discover/data/plans.fmf

@@ -0,0 +1,47 @@
+execute:
+    how: shell
+
+/fmf:
+    /url:
+        /ref:
+            /path:
+                discover:
+                    how: fmf
+                    url: https://github.com/psss/tmt
+                    ref: eae4d52
+                    path: /examples/together
+            /nopath:
+                discover:
+                    how: fmf
+                    url: https://github.com/psss/tmt
+                    ref: 5407fe5
+        /noref:
+            /path:
+                discover:
+                    how: fmf
+                    url: https://github.com/psss/tmt
+                    path: /examples/together
+
+            /nopath:
+                discover:
+                    how: fmf
+                    url: https://github.com/psss/tmt
+    /nourl:
+        /ref:
+            /path:
+                discover:
+                    how: fmf
+                    ref: eae4d52
+                    path: (overriden by --path)
+            /nopath:
+                discover:
+                    how: fmf
+                    ref: 5407fe5
+        /noref:
+            /path:
+                discover:
+                    how: fmf
+                    path: (overriden by --path)
+            /nopath:
+                discover:
+                    how: fmf

tests/discover/data/tests.fmf

@@ -0,0 +1,13 @@
+test: true
+
+/discover1:
+    summary: Discover one
+    tier: 1
+
+/discover2:
+    summary: Discover two
+    tier: 2
+
+/discover3:
+    summary: Discover three
+    tier: 3

tests/discover/filtering.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+. /usr/share/beakerlib/beakerlib.sh || exit 1
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlRun 'pushd data'
+        rlRun 'set -o pipefail'
+    rlPhaseEnd
+
+    rlPhaseStartTest "Filter by test name"
+        plan='plan --name fmf/nourl/noref/nopath'
+        discover='discover --how fmf --test discover1'
+        rlRun 'tmt run -d $discover $plan | tee output'
+        rlAssertGrep '1 test selected' output
+        rlAssertGrep '/tests/discover1' output
+        rlAssertNotGrep '/tests/discover2' output
+        rlAssertNotGrep '/tests/discover3' output
+    rlPhaseEnd
+
+    rlPhaseStartTest "Filter by advanced filter"
+        plan='plan --name fmf/nourl/noref/nopath'
+        discover='discover --how fmf --filter tier:1,2'
+        rlRun 'tmt run -d $discover $plan | tee output'
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep '/tests/discover1' output
+        rlAssertGrep '/tests/discover2' output
+        rlAssertNotGrep '/tests/discover3' output
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun 'rm -f output' 0 'Removing tmp file'
+        rlRun 'popd'
+    rlPhaseEnd
+rlJournalEnd

tests/discover/main.fmf

@@ -0,0 +1,17 @@
+path: /tests/discover
+tier: 1
+
+/options:
+    summary: Check that all supported options are present
+    test: ./options.sh
+
+/references:
+    summary: Verify that basic referencing works.
+    description:
+        Make sure that all combinations of 'url', 'ref' and 'path'
+        parameters work as expected.
+    test: ./references.sh
+
+/filtering:
+    summary: Test selection by name and advanced filter
+    test: ./filtering.sh

tests/discover/options.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+. /usr/share/beakerlib/beakerlib.sh || exit 1
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlRun 'pushd data'
+        rlRun 'set -o pipefail'
+    rlPhaseEnd
+
+    rlPhaseStartTest 'fmf help'
+        rlRun 'tmt run discover --help --how fmf | tee output'
+        rlAssertGrep 'Flexible Metadata Format' 'output'
+        for option in url ref path test filter; do
+            rlAssertGrep "--$option" output
+        done
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun 'rm -f output' 0 'Removing tmp file'
+        rlRun 'popd'
+    rlPhaseEnd
+rlJournalEnd

tests/discover/references.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+
+. /usr/share/beakerlib/beakerlib.sh || exit 1
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlRun 'pushd data'
+        rlRun 'set -o pipefail'
+    rlPhaseEnd
+
+    plan=fmf/nourl/noref/nopath
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertNotGrep 'Cloning into' output
+        rlAssertNotGrep 'Checkout ref' output
+        rlAssertGrep '3 tests selected' output
+        rlAssertGrep /tests/discover1 output
+        rlAssertGrep /tests/discover2 output
+        rlAssertGrep /tests/discover3 output
+    rlPhaseEnd
+
+    plan=fmf/nourl/noref/path
+    path=$(realpath .)
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover --how fmf --path $path plan --name $plan \
+            | tee output'
+        rlAssertNotGrep 'Cloning into' output
+        rlAssertNotGrep 'Checkout ref' output
+        rlAssertGrep '3 tests selected' output
+        rlAssertGrep /tests/discover1 output
+        rlAssertGrep /tests/discover2 output
+        rlAssertGrep /tests/discover3 output
+    rlPhaseEnd
+
+    plan=fmf/nourl/ref/nopath
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertNotGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*5407fe5' output
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep /tests/docs output
+        rlAssertNotGrep /tests/env output
+        rlAssertGrep /tests/ls output
+    rlPhaseEnd
+
+    plan=fmf/nourl/ref/path
+    path=$(realpath ../../../examples/together)
+    echo $path
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover --how fmf --path $path plan --name $plan \
+            | tee output'
+        rlAssertNotGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*eae4d52' output
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep /tests/full output
+        rlAssertGrep /tests/smoke output
+    rlPhaseEnd
+
+    plan=fmf/url/noref/nopath
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*master' output
+        rlAssertGrep '3 tests selected' output
+        rlAssertGrep /tests/docs output
+        rlAssertGrep /tests/env output
+        rlAssertGrep /tests/ls output
+    rlPhaseEnd
+
+    plan=fmf/url/noref/path
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*master' output
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep /tests/full output
+        rlAssertGrep /tests/smoke output
+    rlPhaseEnd
+
+    plan=fmf/url/ref/nopath
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*5407fe5' output
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep /tests/docs output
+        rlAssertNotGrep /tests/env output
+        rlAssertGrep /tests/ls output
+    rlPhaseEnd
+
+    plan=fmf/url/ref/path
+    rlPhaseStartTest $plan
+        rlRun 'tmt run -d discover plan --name $plan | tee output'
+        rlAssertGrep 'Cloning into' output
+        rlAssertGrep 'Checkout ref.*eae4d52' output
+        rlAssertGrep '2 tests selected' output
+        rlAssertGrep /tests/full output
+        rlAssertGrep /tests/smoke output
+        # Before the change was committed
+        rlRun 'tmt run -d discover --how fmf --ref eae4d52^ plan --name $plan \
+            2>&1 | tee output' 1
+        rlAssertGrep 'Metadata tree path .* not found.' output
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun 'rm -f output' 0 'Removing tmp file'
+        rlRun 'popd'
+    rlPhaseEnd
+rlJournalEnd

tmt/options.py

@@ -78,7 +78,7 @@ def get_help(self, context):
 
         def invoke(self, context):
             if self._method:
-                return context.forward(self._method)
+                return self._method.invoke(context)
             return super().invoke(context)
 
     return MethodCommand