Support tests and tasks from bundle under test

[johnsca]

Currently unable to test due to juju-solutions/python-libjuju#7 and https://bugs.launchpad.net/juju/+bug/1645917

[bcsaller]

This generally LGTM, thanks. The handling of the config file was a little confusing to me at first, but after seeing the rest take the comments or not.

[bcsaller]

Doesn't this favor thinking about the config file as residing as part of the matrix checkout? wouldn't cwd()/tests/matrix.yaml or just ./matrix.yaml be a better choice? Also does this interfere with the #!matrix style used in some of the tests? I don't know that we need to support that mode of operations but it would make having a single exec that BT can call simple.

If this should only represent the default tests then we might go about this another way, config_fiiles with type list(path) where the default includes the resource local one?

[johnsca]

If it's not bundled as part of the Matrix, how does someone who wants to run the standard suite against their bundle under development run it?

I was also under the impression that we wanted to merge the in-bundle suite with the standard suite, rather than running it as a separate instance of Matrix nor in place of the standard suite. So, I'm not sure what the benefit of the she-bang line for matrix tests would be.

[johnsca]

This also touches on the discussion we had about whether we wanted BT to be the thing that discovers and "combines" (in some way) the in-bundle test with the standard / default case, and my understanding was that we didn't want that.

If we do, we could rip out the bundle fetch code entirely and always assume we're deploying a local bundle, with BT being responsible for fetching it and running Matrix from within the bundle directory to ensure the right paths.

[bcsaller]

I appreciate the run_in_executor hurdle you jumped here (and we can keep it) but the setup code here doesn't strictly need to be async as nothing else can run when this is called.

[johnsca]

Presumably, we don't want the UI or keyboard handling to hang while it's talking to the charm store. Apparently, the CS is hitting 10s timeouts just for metadata requests occasionally, so I'm a bit wary of it being blocking.

[bcsaller]

Merging lists is almost never a good idea, order and uniqueness are key reasons for using lists, here we'd have to give up one or the other.

[bcsaller]

Nice, thanks

[bcsaller]

LGTM, thank you

[petevg]

This gets pretty complex, so it would be nice to see some examples in this doc string.

Basically, I'm looking for a "tar xvzf" or "ps aux" -- things that a newbie can run and have some success with, even if they don't understand all the options available to them.

[johnsca]

Rendered --help:

usage: matrix [-h] [-l LOG_LEVEL] [-L [LOG_NAME [LOG_NAME ...]]]
              [-f [LOG_FILTER [LOG_FILTER ...]]] [-s {tui,raw}] [-x FILENAME]
              [-i INTERVAL] [-p PATH] [-D] [-B]
              [-t [TEST_PATTERN [TEST_PATTERN ...]]] [-g GLITCH_PLAN]
              [-n GLITCH_NUM] [-o GLITCH_OUTPUT]
              [additional_suites [additional_suites ...]]

Run a local bundle through a suite of tests to ensure that it can handle the
types of operations and failures that are common for all deployments.

positional arguments:
  additional_suites     Additional suites to be merged with the default suite
                        before running

optional arguments:
  -h, --help            show this help message and exit
  -l LOG_LEVEL, --log-level LOG_LEVEL
  -L [LOG_NAME [LOG_NAME ...]], --log-name [LOG_NAME [LOG_NAME ...]]
  -f [LOG_FILTER [LOG_FILTER ...]], --log-filter [LOG_FILTER [LOG_FILTER ...]]
  -s {tui,raw}, --skin {tui,raw}
  -x FILENAME, --xunit FILENAME
                        Create an XUnit report file
  -i INTERVAL, --interval INTERVAL
  -p PATH, --path PATH  Path to local bundle to test (defaults to current
                        directory)
  -D, --no-default-suite
                        Do not include the default suite as the first suite
  -B, --no-bundle-suite
                        Do not include the suite provided by the bundle
  -t [TEST_PATTERN [TEST_PATTERN ...]], --test_pattern [TEST_PATTERN [TEST_PATTERN ...]]
                        Pattern for selecting which tests from the suite(s)
                        are run
  -g GLITCH_PLAN, --glitch_plan GLITCH_PLAN
  -n GLITCH_NUM, --glitch_num GLITCH_NUM
  -o GLITCH_OUTPUT, --glitch_output GLITCH_OUTPUT

A default suite of tests will always be run, unless -D is given.  Additional
suites can be passed in as arguments.  Additionally, the bundle will be
checked for a tests/matrix.yaml file which will be included as an additional
suite if present.  All suites will be merged together, and suites containing
tests with the same name as a test in a previous suite will override that
test.

Examples:

    Run all normal tests (default suite and, if present, ./tests/matrix.yaml)
    on the bundle at ./:

        $ matrix

    Run only the default suite (ignore ./tests/matrix.yaml):

        $ matrix -B

    Run only ~/foo/tests/matrix.yaml on the bundle at ~/foo:

        $ matrix -Dp ~/foo

    Run all normal tests plus ./tests/matrix_extra.yaml on the bundle at ./:

        $ matrix tests/matrix_extra.yaml

    Run only ./tests/matrix_extra.yaml:

        $ matrix -DB tests/matrix_extra.yaml

[petevg]

LGTM/+1