teuthology-describe-tests: a way to examine a qa suite

[jdurgin]

The idea behind this is to help see what qa suites are covering by adding short descriptions or other fields to the yaml fragments in a suite.

The first column is similar to the output of the tree command. The other columns are extracted from comments in the yaml fragments based on the specified prefix.

An example annotated suite is here:

ceph/ceph-qa-suite@c3449c0

This could be extended (perhaps as part of the `teuthology-suite`` command) to output the full matrix of combinations rather than just the individual fragments.

Example output:

$ teuthology-tree -p desc ~/ceph-qa-suite/suites/rbd/basic
/home/jdurgin/ceph-qa-suite/suites/rbd/basic
+----------------------------------------------+-------------------------------------------------+
| path                                         | desc                                            |
+----------------------------------------------+-------------------------------------------------+
| ├── %                                        |                                                 |
| ├── base                                     |                                                 |
| │   └── install.yaml                         | install ceph                                    |
| ├── cachepool                                |                                                 |
| │   ├── none.yaml                            | default replicated pool                         |
| │   └── small.yaml                           | writeback replicated cache pool for rbd         |
| ├── clusters                                 |                                                 |
| │   └── fixed-1.yaml                         | single node cluster                             |
| ├── fs                                       |                                                 |
| │   └── btrfs.yaml                           | btrfs under the osds                            |
| ├── msgr-failures                            |                                                 |
| │   ├── few.yaml                             | infrequent network fault injection              |
| │   └── many.yaml                            | frequent network fault injection                |
| └── tasks                                    |                                                 |
|     ├── rbd_api_tests_old_format.yaml        | c/c++ librbd api tests with format 1 images     |
|     ├── rbd_cli_tests.yaml                   | basic rbd cli tool tests                        |
|     ├── rbd_cls_tests.yaml                   | low-level object class tests for cls_rbd.so     |
|     ├── rbd_lock_and_fence.yaml              | test manual locking and blacklisting via librbd |
|                                              | second line                                     |
|                                              | third line                                      |
|     └── rbd_python_api_tests_old_format.yaml | librbd python api tests                         |
+----------------------------------------------+-------------------------------------------------+

[yuriw]

I would suggest to use maybe two annotation tags to simplify the process: #desc: and #step:

#desc: - would be used to describe a new feature
#step: - would be used to describe a step of a test case/job/comment

We can use more annotations, of cause, if we want. But if we push for a limited amount of descriptions in suites that'd be a good start.

Also we can have wiki pages corresponding to the annotated ceph-qa-suites dirs and have them auto-updated when a new annotation/feature was added, so as results we can have features list and test plans and comments or all together if we wish.

Here is an example http://tracker.ceph.com/projects/ceph-releases/wiki/TEST_TO_BE_DELETED

[zmc]

@yuriw I'm not clear on what you mean by suggesting that #desc would 'describe a new feature' - it seems to me the intent of this is to provide a way to provide multi-purpose annotations to each yaml fragment. Nothing about 'desc' suggests specifically 'new feature' to me.

[zmc]

@jdurgin I like this idea very much! I wonder about the name, though. I don't have strong opinions about this, but I could see something like teuthology-describe-suite as being a little more indicative of what the new tool does.

[yuriw]

@zmc in my opinion, one of the purpose of this tool would be - building annotations for features that are tested by a particular yaml/facet configurations. So my hope was that # desc: can be used for this. However if it's not clear, I'd vote for two annotations:

# feature:
# step:

@zack if agree with the purpose, how would you name them?
@tmuthamizhan may have an oppinion and want to comment as well

[yuriw]

@zmc @jdurgin teuthology-annotations or teuthology-comments?

[vasukulkarni]

@jdurgin +1 for adding this, @zmc I think teutholgoy-describe-test might be better since its describe individual yaml test, If it can be made into high level function call, then pulpito can also benefit and a mouse hover can be used to describe the test.

[zmc]

@vasukulkarni Since this is implemented as yaml comments, none of it will make it into paddles/pulpito. If we want that, it'll have to take a different approach.

@yuriw I guess I don't have an opinion on the purpose of the feature. Perhaps we should get together with @jdurgin at some point to talk about use cases and basic implementation details if you both want?

[jdurgin]

@zmc @vasukulkarni I agree teuthology-describe-suite or similar is a better name - tree just came from the tree command, but it's not very descriptive.

Let's talk about the purpose of this more at the end of the infra standup tomorrow.

[jdurgin]

Updates: renamed to teuthology-describe-tests, added json/csv output, changed the input format to be part of the yaml (see the usage info for details), and added options to output the full matrix rather than just one row per path.

Some example output (using the wip-rbd-basic-tree ceph-qa-suite branch):
https://gist.github.com/jdurgin/09711d5923b583f60afc
https://gist.github.com/jdurgin/8380f42d6846ebed7304

@zmc to handle scheduling suites that have these 'meta' yaml elements, paddles needs to change to allow passing a job with a 'meta' attribute. HTTP 500 is given by the jobs controller right now if any unrecognized attribute is found. Could you help with this paddles change?

[yuriw]

@zmc @jdurgin here is passed job http://pulpito.ceph.com/teuthology-2015-12-03_08:52:57-upgrade:infernalis:older-infernalis-distro-basic-vps/ on this wip-teuthology-tree branch and results seem reported normally.

[jdurgin]

I was wrong about a paddles change being needed - I accidentally had two symlinked files that still had a 'description:' element. Changing those to 'meta:' made scheduling a suite work. So this is all ready for review I think.

[yuriw]

This how the table may look like on the latest code http://tracker.ceph.com/projects/ceph-releases/wiki/TEST_TO_BE_DELETED

[zmc]

@jdurgin this looks really good. I don't think any significant changes are needed. I'd ask two things though if you don't mind:

1. Please make the new files pass flake8
2. In docstrings, please use a newline after the """

I think that's it!

[jdurgin]

@zmc Thanks for looking, I added newlines to docstrings and made the new files pass flake8

[zmc]

@jdurgin Looks great, thanks!