Skip to content

Commit

Permalink
Merge pull request #58 from gregoil/add_rotest_console
Browse files Browse the repository at this point in the history 
Add rotest console and tests discoverer
  • Loading branch information
@gregoil
gregoil committed Jun 14, 2018
2 parents 48c13fd + 7717afe commit 3a6ac72
Show file tree
Hide file tree
Showing 26 changed files with 969 additions and 511 deletions.
77 changes: 45 additions & 32 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ Rotest
:alt: Read the Docs (version)
:target: http://rotest.readthedocs.io/en/stable/

`Watch the demo <https://asciinema.org/a/u3B3aMmkipUDLSgTiv1thiBpP>`_

Rotest is a resource oriented testing framework, for writing system or
integration tests.

Expand All @@ -44,7 +46,7 @@ For a complete step-by-step explanation about the framework, you can read
our documentation at `Read The Docs <http://rotest.rtfd.io>`_. If you just want
to see how it looks, read further.

For our example, let's look at an example for a `Calculator` resource:
For our example, let's look at an example for a ``Calculator`` resource:

.. code-block:: python
Expand Down Expand Up @@ -83,7 +85,7 @@ For our example, let's look at an example for a `Calculator` resource:
self._rpyc.close()
self._rpyc = None
The `CalculatorData` class is a standard Django model that exposes IP
The ``CalculatorData`` class is a standard Django model that exposes IP
address of the calculator machine through the data attribute.
Also, we're using `rpyc` for automating the access to those machines. Except
from that, it's easy to notice how the `connect` method is making the
Expand All @@ -94,8 +96,8 @@ Now, an example for a test:

.. code-block:: python
from rotest.core.runner import main
from rotest.core.case import TestCase
from rotest import main
from rotest.core import TestCase
class SimpleCalculationTest(TestCase):
Expand All @@ -106,40 +108,51 @@ Now, an example for a test:
if __name__ == "__main__":
main(SimpleCalculationTest)
main()
The test can include the `setUp` and `tearDown` methods of `unittest` as
The test may include the ``setUp`` and ``tearDown`` methods of `unittest` as
well, and it differs only in the request for resources.

Following, those are the options exposed when running the test:

.. code-block:: console
$ python test.py --help
Usage: test.py [options]
$ rotest -h
Run tests in a module or directory.
Usage:
rotest [<path>...] [options]
Options:
-h, --help show this help message and exit
-c CONFIG_PATH, --config-path=CONFIG_PATH
Tests' configuration file path
-s, --save-state Enable save state
-d DELTA_ITERATIONS, --delta-iterations=DELTA_ITERATIONS
Enable run of unsuccessful tests only, enter the
number of times the failed tests should run
-p PROCESSES, --processes=PROCESSES
Use multiprocess test runner
-o OUTPUTS, --outputs=OUTPUTS
Output handlers separated by comma. Options: dots,
xml, full, remote, tree, excel, db, artifact,
signature, loginfo, logdebug, pretty
-f FILTER, --filter=FILTER
Run only tests that match the filter expression, e.g
"Tag1* and not Tag13"
-n RUN_NAME, --name=RUN_NAME
Assign run name
-l, --list Print the tests hierarchy and quit
-F, --failfast Stop the run on first failure
-D, --debug Enter ipdb debug mode upon any test exception
-S, --skip-init Skip initialization and validation of resources
-r RESOURCES, --resources=RESOURCES
Specific resources to request by name
-h, --help
Show help message and exit.
--version
Print version information and exit.
-c <path>, --config <path>
Test configuration file path.
-s, --save-state
Enable saving state of resources.
-d <delta-iterations>, --delta <delta-iterations>
Enable run of failed tests only - enter the number of times the
failed tests should be run.
-p <processes>, --processes <processes>
Use multiprocess test runner - specify number of worker
processes to be created.
-o <outputs>, --outputs <outputs>
Output handlers separated by comma.
-f <query>, --filter <query>
Run only tests that match the filter expression,
e.g. 'Tag1* and not Tag13'.
-n <name>, --name <name>
Assign a name for current launch.
-l, --list
Print the tests hierarchy and quit.
-F, --failfast
Stop the run on first failure.
-D, --debug
Enter ipdb debug mode upon any test exception.
-S, --skip-init
Skip initialization and validation of resources.
-r <query>, --resources <query>
Specify resources to request by attributes,
e.g. '-r res1.group=QA,res2.comment=CI'.
136 changes: 91 additions & 45 deletions docs/cli_options/client_options.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,46 +4,94 @@ Client Options

.. program:: rotest

Running tests
=============

Running tests can be done in the following ways:

* Using the `rotest` command:

.. code-block:: console
$ rotest [PATHS]... [OPTIONS]
The command can get every path - either files or directories. Every directory
will be recursively visited for finding more files. If no path was given, the
current working directory will be selected by default.

* Calling the :func:`rotest.main` function:

.. code-block:: python
from rotest import main
from rotest.core import TestCase
class Case(TestCase):
def test(self):
pass
if __name__ == "__main__":
main()
Then, this same file can be ran:

.. code-block:: console
$ python test_file.py [OPTIONS]
Getting Help
============

.. option:: -h, --help

Show a help message and exit.

First, and most important, using the help options :option:`-h` or
:option:`--help`:
If you're not sure what you can do, the help options :option:`-h` and
:option:`--help` are here to help:

.. code-block:: console
$ python some_test_file.py -h
Usage: some_test_file.py [options]
$ rotest -h
Run tests in a module or directory.
Usage:
rotest [<path>...] [options]
Options:
-h, --help show this help message and exit
-c CONFIG_PATH, --config-path=CONFIG_PATH
Tests' configuration file path
-s, --save-state Enable save state
-d DELTA_ITERATIONS, --delta-iterations=DELTA_ITERATIONS
Enable run of failed tests only, enter the number of
times the failed tests should run
-p PROCESSES, --processes=PROCESSES
Use multiprocess test runner
-o OUTPUTS, --outputs=OUTPUTS
Output handlers separated by comma. Options: dots,
xml, full, remote, tree, excel, db, artifact,
signature, loginfo, logdebug, pretty
-f FILTER, --filter=FILTER
Run only tests that match the filter expression, e.g
"Tag1* and not Tag13"
-n RUN_NAME, --name=RUN_NAME
Assign run name
-l, --list Print the tests hierarchy and quit
-F, --failfast Stop the run on first failure
-D, --debug Enter ipdb debug mode upon any test exception
-S, --skip-init Skip initialization and validation of resources
-r RESOURCES, --resources=RESOURCES
Specific resources to request by name
-h, --help
Show help message and exit.
--version
Print version information and exit.
-c <path>, --config <path>
Test configuration file path.
-s, --save-state
Enable saving state of resources.
-d <delta-iterations>, --delta <delta-iterations>
Enable run of failed tests only - enter the number of times the
failed tests should be run.
-p <processes>, --processes <processes>
Use multiprocess test runner - specify number of worker
processes to be created.
-o <outputs>, --outputs <outputs>
Output handlers separated by comma.
-f <query>, --filter <query>
Run only tests that match the filter expression,
e.g. 'Tag1* and not Tag13'.
-n <name>, --name <name>
Assign a name for current launch.
-l, --list
Print the tests hierarchy and quit.
-F, --failfast
Stop the run on first failure.
-D, --debug
Enter ipdb debug mode upon any test exception.
-S, --skip-init
Skip initialization and validation of resources.
-r <query>, --resources <query>
Specify resources to request by attributes,
e.g. '-r res1.group=QA,res2.comment=CI'.
Listing and Filtering
=====================
Expand All @@ -52,7 +100,7 @@ Listing and Filtering

Print the tests hierarchy and quit.

.. option:: -f FILTER, --filter FILTER
.. option:: -f <query>, --filter <query>

Run only tests that match the filter expression, e.g. "Tag1* and not Tag13".

Expand All @@ -61,7 +109,7 @@ Next, you can print a list of all the tests that will be run, using

.. code-block:: console
$ python some_test_file.py -l
$ rotest some_test_file.py -l
CalculatorSuite []
| CasesSuite []
| | PassingCase.test_passing ['BASIC']
Expand Down Expand Up @@ -97,7 +145,7 @@ of tests using the :option:`-f` or :option:`--filter` options:
.. code-block:: console
:emphasize-lines: 13,17,21,26
$ python some_test_file.py -f FLOW -l
$ rotest some_test_file.py -f FLOW -l
CalculatorSuite []
| CasesSuite []
| | PassingCase.test_passing ['BASIC']
Expand Down Expand Up @@ -135,7 +183,7 @@ insensitive):
.. code-block:: console
:emphasize-lines: 4-6,9-10,12
$ python some_test_file.py -f "basic and not skipped*" -l
$ rotest some_test_file.py -f "basic and not skipped*" -l
CalculatorSuite []
| CasesSuite []
| | PassingCase.test_passing ['BASIC']
Expand Down Expand Up @@ -176,7 +224,7 @@ first failure:

.. code-block:: console
$ python some_test_file.py --failfast
$ rotest some_test_file.py --failfast
CalculatorSuite
CasesSuite
PassingCase.test_passing ... OK
Expand Down Expand Up @@ -215,7 +263,7 @@ exceptions are raised at the top level of the code:

.. code-block:: console
$ python some_test_file.py --debug
$ rotest some_test_file.py --debug
AnonymousSuite
FailingCase.test ...
Traceback (most recent call last):
Expand Down Expand Up @@ -257,18 +305,16 @@ Once in the debugging session, you can do any of the following:
Retrying Tests
==============

.. option:: -d DELTA_ITERATIONS,
--delta DELTA_ITERATIONS
--delta-iterations DELTA_ITERATIONS
.. option:: -d <delta-iterations>, --delta <delta-iterations>

Rerun test a specified amount of times until it passes.

In case you have flaky tests, you can automatically rerun a test until getting
a success result. Use options :option:`--delta-iterations` or :option:`-d`:
a success result. Use options :option:`--delta` or :option:`-d`:

.. code-block:: console
$ python some_test_file.py --delta-iterations 2
$ rotest some_test_file.py --delta 2
AnonymousSuite
FailingCase.test ... FAIL
Traceback (most recent call last):
Expand Down Expand Up @@ -302,7 +348,7 @@ a success result. Use options :option:`--delta-iterations` or :option:`-d`:
Running Tests in Parallel
=========================

.. option:: -p PROCESSES, --processes PROCESSES
.. option:: -p <processes>, --processes <processes>

Spawn specified amount of processes to execute tests.

Expand Down Expand Up @@ -337,7 +383,7 @@ The request is of the form:

.. code-block:: console
$ python some_test_file.py --resources <query-for-resource-1>,<query-for-resource-2>,...
$ rotest some_test_file.py --resources <query-for-resource-1>,<query-for-resource-2>,...
As an example, let's suppose we have the following test:

Expand All @@ -354,24 +400,24 @@ You can request resources by their names:

.. code-block:: console
$ python some_test_file.py --resources res1=name1,res2=name2
$ rotest some_test_file.py --resources res1=name1,res2=name2
Alternatively, you can make more complex queries:

.. code-block:: console
$ python some_test_file.py --resources res1.group.name=QA,res2.comment=nightly
$ rotest some_test_file.py --resources res1.group.name=QA,res2.comment=nightly
Activating Output Handlers
==========================

.. option:: -o OUTPUTS, --outputs OUTPUTS
.. option:: -o <outputs>, --outputs <outputs>

To activate an output handler, use options :option:`-o` or :option:`--outputs`,
with the output handlers separated using commas:

.. code-block:: console
$ python some_test_file.py --outputs excel,logdebug
$ rotest some_test_file.py --outputs excel,logdebug
For more about output handlers, read on :ref:`output_handlers`.

0 comments on commit 3a6ac72

Please sign in to comment.