Skip to content

Commit

Permalink
Update and refine the readme
Browse files Browse the repository at this point in the history
  • Loading branch information
NickVolynkin committed Feb 14, 2022
1 parent 2604c46 commit eeeba6a
Showing 1 changed file with 80 additions and 28 deletions.
108 changes: 80 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,86 @@

[![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run)

### Test Suite

Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini`
file. `suite.ini` is basic ini-file, that consists of one section `default`,
and a number of fields:

* `core`
* `description` - Test Suite description
* `script` - shebang file to start tarantool with
* disables:
* `disabled` - tests that must be skipped
* `release_disabled` - tests that must be skipped when Tarantool has been
builded with `Release`
* `valgrind_disabled` - tests that must be skipped when Valgrind is enabled
* `lua_libs` - paths for lua files, that should be copied into the folder,
where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
lua/2.lua`)
* `long_run` - mark tests as long, enabled only with `--long` option (delimited
with the space, e.g. `long_run=t1.test.lua t2.test.lua`)
* `config` - test configuration file name

Field `core` must be one of:

* `tarantool` - Test-Suite for Functional Testing
* `app` - Another functional Test-Suite
* `unittest` - Unit-Testing Test Suite

### Test
## Test suite configuration

Test suite is a bunch of tests located in a subdirectory (recursively) with a `suite.ini`
file. The `suite.ini` is a basic ini-file with one section `[default]`.

```ini
[default]
core = luatest
description = Example test suite using luatest
...
```

All list values in `suite.ini` are delimited by spaces.

A `suite.ini` file can have the following fields.

### General configuration

* `core` — major testing dependency or method
Should have one of the following values:

* `tarantool` — test suite used for functional testing
* `app` — TAP-tests, another functional test suite
* `unittest` — unit testing test suite
* `luatest` — test suite using luatest library

* `description` — test suite description
* `script` (optional) — shebang file to start tarantool with

### Skipping tests
A number of fields are used to disable (skip) certain tests:

* `disabled` — list of tests that should be skipped
* `release_disabled` — list of tests that should be skipped when Tarantool is
built with `Release`
* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled
* `long_run` — mark tests as long, such tests will run only with `--long` option.

```ini
long_run = long_run=t1.test.lua t2.test.lua
```

### Other parameters

* `lua_libs` — paths for lua files, that should be copied into the folder,
where server is started. For example:

```ini
lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua
```

* `config` — name of a test configuration file
* `show_reproduce_content` (optional, `True` by default) — ?
* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel

* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets
* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets for iproto.
This parameter is specific to `core = app`.

### Fragile tests

Tests which fail sometimes due to external reasons can be marked as fragile (flaky).
Test-run will retry each test if it fails.
It's recommended to add a list of related issues in the
[tarantool/tarantool](https://github.com/tarantool/tarantool) repository.

```ini
fragile = {
"retries": 10,
"tests": {
"tarantoolctl.test.lua": {
"issues": [ "gh-5059", "gh-5346" ],
},
{...},
{...},
}
}
```

## Test

Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have
skip condition file `*.skipcond`. On first run (without `.result`) `.result`
Expand Down

0 comments on commit eeeba6a

Please sign in to comment.