Easy way to use box.cfg() in main testing code

[Totktonada]

Forewords

Here I want to more or less precisely declare a particular problem that was highlighted several times by tarantool developers. I have no good vision how it should be resolved.

As result the following things are intentional:

• This issue may be tightly connected with others (such as Sandbox tests #122).
• A discussion about the best solution is open (it should be general enough and convenient in use).

Background

Four test types in test-run

In test-run (the test framework that we use in tarantool/tarantool) we have four types of tests (named below according to the `core` field in the `suite.ini` file):

• core = tarantool: read a test file and feed it line-by-line¹ to tarantool console. A text of requests and responses is squashed and compared against a reference file. Often called diff-based based testing.
• core = app: just Lua script. Usually produces TAP13 output using built-in tap module (and so often called tap-based testing), which is validated by test-run without a reference file. If a test produces non-TAP13 output, stdout is compared against a reference file.
• core = unittest: just executable files. May be TAP13 compliant (after some work) or diff based.
• core = luatest: calls luatest <...> /path/to/foo_test.lua, where luatest is configured to produce TAP13 output. (Reasoning.)

We have a lot of single node tests, which call box.cfg() inside, and works with something related to the database. There are core = tarantool and core = app tests, which just call box.cfg() with necessary parameters and run test cases with box.<...> calls.

We can't do the same with core = luatest test cases and this is the problem I want to raise here.

The problem

If we'll call box.cfg() within main testing code, we'll meet several drawbacks.

The drawbacks

1. If a test configures a non-dynamic box.cfg() option², we unable to run it using pure luatest invocation³. It'll work under test-run, because each test is executed using a separate luatest call.
2. The same problem if we will want to configure box per group / per test case that involves a non-dynamic option (but here it'll fail under test-run too).
3. *.snap, *.xlog and other database files will be placed in the current directory (not some temporary one as for other test types).
4. One test may see database / runtime state that a previous test left (only database state for running under test-run, because nobody cleans up the database files; runtime state as well when run under luatest, because all tests are processed with a single tarantool process). See also Sandbox tests #122 and my comment.
5. The same problem with isolation of test groups / test cases.

In other words, current luatest design push a developer to spawn an instance and interact with it using call/eval, if box.cfg() is prerequisite for a test. It requires to write extra code and makes the testing code less readable as result.

I think we should revisit this design and provide a luatest-standard way to use box.cfg() within a test.

Possibly we should take care to runtime isolation (#122) as well, because those things look tightly connected.

I would also say that we should design the functionality in such way that a developer (which writes a test) may choose, whether isolation should be performed on the test level, on the test group level, on the test case level (or opt out from isolation at all), and choose best compromise between isolation strictness and test timing (with knowledge the test).

Footnotes

1. Multiline statements should be marked explicitly either by set delimiter test-run command or using a backslash at end of line. ↩

2. A box.cfg() option that does not allow runtime change after first box initialization (such as vinyl_page_size). ↩

3. To be precise: the test will fail with Can't set option '<...>' dynamically if it is not the first test that calls box.cfg(). ↩

[ylobankov]

It was decided to proceed with #245 instead of resolving this issue.