Skip to content

Commit

Permalink
Improve reporting from failed luatest tests
Browse files Browse the repository at this point in the history
This patch adds writing the `reject` file contents of failed luatest
tests to the console output like it is done for TAP tests. This is
needed because it contains full information about failed tests.

Before:

    ======================================================================================
    WORKR TEST                                            PARAMS          RESULT
    ---------------------------------------------------------------------------------
    [001] vinyl-luatest/update_optimize_test.lua
    [001] not ok 1 vinyl-luatest.update_optimize.test_optimize_one_index #
    [001] not ok 3 vinyl-luatest.update_optimize.test_optimize_UPDATE_with_field_num_more_than_64 #
    [001] not ok 4 vinyl-luatest.update_optimize.test_optimize_update_does_not_skip_entire_key_during_dump #
    [001] Rejected result file: /tmp/tnt/rejects/vinyl-luatest/update_optimize.reject
    [001] [ fail ]
    [001] Worker "001_vinyl-luatest" got failed test; restarted the server
    [002] vinyl-luatest/gh_6568_replica_initial_join_rem>
    [002] not ok 1 gh-6568-replica-initial-join-removal-of-compacted-run-files.test_replication_compaction_cleanup #
    [002] Rejected result file: /tmp/tnt/rejects/vinyl-luatest/gh_6568_replica_initial_join_removal_of_compacted_run_files.reject
    [002] [ fail ]
    [002] Worker "002_vinyl-luatest" got failed test; restarted the server
    ---------------------------------------------------------------------------------
    ...

Now:

    ======================================================================================
    WORKR TEST                                            PARAMS          RESULT
    ---------------------------------------------------------------------------------
    [001] vinyl-luatest/gh_6568_replica_initial_join_rem>                 [ fail ]
    [001] Test failed! Output from reject file /tmp/tnt/rejects/vinyl-luatest/gh_6568_replica_initial_join_removal_of_compacted_run_files.reject:
    [001] TAP version 13
    [001] 1..1
    [001] # Started on Wed Feb  9 00:25:23 2022
    [001] # Starting group: gh-6568-replica-initial-join-removal-of-compacted-run-files
    [001] not ok 1	gh-6568-replica-initial-join-removal-of-compacted-run-files.test_replication_compaction_cleanup
    [001] #   ...ica_initial_join_removal_of_compacted_run_files_test.lua:72: the total actual size of run files directories of replica must not differ from box.stat.vinyl().disk.data by more than 100.0%
    [001] #   Assertion failed: 3.8947799385875 <= 1
    [001] #   stack traceback:
    [001] #   	...ica_initial_join_removal_of_compacted_run_files_test.lua:72: in function 'gh-6568-replica-initial-join-removal-of-compacted-run-files.test_replication_compaction_cleanup'
    [001] #   	...
    [001] #   	[C]: in function 'xpcall'
    [001] # Ran 1 tests in 0.651 seconds, 0 succeeded, 1 failed
    [001] Worker "001_vinyl-luatest" got failed test; restarted the server
    [001] vinyl-luatest/update_optimize_test.lua                          [ fail ]
    [001] Test failed! Output from reject file /tmp/tnt/rejects/vinyl-luatest/update_optimize.reject:
    [001] TAP version 13
    [001] 1..6
    [001] # Started on Wed Feb  9 00:25:24 2022
    [001] # Starting group: vinyl-luatest.update_optimize
    [001] not ok 1	vinyl-luatest.update_optimize.test_optimize_one_index
    [001] #   ...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:133: expected: {{2, 3, 4, 5, 6, 20}, {3, 4, 5, 6, 7}, {4, 5, 6, 7, 8}, {1, 2, 20, 4, 10, 1}}
    [001] #   actual: {[2, 3, 4, 5, 6, 20], [3, 4, 5, 6, 7], [4, 5, 6, 7, 8], [1, 2, 20, 4, 10]}
    [001] #   stack traceback:
    [001] #   	...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:36: in function 'vinyl-luatest.update_optimize.test_optimize_one_index'
    [001] #   	...
    [001] #   	[C]: in function 'xpcall'
    [001] ok     2	vinyl-luatest.update_optimize.test_optimize_two_indexes
    [001] not ok 3	vinyl-luatest.update_optimize.test_optimize_UPDATE_with_field_num_more_than_64
    [001] #   ...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:338: expected: {1, 2, 3, 4, 5, 5}
    [001] #   actual: [1, 2, 3, 4, 5]
    [001] #   stack traceback:
    [001] #   	...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:306: in function 'vinyl-luatest.update_optimize.test_optimize_UPDATE_with_field_num_more_than_64'
    [001] #   	...
    [001] #   	[C]: in function 'xpcall'
    [001] not ok 4	vinyl-luatest.update_optimize.test_optimize_update_does_not_skip_entire_key_during_dump
    [001] #   ...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:436: expected: {{10, 100, 1000, 10000, 100000, 1}, {20, 200, 2000, 20000, 200000, 2, 5}}
    [001] #   actual: {[10, 100, 1000, 10000, 100000, 1], [20, 200, 2000, 20000, 200000, 2]}
    [001] #   stack traceback:
    [001] #   	...ce/tarantool/test/vinyl-luatest/update_optimize_test.lua:394: in function 'vinyl-luatest.update_optimize.test_optimize_update_does_not_skip_entire_key_during_dump'
    [001] #   	...
    [001] #   	[C]: in function 'xpcall'
    [001] ok     5	vinyl-luatest.update_optimize.test_key_uniqueness_not_checked_if_indexed_fields_not_updated
    [001] ok     6	vinyl-luatest.update_optimize.test_no_phantom_tuples_in_secondary_index
    [001] # Ran 6 tests in 0.388 seconds, 3 succeeded, 3 failed
    [001] Worker "001_vinyl-luatest" got failed test; restarted the server
    ---------------------------------------------------------------------------------
    ...

Closes #319
  • Loading branch information
ylobankov committed Feb 8, 2022
1 parent 2604c46 commit 0156fb4
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions lib/luatest_server.py
Expand Up @@ -10,6 +10,7 @@
from lib.sampler import sampler
from lib.server import Server
from lib.tarantool_server import Test
from lib.tarantool_server import TestExecutionError
from lib.tarantool_server import TarantoolServer


Expand Down Expand Up @@ -50,6 +51,8 @@ def execute(self, server):
proc = Popen(command, cwd=project_dir, stdout=PIPE, stderr=STDOUT)
sampler.register_process(proc.pid, self.id, server.name)
sys.stdout.write_bytes(proc.communicate()[0])
if proc.returncode != 0:
raise TestExecutionError


class LuatestServer(Server):
Expand Down

0 comments on commit 0156fb4

Please sign in to comment.