Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Understand why it takes so long to pass the tests #690

Closed
LeonidVas opened this issue Feb 17, 2022 · 2 comments · Fixed by #726
Closed

Understand why it takes so long to pass the tests #690

LeonidVas opened this issue Feb 17, 2022 · 2 comments · Fixed by #726
Assignees
@better0fdead
Labels
code health Improve code readability, simplify maintenance and so on

Comments

@LeonidVas
Copy link
Collaborator

Now it takes a very long time to pass the tests (> 1 hour).
What's needed:

  • to understand why
  • suggest solutions
@kyukhin kyukhin added the teamE label Feb 25, 2022
@LeonidVas LeonidVas added the code health Improve code readability, simplify maintenance and so on label Jun 7, 2022
@LeonidVas
Copy link
Collaborator Author

Some thoughts about possible fixes (to make the situation with tests more acceptable):

  • restart a failed test several times (for flaky tests)
  • split tests to fast / slow and make it possible to run them separately (run slow tests only if fast ones were successfully completed in CI)

@better0fdead
Copy link
Contributor

better0fdead commented Jun 29, 2022 •
edited
Loading

Made a table with information about testing times, suggestions and flaky tests.
cli_test.csv

Preview with longest tests:

Time Type Test Why so long? Project flaky tests: project_with_topology_and_vshard Failed to bootstrap vshard: [string "local function func(...) local ' 'cartridge = re..."]:1: No remotes with role "vshard-router" available\n')
131.91s setup test/integration/pack/test_pack_docker.py::test_pack докер project_without_dependencies test_setup
86.56s call test/integration/cli/test_version.py::test_duplicate_cartridge_no_rocks_flag[--version] build, инсталл картридж project_with_cartridge test_setup_bootstrap_vshard Failed to bootstrap vshard: [string "local function func(...) local cartridge = re..."]:1: No remotes with role "vshard-router" available
82.08s call test/integration/cli/test_version.py::test_duplicate_cartridge_no_rocks_flag[version] build, инсталл картридж project_with_cartridge custom_admin_running_instances : Failed downloading http://rocks.tarantool.org/checks-3.1.0-1.all.rock
74.37s call test/integration/cli/test_version.py::test_duplicate_cartridge_no_rocks_flag[-v] build, инсталл картридж project_with_cartridge test_default_application Failed to bootstrap vshard: [string "local function func(...) local cartridge = re..."]:1: No remotes with role "vshard-router" available
60.89s call test/integration/cli/test_version.py::test_duplicate_rocks[--version] build, инсталл graphql project_with_cartridge test_start_with_timeout Application dir does not contain ".rocks" directory. Make sure you ran "cartridge build" before running "cartridge start"
60.46s call test/integration/create/test_create.py::test_project build, install deps, luachek/test default_project test_join Failed to join instances: Failed to edit topology: [string "local function func(...) local cartridge = re..."]:1: "localhost:3304": Replication setup failed, instance orphaned
57.04s call test/integration/cli/test_version.py::test_duplicate_rocks[-v] build, инсталл graphql project_with_cartridge e2e
56.31s call test/integration/cli/test_version.py::test_duplicate_rocks[version] build, инсталл graphql project_with_cartridge
49.84s call test/integration/bench/test_bench.py::test_bench wait_for_connect(<15), bench ---
47.82s setup test/integration/failover/test_failover.py::test_setup_etcd2_failover configure failover project_with_topology_and_vshard
45.85s setup test/integration/failover/test_status.py::test_status_disabled configure failover project_with_topology_and_vshard
45.43s setup test/integration/failover/test_status.py::test_status_eventual configure failover project_with_topology_and_vshard
44.52s call test/integration/replicasets/test_setup.py::test_default_application build project, setup replica project_with_cartridge
44.51s call test/integration/repair/test_repair_reload.py::test_repair_reload_remove_instance build project project_with_cartridge
44.35s setup test/integration/failover/test_failover.py::test_disable_failover_from_sub_command configure failover project_with_cartridge Паралелить -
43.74s call test/integration/repair/test_repair_reload.py::test_repair_reload_old_cartridge[None] build project project_with_cartridge
43.43s setup test/integration/admin/test_default.py::test_default_admin_func[instance] conn default_admin_running_instances['project']
43.37s call test/integration/cli/test_version.py::test_version_command_with_rocks[version] build project project_with_cartridge
42.98s call test/integration/repair/test_repair_reload.py::test_repair_reload_set_leader build project project_with_cartridge
42.84s call test/integration/cli/test_version.py::test_version_command_with_rocks[--version] build project project_with_cartridge
42.72s setup test/integration/failover/test_status.py::test_status_stateful_stateboard configure failover project_with_topology_and_vshard
42.46s setup test/integration/failover/test_failover.py::test_failover_disabled_command configure failover project_with_topology_and_vshard
42.18s setup test/integration/replicasets/test_bootstrap_vshard.py::test_bootstrap[new-cartridge] bootstrap vshard x2 project_with_vshard_replicasets.project
42.14s setup test/integration/failover/test_failover.py::test_setup_eventual_failover configure failover project_with_topology_and_vshard
42.11s setup test/integration/admin/test_default.py::test_default_admin_func[find-socket] conn default_admin_running_instances['project']
42.10s call test/integration/repair/test_repair_reload.py::test_repair_reload_set_uri build project project_with_cartridge
41.91s setup test/integration/replicasets/test_list_replicasets.py::test_default_application setup replicasets default_project_with_instances.project
41.76s call test/integration/cli/test_version.py::test_version_command_with_project[version] build project project_with_cartridge
41.45s setup test/integration/replicasets/test_bootstrap_vshard.py::test_bootstrap[old-cartridge] bootstrap vshard project_with_vshard_replicasets.project
41.06s setup test/integration/failover/test_status.py::test_status_stateful_etcd2 configure failover project_with_topology_and_vshard
40.91s call test/integration/repair/test_repair_reload.py::test_repair_reload_old_cartridge[1.2.0] build project project_with_cartridge
39.15s setup test/integration/connect/test_connect.py::test_bad_uri connect project_with_instances.project
39.06s setup test/integration/failover/test_failover.py::test_default_app_stateboard_failover configure failover project_with_topology_and_vshard
39.02s call test/integration/cli/test_version.py::test_version_command_with_project[--version] build project project_with_cartridge
38.90s call test/integration/cli/test_version.py::test_version_command_with_rocks[-v] build project project_with_cartridge
38.27s setup test/integration/admin/test_default.py::test_default_admin_func[connect] conn default_admin_running_instances['project']
37.20s call test/integration/cli/test_version.py::test_version_command_with_project[-v] build project project_with_cartridge

better0fdead added a commit that referenced this issue Jul 4, 2022
@better0fdead
workflow: separate integration tests from other checks
b9f2b3e 
Separated integration tests from linter, unit, e2e and examples checks.

Part of #690
better0fdead added a commit that referenced this issue Jul 4, 2022
@better0fdead
workflow: separate integration tests from other checks
4b4d0d2 
Separated integration tests from linter, unit, e2e and examples checks.

Part of #690
@better0fdead better0fdead mentioned this issue Jul 5, 2022
Merged
@better0fdead better0fdead linked a pull request Jul 5, 2022 that will close this issue
tests: make integration testing faster #726
Merged
better0fdead added a commit that referenced this issue Jul 14, 2022
@better0fdead
workflow: separate integration tests from other checks
e8f3dee 
Separated integration tests from linter, unit, e2e and examples checks,
because integration tests run longer then all other.
Result: 15% faster testing.

Part of #690
LeonidVas pushed a commit that referenced this issue Jul 15, 2022
@better0fdead @LeonidVas
workflow: separate integration tests from other checks
6edd090 
Separated integration tests from linter, unit, e2e and examples checks,
because integration tests run longer then all other.
Result: 15% faster testing.

Part of #690
@LeonidVas LeonidVas added the 5sp label Jul 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@better0fdead better0fdead

Labels
code health Improve code readability, simplify maintenance and so on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tests: make integration testing faster tarantool/cartridge-cli
3 participants
@kyukhin @LeonidVas @better0fdead