Fixing global distrib suite #4064

DenysGonchar · 2023-07-19T15:23:34Z

Timeout for trees_for_connections_present action is increased from 5 to 10 seconds:
• in most cases waiting takes 4,5-5,5 seconds on GH Actions runner
• and almost always above 5 seconds for mysql_redis preset (as redis is used for session data as well)
so this test case failed a lot and practically blocked passing of GH Actions builds

Also, wait_until helper code is improved a bit.

Successful GH Actions build: https://github.com/esl/MongooseIM/actions/runs/5605253063

mongoose-im · 2023-07-19T15:31:22Z

small_tests_24 / small_tests / 5cc9562
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 5cc9562
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / 5cc9562
Reports root / small

small_tests_25_arm64 / small_tests / 5cc9562
Reports root / small

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 5cc9562
Reports root/ big
OK: 4215 / Failed: 1 / User-skipped: 82 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.103996582>,
           [true,#{batch_flushes => 0,timed_flushes => 205}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

ldap_mnesia_24 / ldap_mnesia / 5cc9562
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 5cc9562
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 5cc9562
Reports root/ big
OK: 4215 / Failed: 1 / User-skipped: 82 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 0,timed_flushes => 225}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mam_SUITE:rdbms_async_pool_muc_all:muc_configurable_archiveid:muc_only_stanzaid

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,360}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,965}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1626}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1334}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1223}]}]}}}}

Report log

dynamic_domains_mysql_redis_25 / mysql_redis / 5cc9562
Reports root/ big
OK: 4189 / Failed: 1 / User-skipped: 108 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 1,timed_flushes => 146}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

pgsql_mnesia_24 / pgsql_mnesia / 5cc9562
Reports root/ big
OK: 4598 / Failed: 1 / User-skipped: 89 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.103996582>,
           [true,#{batch_flushes => 2,timed_flushes => 118}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 5cc9562
Reports root/ big
OK: 4212 / Failed: 1 / User-skipped: 85 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 2,timed_flushes => 143}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

internal_mnesia_25 / internal_mnesia / 5cc9562
Reports root/ big
OK: 2400 / Failed: 0 / User-skipped: 680 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 5cc9562
Reports root/ big
OK: 4584 / Failed: 1 / User-skipped: 103 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 2,timed_flushes => 136}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

pgsql_mnesia_25 / pgsql_mnesia / 5cc9562
Reports root/ big
OK: 4598 / Failed: 1 / User-skipped: 89 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 2,timed_flushes => 131}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mssql_mnesia_25 / odbc_mssql_mnesia / 5cc9562
Reports root/ big
OK: 4595 / Failed: 1 / User-skipped: 92 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_mam_all:archived:metrics_incremented_for_async_pools

{error,{{badarity,{#Fun<mam_SUITE.83.1615719>,
           [true,#{batch_flushes => 2,timed_flushes => 97}]}},
    [{mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {mongoose_helper,do_wait_until,2,
              [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
               {line,364}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

codecov · 2023-07-19T15:34:20Z

Codecov Report

Patch coverage has no change and project coverage change: +0.02 🎉

Comparison is base (1805ab6) 83.87% compared to head (ee60154) 83.89%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4064      +/-   ##
==========================================
+ Coverage   83.87%   83.89%   +0.02%     
==========================================
  Files         526      526              
  Lines       33159    33159              
==========================================
+ Hits        27812    27819       +7     
+ Misses       5347     5340       -7

see 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

mongoose-im · 2023-07-20T00:06:37Z

small_tests_24 / small_tests / b3bdacb
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / b3bdacb
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / b3bdacb
Reports root / small

small_tests_25_arm64 / small_tests / b3bdacb
Reports root / small

ldap_mnesia_24 / ldap_mnesia / b3bdacb
Reports root/ big
OK: 2253 / Failed: 1 / User-skipped: 826 / Auto-skipped: 0

pubsub_SUITE:tree+basic:publish_with_max_items_test

{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / b3bdacb
Reports root/ big
OK: 4216 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / b3bdacb
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / b3bdacb
Reports root/ big
OK: 4216 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / b3bdacb
Reports root/ big
OK: 4190 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_muc_all:muc06:muc_message_with_stanzaid

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,357}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,954}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1626}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1334}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1223}]}]}}}}

Report log

pgsql_mnesia_24 / pgsql_mnesia / b3bdacb
Reports root/ big
OK: 4599 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / b3bdacb
Reports root/ big
OK: 4213 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / b3bdacb
Reports root/ big
OK: 2400 / Failed: 0 / User-skipped: 680 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / b3bdacb
Reports root/ big
OK: 4599 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / b3bdacb
Reports root/ big
OK: 4585 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / b3bdacb
Reports root/ big
OK: 4596 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

ldap_mnesia_24 / ldap_mnesia / b3bdacb
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

…st_host_refreshing/1

mongoose-im · 2023-07-20T11:54:31Z

small_tests_24 / small_tests / ee60154
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / ee60154
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / ee60154
Reports root / small

small_tests_25_arm64 / small_tests / ee60154
Reports root / small

ldap_mnesia_24 / ldap_mnesia / ee60154
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / ee60154
Reports root/ big
OK: 4216 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / ee60154
Reports root/ big
OK: 2254 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / ee60154
Reports root/ big
OK: 4190 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / ee60154
Reports root/ big
OK: 4216 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / ee60154
Reports root/ big
OK: 4213 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / ee60154
Reports root/ big
OK: 4599 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / ee60154
Reports root/ big
OK: 2400 / Failed: 0 / User-skipped: 680 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / ee60154
Reports root/ big
OK: 4585 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / ee60154
Reports root/ big
OK: 4599 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / ee60154
Reports root/ big
OK: 4596 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

JanuszJakubiec · 2023-07-20T11:56:20Z

big_tests/tests/mongoose_helper.erl

-                      name := Name} = Opts) when TimeLeft =< 0 ->
+    Opts1 = maps:merge(Defaults, Opts0),
+    TimeLeft = maps:get(time_left, Opts1),
+    Opts = Opts1#{waiting_time => 0, max_waiting_time => TimeLeft,


why the max_waiting_time is added to opts? I cannot see anywhere in the code the usage of it.

For the traceability reasons only.

it's nice to have an unchanging initial value among the dynamically changing counter

Ok, now I understand. Thanks for explaining it to me.

JanuszJakubiec

Looks good

improving wait_until helper function in big and small tests

8e64270

DenysGonchar force-pushed the fixing-global-distrib-suite branch from 5cc9562 to b3bdacb Compare July 19, 2023 23:59

DenysGonchar marked this pull request as ready for review July 20, 2023 08:40

increasing timeout for wait_until call in mod_global_distrib_SUITE:te…

ee60154

…st_host_refreshing/1

DenysGonchar force-pushed the fixing-global-distrib-suite branch from b3bdacb to ee60154 Compare July 20, 2023 11:46

JanuszJakubiec reviewed Jul 20, 2023

View reviewed changes

JanuszJakubiec approved these changes Jul 20, 2023

View reviewed changes

JanuszJakubiec merged commit 41cdb68 into master Jul 20, 2023
4 checks passed

JanuszJakubiec deleted the fixing-global-distrib-suite branch July 20, 2023 12:15

chrzaszcz added this to the 6.2.0 milestone Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing global distrib suite #4064

Fixing global distrib suite #4064

DenysGonchar commented Jul 19, 2023 •

edited

mongoose-im commented Jul 19, 2023 •

edited

codecov bot commented Jul 19, 2023 •

edited

mongoose-im commented Jul 20, 2023 •

edited

mongoose-im commented Jul 20, 2023 •

edited

JanuszJakubiec Jul 20, 2023

DenysGonchar Jul 20, 2023

DenysGonchar Jul 20, 2023

JanuszJakubiec Jul 20, 2023

JanuszJakubiec left a comment

Fixing global distrib suite #4064

Fixing global distrib suite #4064

Conversation

DenysGonchar commented Jul 19, 2023 • edited

mongoose-im commented Jul 19, 2023 • edited

codecov bot commented Jul 19, 2023 • edited

Codecov Report

mongoose-im commented Jul 20, 2023 • edited

mongoose-im commented Jul 20, 2023 • edited

JanuszJakubiec Jul 20, 2023

Choose a reason for hiding this comment

DenysGonchar Jul 20, 2023

Choose a reason for hiding this comment

DenysGonchar Jul 20, 2023

Choose a reason for hiding this comment

JanuszJakubiec Jul 20, 2023

Choose a reason for hiding this comment

JanuszJakubiec left a comment

Choose a reason for hiding this comment

DenysGonchar commented Jul 19, 2023 •

edited

mongoose-im commented Jul 19, 2023 •

edited

codecov bot commented Jul 19, 2023 •

edited

mongoose-im commented Jul 20, 2023 •

edited

mongoose-im commented Jul 20, 2023 •

edited