Cets/prevent crash on upgrade #4256

chrzaszcz · 2024-04-09T10:03:44Z

Add CETS major.minor version as a suffix to cluster name in the discovery_nodes table, preventing CETS from crashing if a rolling upgrade is performed, and the CETS versions are incompatible.

This would occur when upgrading MIM from 6.2.0 to the upcoming 6.2.1, which translates to CETS upgrade from 0.1.0 to 0.2.0. If only the patch version of CETS is changed, e.g. 0.2.0 to 0.2.1, they are still considered compatible. From now on, we only need to ensure that CETS versions are marked accordingly before MIM releases.

This change was tested manually by upgrading from 6.2.0 to this PR. The test was performed twice:

In local environment: 3-node cluster created with test-helper,
With helm upgrade.

Note: The version is not a separate DB column, because the nodes running version 6.2.0 would still try to connect to the upgraded nodes. Extending the name seems like a sane option, and similar solutions are used in well-known tools and platforms like k8s.

When performing a rolling upgrade e.g. from 6.2.0 to the upcoming 6.2.1, CETS would crash on all nodes due to version incompatibility. This can be prevented by putting cets 'Major.Minor' version in the cluster name, making the two cluster components disjoint during upgrade. The only thing to keep in mind is to bump the Major or Minor version number when there is any incompatibility in the CETS inter-node API.

mongoose-im · 2024-04-09T10:10:44Z

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / e4ef412
Reports root/ big
OK: 437 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / e4ef412
Reports root / small

small_tests_26 / small_tests / e4ef412
Reports root / small

small_tests_26_arm64 / small_tests / e4ef412
Reports root / small

ldap_mnesia_25 / ldap_mnesia / e4ef412
Reports root/ big
OK: 2283 / Failed: 1 / User-skipped: 897 / Auto-skipped: 0

pubsub_SUITE:dag+basic:request_all_items_test

{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / e4ef412
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / e4ef412
Reports root/ big
OK: 4508 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / e4ef412
Reports root/ big
OK: 4428 / Failed: 0 / User-skipped: 176 / Auto-skipped: 30

graphql_cets_SUITE:init_per_suite

{'EXIT',{{timeout,true,[{times,50,false}],ok},
     [{mongoose_helper,do_wait_until,2,
               [{file,"/home/circleci/project/big_tests/../test/common/mongoose_helper.erl"},
              {line,359}]},
      {graphql_cets_SUITE,init_per_suite,1,
                [{file,"/home/circleci/project/big_tests/tests/graphql_cets_SUITE.erl"},
                 {line,51}]},
      {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
      {test_server,run_test_case_eval1,6,
             [{file,"test_server.erl"},{line,1390}]},
      {test_server,run_test_case_eval,9,
             [{file,"test_server.erl"},{line,1234}]}]}}

Report log

internal_mnesia_26 / internal_mnesia / e4ef412
Reports root/ big
OK: 2424 / Failed: 0 / User-skipped: 757 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412
Reports root/ big
OK: 4538 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / e4ef412
Reports root/ big
OK: 4909 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412
Reports root/ big
OK: 4927 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

codecov · 2024-04-09T10:11:11Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.41%. Comparing base (cb92b6a) to head (082f1e0).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4256      +/-   ##
==========================================
- Coverage   84.43%   84.41%   -0.03%     
==========================================
  Files         552      552              
  Lines       33547    33551       +4     
==========================================
- Hits        28327    28322       -5     
- Misses       5220     5229       +9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mongoose-im · 2024-04-09T10:46:06Z

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 082f1e0
Reports root/ big
OK: 437 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / 082f1e0
Reports root / small

small_tests_26 / small_tests / 082f1e0
Reports root / small

small_tests_26_arm64 / small_tests / 082f1e0
Reports root / small

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4540 / Failed: 1 / User-skipped: 111 / Auto-skipped: 0

graphql_SUITE:tls_enabled:tls_connect_admin_unknown_certificate

{error,{{assertMatch,[{module,graphql_SUITE},
            {line,253},
            {expression,"Result"},
            {pattern,"{ error , { tls_alert , { unknown_ca , _ } } }"},
            {value,{error,connection_closed}}]},
    [{graphql_SUITE,tls_connect_admin_unknown_certificate,1,
            [{file,"/home/circleci/project/big_tests/tests/graphql_SUITE.erl"},
             {line,253}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

ldap_mnesia_25 / ldap_mnesia / 082f1e0
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / 082f1e0
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 082f1e0
Reports root/ big
OK: 4508 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 082f1e0
Reports root/ big
OK: 2424 / Failed: 0 / User-skipped: 757 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4538 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 082f1e0
Reports root/ big
OK: 4456 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 082f1e0
Reports root/ big
OK: 4909 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4922 / Failed: 5 / User-skipped: 121 / Auto-skipped: 0

mod_blocking_SUITE:manage:remove_user_from_blocklist

{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-remove_user_from_blocklist/1-fun-0-',2,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,196}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mod_blocking_SUITE:manage:add_another_user_to_blocklist

{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-add_another_user_to_blocklist/1-fun-0-',2,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,176}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mod_blocking_SUITE:manage:add_many_users_to_blocklist

{error,{{badmatch,0},
    [{escalus_story,drop_presences,2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,191}]},
     {escalus_story,'-start_ready_clients/2-fun-0-',3,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,135}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
     {escalus_story,start_ready_clients,2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,128}]},
     {lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
     {lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,69}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]}]}}

Report log

mod_blocking_SUITE:manage:remove_many_user_from_blocklist

{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {{escalus_session,session},
          {client,
            <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635@localhost/res1">>,
            escalus_tcp,<0.64064.0>,undefined,
            [{event_client,
               [{event_manager,<0.64036.0>},
              {server,<<"localhost">>},
              {username,
                <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
              {resource,<<"res1">>}]},
             {resource,<<"res1">>},
             {username,
               <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
             {server,<<"localhost">>},
             {host,<<"localhost">>},
             {port,5222},
             {auth,{escalus_auth,auth_plain}},
             {wspath,undefined},
             {username,
               <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
             {server,<<"localhost">>},
             {password,<<"makrowe;p">>},
             {stream_id,<<"65f6807913ba58f2">>}]},
          [{compression,false},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,session_reply}}}},
   [{escalus_story,'-start_ready_clients/2-fun-0-',3,
      [{file,
         "/home/circleci/project/big_tests/_...

Report log

mod_blocking_SUITE:manage:clear_blocklist

{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-clear_blocklist/1-fun-0-',3,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,217}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4927 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

JanuszJakubiec

Looks good

chrzaszcz added 2 commits April 9, 2024 09:29

Check that CETS version is added to cluster name

e4ef412

chrzaszcz changed the base branch from master to migration-6.2.1 April 9, 2024 10:04

Base automatically changed from migration-6.2.1 to master April 9, 2024 10:33

Update tests with the new cluster name

082f1e0

chrzaszcz marked this pull request as ready for review April 9, 2024 12:34

JanuszJakubiec approved these changes Apr 9, 2024

View reviewed changes

JanuszJakubiec merged commit e798ab4 into master Apr 9, 2024
4 checks passed

JanuszJakubiec deleted the cets/prevent-crash-on-upgrade branch April 9, 2024 14:27

chrzaszcz added this to the 6.2.1 milestone Apr 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cets/prevent crash on upgrade #4256

Cets/prevent crash on upgrade #4256

chrzaszcz commented Apr 9, 2024 •

edited

mongoose-im commented Apr 9, 2024 •

edited

codecov bot commented Apr 9, 2024 •

edited

mongoose-im commented Apr 9, 2024 •

edited

JanuszJakubiec left a comment

Cets/prevent crash on upgrade #4256

Cets/prevent crash on upgrade #4256

Conversation

chrzaszcz commented Apr 9, 2024 • edited

mongoose-im commented Apr 9, 2024 • edited

codecov bot commented Apr 9, 2024 • edited

Codecov Report

mongoose-im commented Apr 9, 2024 • edited

JanuszJakubiec left a comment

Choose a reason for hiding this comment

chrzaszcz commented Apr 9, 2024 •

edited

mongoose-im commented Apr 9, 2024 •

edited

codecov bot commented Apr 9, 2024 •

edited

mongoose-im commented Apr 9, 2024 •

edited