Custom epmd to avoid race conditions in DNS name resolutions #4174

arcusfelis · 2023-11-21T01:19:12Z

This PR addresses:
Sometimes, it could take 30 seconds for DNS record to become resolvable in k8s.
In this time, a node could try establishing outgoing erlang distribution connections but could not accept incoming erlang distribution connections. But, global tries to maintain a fully connected mesh between nodes. This means that

new node connects to the cluster
other nodes try to connect to that new node (coordinated by global)
they get nxdomain error inside net_kernel and global gets pang
global gets pang (and nodedown), asks other nodes to disconnect from this new node:

 when=2023-11-02T14:06:42.939527+00:00 level=warning pid=<0.59.0> at=: unstructured_log="'global' at node 'mongooseim@mongooseim-5.mongooseim.default.svc.cluster.local' disconnected node 'mongooseim@mongooseim-0.mongooseim.default.svc.cluster.local' in order to prevent overlapping partitions"
when=2023-11-02T14:06:42.936864+00:00 level=warning pid=<0.1310.0> at=: unstructured_log="'global' at 'mongooseim@mongooseim-5.mongooseim.default.svc.cluster.local' failed to connect to 'mongooseim@mongooseim-4.mongooseim.default.svc.cluster.local'\

Proposed changes include:

Uses Rewrite ping cets#41 - improved logging on nodeup/nodedown, ping logic, that does not ask net_kernel to contact dead nodes.
Uses our own epmd module - it intercepts DNS lookups from net_kernel.
mongoose_node_address - keeps a list of IP addresses and nodenames - shares it between nodes.
RDBMS backend for CETS disco is patched to allow to set the address field.
The operator could set the IP addess using POD_IP.

…cluster_id

It collects addresses of nodes from RDBMS and from other nodes

Could lead to deadlocks or busy net_kernel

Add xref ingores

codecov · 2023-11-21T01:27:08Z

Codecov Report

Attention: 26 lines in your changes are missing coverage. Please review.

Comparison is base (07c2263) 84.03% compared to head (ec262fe) 83.98%.
Report is 13 commits behind head on master.

Files	Patch %	Lines
src/mongoose_epmd.erl	23.52%	13 Missing ⚠️
src/mongoose_node_address.erl	78.57%	12 Missing ⚠️
src/mongoose_cluster_id.erl	95.45%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4174      +/-   ##
==========================================
- Coverage   84.03%   83.98%   -0.05%     
==========================================
  Files         564      566       +2     
  Lines       34064    34167     +103     
==========================================
+ Hits        28625    28695      +70     
- Misses       5439     5472      +33

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Reorder columns to match table, insert, update and select queries

Add remember_addresses function Fix dialyzer

mongoose-im · 2023-11-21T10:25:50Z

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / b183f80
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / b183f80
Reports root / small

small_tests_26 / small_tests / b183f80
Reports root / small

small_tests_26_arm64 / small_tests / b183f80
Reports root / small

ldap_mnesia_25 / ldap_mnesia / b183f80
Reports root/ big
OK: 2311 / Failed: 0 / User-skipped: 867 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / b183f80
Reports root/ big
OK: 2311 / Failed: 0 / User-skipped: 867 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / b183f80
Reports root/ big
OK: 4278 / Failed: 0 / User-skipped: 112 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / b183f80
Reports root/ big
OK: 4246 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / b183f80
Reports root/ big
OK: 4278 / Failed: 0 / User-skipped: 112 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / b183f80
Reports root/ big
OK: 2461 / Failed: 0 / User-skipped: 717 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / b183f80
Reports root/ big
OK: 4275 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / b183f80
Reports root/ big
OK: 4327 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / b183f80
Reports root/ big
OK: 4667 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / b183f80
Reports root/ big
OK: 4646 / Failed: 1 / User-skipped: 139 / Auto-skipped: 0

carboncopy_SUITE:one2one:dropped_client_doesnt_create_duplicate_carbons

{error,
  {{badmatch,
     [{xmlel,<<"message">>,
        [{<<"from">>,
        <<"alice_dropped_client_doesnt_create_duplicate_carbons_592@localhost">>},
         {<<"to">>,
        <<"alice_dropped_client_doesnt_create_duplicate_carbons_592@localhost/res2">>},
         {<<"xmlns">>,<<"jabber:client">>},
         {<<"type">>,<<"chat">>}],
        [{xmlel,<<"sent">>,
           [{<<"xmlns">>,<<"urn:xmpp:carbons:2">>}],
           [{xmlel,<<"forwarded">>,
            [{<<"xmlns">>,<<"urn:xmpp:forward:0">>}],
            [{xmlel,<<"message">>,
               [{<<"from">>,
                 <<"alice_dropped_client_doesnt_create_duplicate_carbons_592@localhost/res1">>},
                {<<"type">>,<<"chat">>},
                {<<"to">>,
                 <<"bob_dropped_client_doesnt_create_duplicate_carbons_592@localhost/res1">>},
                {<<"xmlns">>,<<"jabber:client">>}],
               [{xmlel,<<"body">>,[],
                  [{xmlcdata,
                     <<"And pious action">>}]}]}]}]}]}]},
   [{carboncopy_SUITE,
      '-dropped_client_doesnt_create_duplicate_carbons/1-fun-0-',4,
      [{file,
         "/home/circleci/project/big_tests/tests/carboncopy_SUITE.erl"},
       {line,189}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_serv...

+                                  {ok, IP};
+                              false ->
+                                  maybe_retry(Node, Start, Timeout, Sleep,
+                                             {cannot_connect_to_epmd, Node, IP})

	{cannot_connect_to_epmd, Node, IP})
	{cannot_connect_to_epmd, Node, IP})

+                  Time = os:system_time(millisecond),
+                  case (Time - Start + Sleep) < Timeout of
+                      true ->
+                          lookup_loop(Node, Start, Timeout, Sleep);

+                  Pairs2 = lists:keydelete(node(), 1, Pairs),
+                  remember_addresses(Pairs2),
+                  State;
+              handle_get_pairs_result(_Node, _Res, State) ->

+              address_please(Name, Host, AddressFamily) ->
+                  Node = list_to_atom(Name ++ "@" ++ Host),
+                  case mongoose_node_address:lookup(Node) of

+                  %% undocumented (so, we cannot match in in handle_info)
+                  spawn_link(fun() ->
+                          %% We have to ignore our node name when processing result
+                          Server ! {get_pairs_result, Node, rpc:call(Node, ?MODULE, get_pairs, [])},

+                      {mongoose_node_address,
+                        {mongoose_node_address, start_link, []},
+                        permanent, infinity, supervisor, [mongoose_node_address]},
+                  [CetsDisco, Addr].

                   cluster_name varchar(250),
-                  updated_timestamp BIGINT NOT NULL, -- in seconds
+                  node_name varchar(250),

-                  ?assertMatch({error, _}, insert_new(CN, <<"test1">>, TS, 2)),
-                  ?assertMatch({error, _}, insert_new(CN, <<"test2">>, TS, 1)),
-                  ?assertEqual({updated, 1}, insert_new(CN, <<"test2">>, TS, 2)),
+                  ?assertEqual({updated, 1}, insert_new(CN, <<"test1">>, 1, <<>>, TS)),

Custom epmd to avoid race conditions in DNS name resolutions #4174

Custom epmd to avoid race conditions in DNS name resolutions #4174

Conversation

arcusfelis commented Nov 21, 2023 • edited

This comment was marked as outdated.

codecov bot commented Nov 21, 2023 • edited

Codecov Report

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Nov 21, 2023 • edited

mongoose-im commented Nov 21, 2023 • edited

mongoose-im commented Nov 21, 2023 • edited

mongoose-im commented Nov 21, 2023 • edited

chrzaszcz left a comment

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023 • edited

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

chrzaszcz Nov 24, 2023

Choose a reason for hiding this comment

mongoose-im commented Nov 24, 2023 • edited

arcusfelis commented Nov 28, 2023

arcusfelis commented Nov 21, 2023 •

edited

codecov bot commented Nov 21, 2023 •

edited

mongoose-im commented Nov 21, 2023 •

edited

mongoose-im commented Nov 21, 2023 •

edited

mongoose-im commented Nov 21, 2023 •

edited

mongoose-im commented Nov 21, 2023 •

edited

chrzaszcz Nov 24, 2023 •

edited

mongoose-im commented Nov 24, 2023 •

edited