fix: network fork on restart

[strophy]

Issue being fixed or feature implemented

Testnet platform chain halted due to a disagreement about how to replay when syncing from scratch or after restarting in some cases. This was investigated and found to be due to the use of two seed nodes that do not have knowledge of each other.

What was done?

Add seed node addresses to config for seed nodes.

How Has This Been Tested?

Deployed on schnapps, stopped all nodes, restarted seed nodes one at a time, then restarted masternodes, network came back without any issues.

Breaking Changes

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

[strophy]

I've unrolled and indented the code here so it is easier to read:

# Comma separated list of seed nodes to connect to
seeds = "
{% if inventory_hostname not in groups.seed_nodes %}
  {% for seed in groups.seed_nodes %}
    {{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}
    {% if not loop.last %},{% endif %}
  {% endfor %}
{%- elif inventory_hostname in groups.seed_nodes -%}
  {% for seed in groups.seed_nodes %}
    {% if seed != inventory_hostname %}
      {{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}
    {% endif %}
  {% endfor %}
{%- endif -%}"

[lklimek]

Can't we simplify it to:

Suggested change

seeds = "{% if inventory_hostname not in groups.seed_nodes %}{% for seed in groups.seed_nodes %}{{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}{% if not loop.last %},{% endif %}{% endfor %}{%- elif inventory_hostname in groups.seed_nodes -%}{% for seed in groups.seed_nodes %}{% if seed != inventory_hostname %}{{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}{% endif %}{% endfor %}{%- endif -%}"

seeds = "{% for seed in groups.seed_nodes %}{% if seed != inventory_hostname %}{{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}{% endif %}{% endfor %}"

[strophy]

This would result in a syntax error on masternodes which should list both seeds, the seed nodes must be separated by a comma. We could probably simplify it like this, but then the seed nodes would also contain their own address:

{% for seed in groups.seed_nodes %}
  {{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}
  {% if not loop.last %},{% endif %}
{% endfor %}

Is that acceptable?

[lklimek]

In theory, we have protection against "dialing myself", but we should not add the current node as this is NOT a correct solution. The correct solution is to put all seeds except ourselves, comma-separated.

And, if I understand correctly, your original solution will fail for 3 seeds - because there will be no comma between them?

[strophy]

Correct. We need to find a way to remove the "self" seed from the group of possible seeds, which might exceed 2. Or we can use my initial code suggestion, and limit the number of seed nodes to 2 per network.

[strophy]

OK, I've found a solution. There are no loop controls like break or continue in standard Jinja2 so we need two loops, and set temp = is necessary to avoid outputting the name of the node being removed from the list. So we end up with something like this:

# Comma separated list of seed nodes to connect to
seeds = "
{% for seed in groups.seed_nodes %}
  {% if seed == inventory_hostname %}
    {% set temp = groups.seed_nodes.pop(loop.index0) %}
  {% endif %}
{% endfor %}
{% for seed in groups.seed_nodes %}
  {{ seed_nodes[seed].node_key.id }}@{{ hostvars[seed].public_ip }}:{{ tendermint_p2p_port }}
  {% if not loop.last %},{% endif %}
{% endfor %}"

[shumkov]

👍

agree with Lukasz, the variable with list must be prepared in ansible

[shumkov]

LGTM! 👍