Network Partition Fault: Not working as expected

[prb-aerospike]

Describe the bug
I am trying to inject network partition fault in a 4 Node cluster with EC2 instances where 2 nodes are specified as an endpoint group and other 2 nodes are defined as hosts which need to be partitioned from given endpoint group. I see that only 1 out of 2 hosts is partitioned from the network and remaining nodes still remain and form a cluster of 3 nodes that includes one of the hosts.

To Reproduce
Steps to reproduce the behavior:

1. Create a Cluster of 4 nodes with Aerospike database Server (https://github.com/aerospike/aerospike-server.git) in AWS EC2 instances.
2. Add 2 of the nodes from cluster as an remote endpoint in Mangle and create and endpoint group from these 2 nodes
3. Inject Network partition fault from Mangle UI with other 2 nodes as the hosts.
4. Fault is successfully injected in Mangle. However, I observed that only 1 of the given 2 hosts was removed from the cluster of 4 nodes.

Please let me know if the more details are required on any step to reproduce the issue.

Expected behavior
Both of the given hosts should be the removed from the cluster and only 2 nodes (which were defined as endpoints) should be the part of the remaining cluster.

Screenshots
N/A

Logs
If applicable, add application logs/troubleshooting bundle to help in root cause analysis.

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single Node
• Client OS: Ubuntu 20.04
• Client Browser : Firefox
• Version 94.0

Additional context
Add any other context about the problem here.

[ashrimalivmware]

Hi @prb-aerospike

Can you please confirm if you have selected "Select Random Endpoint" field as True or False. If it was true can you please repeat the experiment with Random Injection as False?

[prb-aerospike]

Hi @ashrimalivmware,

Thanks for the quick reply!

I have tested this fault with "Select Random Endpoint" both as True and False but found the same result in both cases.
To add more info, only the first host that is added to the hosts list is being removed from the cluster.
For e.g, from the attached screenshot, only the host 172.34.9.0 gets out of cluster while 172.34.1.252 remains part of it.

Thanks,
Prakhar

[ashrimalivmware]

Thanks @prb-aerospike ,
Can you please provide the logs to debug the issue further?
Log file can be found at: /tmp/infra_agent/infra_agent.log
Please attach the logs from both of the hosts which was part of Endpoint Group.

Also can you please attach the screenshot of the injected task which says injection completed on "Requests and Reports" page?

Thanks,
-Avinash

[prb-aerospike]

Hi @ashrimalivmware,

Please find attached the requested logs from both hosts of endpoint group. In this experiment, hosts that need to be partitioned from endpoint group were 172.34.9.5 and 172.34.1.9 and only 172.34.9.5 was removed from the cluster.

infra_agent_13.57.194.156.log
infra_agent_18.144.68.151.log

Following is the screen shot of the injected task. Endpoint node1 corresponds to host 18.144.68.151 and node2 corresponds to 13.57.194.156:

Please let me know for any other information.

Thanks,
Prakhar

[ashrimalivmware]

Thanks @prb-aerospike for providing the details. Team is working on this issue and will try to provide a fix by next week. We shall provide you a patch as it will take time to get the fix in official release.

[prb-aerospike]

Thanks @ashrimalivmware for looking into the issue!

Is there any update on it yet?

Thanks,
Prakhar

[ashrimalivmware]

Hi @prb-aerospike ,
Team is working on this, will update you more about it sometime early next week.
Thanks,
-Avinash

[ashrimalivmware]

Hi @prb-aerospike , the issue has been fixed. Please let me know if we can connect over a zoom call to discuss about the fix as to release mangle with the fix would take some time.

[prb-aerospike]

Thank you @ashrimalivmware for the update!
Sure, we can connect over zoom.
My email Id is prakharbansal0910@gmail.com if you would like to send the invite.

[ashrimalivmware]

@prb-aerospike Can we close the ticket?

[prb-aerospike]

Yes @ashrimalivmware .
It's fixed for me now.
Thank you for quick turnaround on the solution for this issue!