ISPN000314: Lost at least half of the stable members, possible split brain causing data inconsistency : How to segregate the clusters?

[prettypot]

Hi,

We have two clusters on the same VLAN. One is Test, and the other is UAT. Each cluster has a different database backe-end . We cannot move the servers to a different network.

The Test servers use the default settings out of the box.

The UAT servers use a different infinicache setting:

<jgroups>
    <stack name="qa1" extends="udp">
    <SSL_KEY_EXCHANGE keystore_name="/opt/keycloak/pki/truststore.jks"
        keystore_password="password"
        stack.combine="INSERT_AFTER"
        stack.position="VERIFY_SUSPECT2"/>
        <ASYM_ENCRYPT asym_keylength="2048"
        asym_algorithm="RSA"
        change_key_on_coord_leave = "false"
        change_key_on_leave = "false"
        use_external_key_exchange = "true"
        stack.combine="INSERT_BEFORE"
        stack.position="pbcast.NAKACK2"/>
    </stack>
</jgroups>

    <cache-container name="keycloak" statistics="true">
        <transport lock-timeout="60000" stack="qa1"/>

However, I still get these errors in the logs, which breaks keycloak:-

2023-10-31 16:23:22,737 WARN  [org.infinispan.CLUSTER] (non-blocking-thread--p2-t4) [Context=actionTokens]
ISPN000314: Lost at least half of the stable members, possible split brain causing data inconsistency.
Current members are [node2-6495], lost members are [testnode1-3541], stable members are [testnode1-3541, node2-6495]

I tried getting the nodes to drop packets from each other, but this won't stop multicast addresses. e.g ip route add prohibit 10.1.2.88/32

How can I stop the two clusters from talking to each other?

[prettypot]

I tried with TCPPING in the cache config, but keycloak was sick.

Is there a guide to configuring TCPPING clusters? Because it looks like tcpping can specify a list of hosts and ports to connect with to form a cluster from.