How to find out which node is an owner

[hypsrizzari]

Hello,

From an operations perspective, as I understand the order of startup and shutdown matters. Is there any way to query a node to find out if it is currently an owner? I'm asking because today I run this service on AWS ECS and I don't have much control over the order of operations. However, if I used just regular EC2 instances with CACHE_OWNERS set to 2, I would still have to track which one was started first and second to know if it was an owner. And if one was replaced then I might not know which one was an owner. If there was a command to query this it would help, but I haven't been able to find something like this in the docs.

Thanks!

[tristantarrant]

Ownership is per-entry.
Infinispan uses murmurhash3 as a consistent hashing algorithm. It divides the "hash space" into segments and maps each segment to a node. With the defaults (256 segments, 2 owners) and a 2-node cluster, each node will be primary owner of half of the segments and backup owner of the other half. In case of nodes joining and leaving, rebalancing happens and ownership changes.
Please read https://infinispan.org/docs/stable/titles/configuring/configuring.html#key-ownership_clustered-caches which provides an explanation about how Infinispan maps entries to nodes.
There are APIs that allow you to determine the primary owner of an entry.

[hypsrizzari]

Thank you for the insight. Anecdotally, we haven't paid much attention to the ordering of shut down, although it is mentioned here:

If you shutdown clustered nodes with the shutdown server command, you must restart each server in reverse order to avoid potential issues related to data loss and stale entries in the cache.

We have a problem with cache rebalancing under load, where every node is configured as an owner. I was attributing it to the order of nodes joining and leaving, but it seems it's really more about quick succession of nodes leaving and joining, and not having enough time to do the rebalance in between (and probably too many owners), especially under load.

Thanks again.