Partition loss during join #123

Closed
hzwaal opened this Issue Apr 12, 2012 · 6 comments

Comments

Projects
None yet
4 participants

hzwaal commented Apr 12, 2012

When a new node joins while the last node of a cluster is performing a shutdown, the join will succeed, but may not have received all partitions. If any of the missing partitions contained data, there will be data loss. The fact that this happened can only be detected in the logs, whereas it is very relevant to an application using Hazelcast to decide what should be done in this situation.

The partition loss could be handled as follows (and possible otherwise):

  1. Add a configuration parameter that can be used to tell the new node to drop all partitions and start clean, should the partition loss be detected.
  2. Add a configuration parameter that points to a policy class that instructs Hazelcast what to do (continue or restart empty).
  3. Add a method (on the PartitionService?) that can be used to inspect how many of the partitions were initialised properly. This method can be used after creating a Hazelcast instance to verify its sanity and take corrective measures if necessary.
Owner

mdogan commented Apr 13, 2012

Can you reproduce this loss easily? You should not loss any partition
during node joins or leaves. (There can be data loss related to backup
count or strategy.)

On Thu, Apr 12, 2012 at 3:58 PM, hzwaal <
reply@reply.github.com

wrote:

When a new node joins while the last node of a cluster is performing a
shutdown, the join will succeed, but may not have received all partitions.
If any of the missing partitions contained data, there will be data loss.
The fact that this happened can only be detected in the logs, whereas it is
very relevant to an application using Hazelcast to decide what should be
done in this situation.

The partition loss could be handled as follows (and possible otherwise):

  1. Add a configuration parameter that can be used to tell the new node to
    drop all partitions and start clean, should the partition loss be detected.
  2. Add a configuration parameter that points to a policy class that
    instructs Hazelcast what to do (continue or restart empty).
  3. Add a method (on the PartitionService?) that can be used to inspect how
    many of the partitions were initialised properly. This method can be used
    after creating a Hazelcast instance to verify its sanity and take
    corrective measures if necessary.

Reply to this email directly or view it on GitHub:
#123

hzwaal commented Apr 13, 2012

I haven't tried yet. I do have logs that I could send you. The issue occurs under the dubious condition that the cluster consists of only a single node when a second node joins. Before the join of the second node completes, the first node is (ungracefully) shutdown. After that the join of the second node succeeds, but with only part of the partitions/data.

I can understand that given the above scenario there is no way to save all partitions. My worry is that the newly joined node is never notified and has no way of finding out that partitions were lost.

I experienced this on 1.9.4.8, but I've also discussed this with Talip and feel that it could also occur in 2.X.

Owner

mdogan commented Apr 18, 2012

Although 2.x is more robust on backing up and migration features; if cluster has no backups at that moment, losing partition (also data) is inevitable during a node failure.

On 2.x, we don't have exposed as an API yet, but Hazelcast can know when a partition is lost (not knowing if actually any data is lost) and logs that as a warning.

We may inform user about this partition loss programmatically, like as a call to a listener.

hzwaal commented Apr 18, 2012

Because the partition loss occurs during the join/creating of a hazelcast instance, the listener is a bit problematic, unless it can be provided in the config.

Owner

mdogan commented May 28, 2014

Can be implemented along with new partition event api;

https://hazelcast.atlassian.net/wiki/display/PM/Partition+Event+API

mdogan added this to the 3.3 milestone May 28, 2014

@mdogan mdogan modified the milestone: 3.x Backlog, 3.3 Jul 7, 2014

@gurbuzali gurbuzali added a commit to gurbuzali/hazelcast that referenced this issue Jul 17, 2014

@gurbuzali gurbuzali fixes #123 asasda 451f642

@ajermakovics ajermakovics added Team: Core and removed Team: Core labels Oct 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment