Skip to content
Permalink
Browse files

Introduce InvocationTargetLeaveAware for raft operations

InvocationTargetLeaveAware is used for determining if a raft operation
is safe to retry on leader leave and it is ok to commit the operation
multiple times. If the operation is safe, it is retried automatically if
the leader returns commit response of the invoked operation. If the
operation is not known to be safe, i.e. it either does not implement
InvocationTargetLeaveAware or returns false from
InvocationTargetLeaveAware#isSafeToRetryOnTargetLeave(), then
retry is decided based on the
RaftAlgorithmConfig#failOnIndeterminateOperationState config parameter.

Safe to retry raft operations are expected to be idempotent.
For instance, lock, session-aware semaphore, and count down latch
operations are idempotent. It does not make a difference to commit an
idempotent operation once or multiple times. However, although some
raft operations are idempotent in terms of state changes they cause,
they may not return the same response on duplicate commits. They can be
still considered to be safe to retry on leader node leave, depending on
their context and how they are used.
  • Loading branch information...
metanet authored and mdogan committed Nov 8, 2018
1 parent 4427c9c commit 550ddb4a85ab899fa41fddd5e7c265db1c658cf3
Showing with 589 additions and 87 deletions.
  1. +7 −1 ...-raft-dataservices/src/main/java/com/hazelcast/raft/service/atomiclong/operation/AddAndGetOp.java
  2. +7 −1 ...-raft-dataservices/src/main/java/com/hazelcast/raft/service/atomiclong/operation/GetAndAddOp.java
  3. +7 −1 ...st-raft-dataservices/src/main/java/com/hazelcast/raft/service/atomicref/operation/ContainsOp.java
  4. +6 −1 hazelcast-raft-dataservices/src/main/java/com/hazelcast/raft/service/atomicref/operation/GetOp.java
  5. +7 −1 ...-raft-dataservices/src/main/java/com/hazelcast/raft/service/countdownlatch/operation/AwaitOp.java
  6. +7 −1 ...t-dataservices/src/main/java/com/hazelcast/raft/service/countdownlatch/operation/CountDownOp.java
  7. +7 −1 ...rvices/src/main/java/com/hazelcast/raft/service/countdownlatch/operation/GetRemainingCountOp.java
  8. +7 −1 ...ft-dataservices/src/main/java/com/hazelcast/raft/service/countdownlatch/operation/GetRoundOp.java
  9. +7 −1 ...cast-raft-dataservices/src/main/java/com/hazelcast/raft/service/lock/operation/ForceUnlockOp.java
  10. +26 −1 ...dataservices/src/main/java/com/hazelcast/raft/service/lock/operation/GetLockOwnershipStateOp.java
  11. +8 −2 hazelcast-raft-dataservices/src/main/java/com/hazelcast/raft/service/lock/operation/LockOp.java
  12. +8 −2 hazelcast-raft-dataservices/src/main/java/com/hazelcast/raft/service/lock/operation/TryLockOp.java
  13. +7 −1 hazelcast-raft-dataservices/src/main/java/com/hazelcast/raft/service/lock/operation/UnlockOp.java
  14. +9 −1 ...t-dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/AcquirePermitsOp.java
  15. +7 −1 ...dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/AvailablePermitsOp.java
  16. +9 −1 ...ft-dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/ChangePermitsOp.java
  17. +9 −1 ...aft-dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/DrainPermitsOp.java
  18. +7 −1 ...ft-dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/InitSemaphoreOp.java
  19. +9 −1 ...t-dataservices/src/main/java/com/hazelcast/raft/service/semaphore/operation/ReleasePermitsOp.java
  20. +18 −6 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/service/RaftInvocationManager.java
  21. +7 −1 ...raft-server/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/AddRaftMemberOp.java
  22. +8 −2 ...er/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/CheckRemovedRaftMemberOp.java
  23. +9 −3 ...src/main/java/com/hazelcast/raft/impl/service/operation/metadata/CompleteDestroyRaftGroupsOp.java
  24. +7 −1 ...java/com/hazelcast/raft/impl/service/operation/metadata/CompleteRaftGroupMembershipChangesOp.java
  25. +7 −1 ...r/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/CreateMetadataRaftGroupOp.java
  26. +9 −3 ...ft-server/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/CreateRaftGroupOp.java
  27. +7 −1 ...ver/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/ForceDestroyRaftGroupOp.java
  28. +8 −2 ...rver/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/GetActiveRaftMembersOp.java
  29. +8 −2 ...src/main/java/com/hazelcast/raft/impl/service/operation/metadata/GetDestroyingRaftGroupIdsOp.java
  30. +8 −2 ...elcast/raft/impl/service/operation/metadata/GetInitialRaftGroupMembersIfCurrentGroupMemberOp.java
  31. +8 −2 ...rc/main/java/com/hazelcast/raft/impl/service/operation/metadata/GetMembershipChangeContextOp.java
  32. +7 −1 ...-raft-server/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/GetRaftGroupOp.java
  33. +9 −3 ...r/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/TriggerDestroyRaftGroupOp.java
  34. +7 −1 ...r/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/TriggerExpandRaftGroupsOp.java
  35. +7 −1 ...rc/main/java/com/hazelcast/raft/impl/service/operation/metadata/TriggerRebalanceRaftGroupsOp.java
  36. +8 −2 ...r/src/main/java/com/hazelcast/raft/impl/service/operation/metadata/TriggerRemoveRaftMemberOp.java
  37. +9 −4 ...-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/ChangeRaftGroupMembershipOp.java
  38. +11 −2 ...lcast-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/DefaultRaftReplicateOp.java
  39. +6 −1 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/DestroyRaftGroupOp.java
  40. +26 −0 ...t-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/InvocationTargetLeaveAware.java
  41. +9 −2 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/RaftQueryOp.java
  42. +3 −3 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/service/proxy/RaftReplicateOp.java
  43. +7 −1 ...-raft-server/src/main/java/com/hazelcast/raft/impl/session/operation/CloseInactiveSessionsOp.java
  44. +7 −1 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/session/operation/CloseSessionOp.java
  45. +7 −1 hazelcast-raft-server/src/main/java/com/hazelcast/raft/impl/session/operation/CreateSessionOp.java
  46. +7 −1 ...lcast-raft-server/src/main/java/com/hazelcast/raft/impl/session/operation/HeartbeatSessionOp.java
  47. +7 −1 ...ast-raft-server/src/main/java/com/hazelcast/raft/impl/session/operation/InvalidateSessionsOp.java
  48. +26 −12 hazelcast-raft-server/src/main/java/com/hazelcast/spi/impl/operationservice/impl/RaftInvocation.java
  49. +1 −3 hazelcast-raft-server/src/test/java/com/hazelcast/raft/impl/util/DummyOp.java
  50. +160 −0 ...-server/src/test/java/com/hazelcast/spi/impl/operationservice/impl/RaftInvocationFailureTest.java
@@ -19,6 +19,7 @@
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.atomiclong.RaftAtomicLongDataSerializerHook;
import com.hazelcast.raft.service.atomiclong.RaftAtomicLong;

@@ -27,7 +28,7 @@
/**
* TODO: Javadoc Pending...
*/
public class AddAndGetOp extends AbstractAtomicLongOp {
public class AddAndGetOp extends AbstractAtomicLongOp implements InvocationTargetLeaveAware {

private long delta;

@@ -45,6 +46,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return atomic.addAndGet(delta);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return delta == 0;
}

@Override
public void writeData(ObjectDataOutput out) throws IOException {
super.writeData(out);
@@ -19,6 +19,7 @@
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.atomiclong.RaftAtomicLongDataSerializerHook;
import com.hazelcast.raft.service.atomiclong.RaftAtomicLong;

@@ -27,7 +28,7 @@
/**
* TODO: Javadoc Pending...
*/
public class GetAndAddOp extends AbstractAtomicLongOp {
public class GetAndAddOp extends AbstractAtomicLongOp implements InvocationTargetLeaveAware {

private long delta;

@@ -45,6 +46,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return atomic.getAndAdd(delta);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return delta == 0;
}

@Override
public void writeData(ObjectDataOutput out) throws IOException {
super.writeData(out);
@@ -21,6 +21,7 @@
import com.hazelcast.nio.serialization.Data;
import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.atomicref.RaftAtomicReferenceDataSerializerHook;
import com.hazelcast.raft.service.atomicref.RaftAtomicRef;

@@ -29,7 +30,7 @@
/**
* TODO: Javadoc Pending...
*/
public class ContainsOp extends AbstractAtomicRefOp implements IdentifiedDataSerializable {
public class ContainsOp extends AbstractAtomicRefOp implements InvocationTargetLeaveAware, IdentifiedDataSerializable {

private Data value;

@@ -47,6 +48,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return ref.contains(value);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftAtomicReferenceDataSerializerHook.CONTAINS_OP;
@@ -18,12 +18,13 @@

import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.atomicref.RaftAtomicReferenceDataSerializerHook;

/**
* TODO: Javadoc Pending...
*/
public class GetOp extends AbstractAtomicRefOp implements IdentifiedDataSerializable {
public class GetOp extends AbstractAtomicRefOp implements InvocationTargetLeaveAware, IdentifiedDataSerializable {

public GetOp() {
}
@@ -37,6 +38,10 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return getAtomicRef(groupId).get();
}

public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftAtomicReferenceDataSerializerHook.GET_OP;
@@ -19,6 +19,7 @@
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.impl.util.PostponedResponse;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchDataSerializerHook;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchService;
@@ -28,7 +29,7 @@
/**
* TODO: Javadoc Pending...
*/
public class AwaitOp extends AbstractCountDownLatchOp {
public class AwaitOp extends AbstractCountDownLatchOp implements InvocationTargetLeaveAware {

private long timeoutMillis;

@@ -50,6 +51,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return timeoutMillis > 0 ? PostponedResponse.INSTANCE : false;
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftCountDownLatchDataSerializerHook.AWAIT_OP;
@@ -19,6 +19,7 @@
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchDataSerializerHook;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchService;

@@ -28,7 +29,7 @@
/**
* TODO: Javadoc Pending...
*/
public class CountDownOp extends AbstractCountDownLatchOp {
public class CountDownOp extends AbstractCountDownLatchOp implements InvocationTargetLeaveAware {

private int expectedRound;
private UUID invocationUid;
@@ -48,6 +49,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return service.countDown(groupId, name, expectedRound, invocationUid);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftCountDownLatchDataSerializerHook.COUNT_DOWN_OP;
@@ -17,13 +17,14 @@
package com.hazelcast.raft.service.countdownlatch.operation;

import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchDataSerializerHook;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchService;

/**
* TODO: Javadoc Pending...
*/
public class GetRemainingCountOp extends AbstractCountDownLatchOp {
public class GetRemainingCountOp extends AbstractCountDownLatchOp implements InvocationTargetLeaveAware {

public GetRemainingCountOp() {
}
@@ -38,6 +39,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return service.getRemainingCount(groupId, name);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftCountDownLatchDataSerializerHook.GET_REMAINING_COUNT_OP;
@@ -17,13 +17,14 @@
package com.hazelcast.raft.service.countdownlatch.operation;

import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchDataSerializerHook;
import com.hazelcast.raft.service.countdownlatch.RaftCountDownLatchService;

/**
* TODO: Javadoc Pending...
*/
public class GetRoundOp extends AbstractCountDownLatchOp {
public class GetRoundOp extends AbstractCountDownLatchOp implements InvocationTargetLeaveAware {

public GetRoundOp() {
}
@@ -38,6 +39,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return service.getRound(groupId, name);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftCountDownLatchDataSerializerHook.GET_ROUND_OP;
@@ -21,6 +21,7 @@
import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.RaftOp;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.lock.RaftLockDataSerializerHook;
import com.hazelcast.raft.service.lock.RaftLockService;

@@ -30,7 +31,7 @@
/**
* TODO: Javadoc Pending...
*/
public class ForceUnlockOp extends RaftOp implements IdentifiedDataSerializable {
public class ForceUnlockOp extends RaftOp implements InvocationTargetLeaveAware, IdentifiedDataSerializable {

private String name;
private long expectedFence;
@@ -52,6 +53,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return true;
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public final String getServiceName() {
return RaftLockService.SERVICE_NAME;
@@ -1,16 +1,36 @@
/*
* Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.hazelcast.raft.service.lock.operation;

import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.RaftOp;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.service.lock.RaftLockDataSerializerHook;
import com.hazelcast.raft.service.lock.RaftLockService;

import java.io.IOException;

public class GetLockOwnershipStateOp extends RaftOp implements IdentifiedDataSerializable {
/**
* TODO: Javadoc Pending...
*/
public class GetLockOwnershipStateOp extends RaftOp implements InvocationTargetLeaveAware, IdentifiedDataSerializable {

private String name;

@@ -27,6 +47,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return service.getLockOwnershipState(groupId, name);
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public final String getServiceName() {
return RaftLockService.SERVICE_NAME;
@@ -17,18 +17,19 @@
package com.hazelcast.raft.service.lock.operation;

import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.impl.util.PostponedResponse;
import com.hazelcast.raft.service.lock.LockEndpoint;
import com.hazelcast.raft.service.lock.RaftLockDataSerializerHook;
import com.hazelcast.raft.service.lock.RaftLockService;
import com.hazelcast.raft.service.lock.RaftLockOwnershipState;
import com.hazelcast.raft.service.lock.RaftLockService;

import java.util.UUID;

/**
* TODO: Javadoc Pending...
*/
public class LockOp extends AbstractLockOp {
public class LockOp extends AbstractLockOp implements InvocationTargetLeaveAware {

public LockOp() {
}
@@ -45,6 +46,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return ownership.isLocked() ? ownership : PostponedResponse.INSTANCE;
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public int getId() {
return RaftLockDataSerializerHook.LOCK_OP;
@@ -19,18 +19,19 @@
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.raft.RaftGroupId;
import com.hazelcast.raft.impl.service.proxy.InvocationTargetLeaveAware;
import com.hazelcast.raft.impl.util.PostponedResponse;
import com.hazelcast.raft.service.lock.RaftLockDataSerializerHook;
import com.hazelcast.raft.service.lock.RaftLockService;
import com.hazelcast.raft.service.lock.RaftLockOwnershipState;
import com.hazelcast.raft.service.lock.RaftLockService;

import java.io.IOException;
import java.util.UUID;

/**
* TODO: Javadoc Pending...
*/
public class TryLockOp extends AbstractLockOp {
public class TryLockOp extends AbstractLockOp implements InvocationTargetLeaveAware {

private long timeoutMs;

@@ -55,6 +56,11 @@ public Object run(RaftGroupId groupId, long commitIndex) {
return ownership;
}

@Override
public boolean isSafeToRetryOnTargetLeave() {
return true;
}

@Override
public void writeData(ObjectDataOutput out) throws IOException {
super.writeData(out);

0 comments on commit 550ddb4

Please sign in to comment.
You can’t perform that action at this time.