[scheduled-executor] ScheduledTaskDescriptor.stopForMigration Nullpointer

[ruslan-belinskyy]

Guys,
i'm currently getting:

java.lang.NullPointerException
at com.hazelcast.scheduledexecutor.impl.ScheduledTaskDescriptor.stopForMigration(ScheduledTaskDescriptor.java:146) ~[hazelcast-all-3.8.3.jar:3.8.3]

When trying to schedule the Task:

IScheduledExecutorService service = hazelcastInstance.getScheduledExecutorService(executorName);
service.scheduleAtFixedRate(new HzcastTimerExchangeSender(this.getEndpoint().getEndpointUri(), executorName), endpoint.getDelay(), endpoint.getPeriod(), TimeUnit.MILLISECONDS);

I have looked to the fix you have here (#10604) and it doesn't look safe and could cause issue i have.

You had:

try {
 descriptor.cancel(true);
 descriptor.setScheduledFuture(null);
 descriptor.setTaskOwner(false); 
} catch (Exception ex) {
 throw rethrow(ex); 
}

And now:

try {
 descriptor.stopForMigration();
 } catch (Exception ex) {
 throw rethrow(ex); 
}

Where descriptor.stopForMigration() has:

void stopForMigration() {

   // Result is not set, allowing task to get re-scheduled, if/when needed.
   this.isTaskOwner = false;
   this.future.cancel(true); //Nullpointer here
   this.future = null;
}

And in old code :

descriptor.cancel(true);

Which has:

boolean cancel(boolean mayInterrupt)
throws ExecutionException, InterruptedException {
if (!resultRef.compareAndSet(null, new ScheduledTaskResult(true)) || future == null) {
return false;
}

    return future.cancel(mayInterrupt);
}

This look way safer.

Nullpointer i have on 2 machines which have code you can see above.

[tkountis]

Thanks for the detailed report @Batter2014 . Yes the fix should be very trivial, but I was wondering if you could share the logs as well, to see what action led to this result. Nevertheless, i will try to push a fix in a bit, but I would rather have a solid reproducer as a test.

[ruslan-belinskyy]

I don't have anything special, just two hosts with code:

IScheduledExecutorService service = hazelcastInstance.getScheduledExecutorService(executorName);
service.scheduleAtFixedRate(new HzcastTimerExchangeSender(this.getEndpoint().getEndpointUri(), executorName), endpoint.getDelay(), endpoint.getPeriod(), TimeUnit.MILLISECONDS);

When i start both of them, one after another i will always get that exception

Full stack:

java.lang.NullPointerException
	at com.hazelcast.scheduledexecutor.impl.ScheduledTaskDescriptor.stopForMigration(ScheduledTaskDescriptor.java:146) ~[hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.scheduledexecutor.impl.ScheduledExecutorContainer.prepareForReplication(ScheduledExecutorContainer.java:294) ~[hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.scheduledexecutor.impl.ScheduledExecutorPartition.prepareReplicationOperation(ScheduledExecutorPartition.java:70) ~[hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.scheduledexecutor.impl.DistributedScheduledExecutorService.prepareReplicationOperation(DistributedScheduledExecutorService.java:140) ~[hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.internal.partition.operation.MigrationRequestOperation.prepareMigrationOperations(MigrationRequestOperation.java:218) [hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.internal.partition.operation.MigrationRequestOperation.run(MigrationRequestOperation.java:86) [hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:186) [hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:120) [hazelcast-all-3.8.3.jar:3.8.3]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:100) [hazelcast-all-3.8.3.jar:3.8.3]

[tkountis]

@Batter2014 I posted a fix for 3.9 which i will also backport once merged to 3.8 (the version you are using). However, I wasn't able to reproduce it not even once, and since you said that you see this always, I wanted to make sure I am not missing sth. Your finding is absolutely valid, it is just the always part that got my attention. Would you be able to share your (or part of) the Runnable code ? Also logs of the nodes would also help, to see if there is sth wrong going on during migration of things between the nodes that cause this misbehaviour.