Remote exceptions

[aweisberg]

No description provided.

[dcapwell]

Suggested change

	node.reply(replyTo, replyContext, reply, null);
	node.reply(replyTo, replyContext, reply, failure);

you didn't check if failure is non-null, so pass that on to the sink to check

[aweisberg]

Yeah makes sense to have it assert.

[dcapwell]

style: remove {}

[dcapwell]

this doesn't look right, if we see a failure will we actually decrement completely? Also wouldn't the exception be lost if it isn't the last event?

[aweisberg]

Good catch. I will refactor it to check in accept and only send the failure response if it hasn't already sent a regular response and log if somehow we have failure but already sent a response.

[dcapwell]

we should report back still, else the future can hang forever right?

[aweisberg]

We can't report back because without the success there is no requestId to respond with. We can't use uncaught exception handler because the exception isn't typically uncaught tit's actually been passed by the caller of onRecover. onRecover is an odd one because it's kind of informative, but it isn't the callback for recover it's just some notification the agent gets.

[aweisberg]

We can't report back because without the success there is no requestId to respond with. We can't use uncaught exception handler because the exception isn't typically uncaught tit's actually been passed by to the callback of Recover. onRecover is an odd one because it's kind of informative, but it isn't the callback for recover it's just some notification the agent gets.

I need to look at why onRecover is involved in sending the reply at all. Why doesn't it get done in the callback for Recover? Why does it exist at all?

[belliottsmith]

This is only for the ListAgent, which is used only for testing purposes. We keep enough information to issue a response directly to clients for testing so we get more responses to analyse the correctness of.

[dcapwell]

Suggested change

	node.reply(client, replyContext, (ListResult) success, null);
	node.reply(client, replyContext, (ListResult) success, fail);

didn't check fail yet

[dcapwell]

Suggested change

	checkState(send == null, "fail (%s) and send (%s) are both not null", failure, send);
	if (send != null)
	agent().onUncaughtException(new IllegalArgumentException(String.format("fail (%s) and send (%s) are both not null", failure, send)));

this situation would currently cause us to loose the real exception

[dcapwell]

Suggested change

	logger.error("Had error in WaitOnCommit, but already replied so can't send failure response", failure);
	node.agent().onUncaughtException(new IllegalStateException("Had error in WaitOnCommit, but already replied so can't send failure response", failure));

[dcapwell]

you are not checking that throwable is non-null, so this could lead to an exception at fail.addSuppressed(throwable);

[aweisberg]

Ouch yes.

[dcapwell]

we don't actually know there is a read failure, we are a callback, so we see both success and failure! we have to first check which one we are

[dcapwell]

if there is a failure do we really want to call readComplete? Feel like we should try to halt right away as the coordinator has already seen our failure

[aweisberg]

I left it calling readComplete (exceptionally or otherwise) so that it would eventually send the response once all command stores responded with fail containing all the errors that occurred if there are more than one.

I am not married to that decision though. Sending back the first error is fine.

readComplete manipulates the waitingOn bit set so I left it at that level. readComplete does have one override that actually does something different, but since everything always goes through ack and ack checks for failures I was going to call it good enough.

It might be cleaner to put the waitingOn update into ack and call ack directly. WDYT?

[dcapwell]

ill look closer before replying directly to your comments, but one thing I am thinking about; when we lived with everyone needing to handle exceptions, no one did... this patch is trying to improve that.... so for this to hope all extending classes "do the right thing" might be good much to ask.... but ill reread this code and reflect on your comment

[aweisberg]

The derived classes aren't the ones that send back the errors. ack does and that is private.

The derived classes can (and do) drop errors as part of their execution such as FetchRequest.readComplete which starts a new AsyncChain and adds a callback which passes the failure to uncaughtExceptionHandler.

I think the interface for listeners, task submission, and callbacks errors is possibly what is wrong. Every execution should occur with a context that specifies what the uncaught error handling should be, but you don't have to provide that context along with the thing you are executing. Or you do, but it's bespoke to that one thing you want to run (parameter to callback) rather than common across a variety of things that might be done to execute a request.

That would help with making it so can't get an error and then drop it or consume it in the wrong place because you would never handle it in the first place unless you specifically provided your own error handler rather than the one in the context. And when you do specify it should still hand off errors you don't want to deal with to the parent error handling context.

[dcapwell]

before this patch this is what I see

• accord.messages.ReadData#read would reply Error on the first exception and not call readComplete. This would not update waitingOn, but more importantly it would not call ack; which means if (-1 == --waitingOnCount) will never be true!
• accord.impl.AbstractFetchCoordinator.FetchRequest#readComplete makes the read 2 operations:txn.read() and store.maxAppliedFor(...).onSuccess(a -> {this.maxApplied = a; super.readComplete(...)}) (I overly simplified). Even if a fetch had 2 reads (C* only has 1 atm), then we would still not see the ACK so would never complete the operation....
• accord.messages.ReadTxnData#readComplete documents that it should unregister its listener... but because we call cancel we will schedule removal of the listener anyways... so thats prob fine.

I am not married to that decision though. Sending back the first error is fine.

I guess I would ask, what will the coordinator do differently with a list of failures? If a single one exists we will do the same as if 100 existed... so by only sending the first we have a smaller msg payload and can start handling the exception faster! We could detect this case and just log locally that these exceptions are getting ignored....

So, in short, I am in favor of failing fast and not pushing handling exceptions properly to each impl.

[dcapwell]

I think the interface for listeners, task submission, and callbacks errors is possibly what is wrong. Every execution should occur with a context that specifies what the uncaught error handling should be,

I agree with you on this point... AsyncChain and Futures are best when we "chain" them and centralize the error handling, but when we keep mixing and matching it gets hard to get right...

the common pattern is

// this is actually async, but to simplify the code I didn't do that
A value = null;
for store in stores.intersects(request):
  A local = store.process(request.map)
  value = request.reduce(value, local);
request.accept(value)

and in this case we try to add more state to make sure all map are called and accept is called (which is why we use -1 in -1 == --waitingOnCount)...

We could change this pattern to be

List<AsyncChain<A>> chains;
for store in stores.intersects(request):
  chains.add(store.process(request.map));
AsyncChains.reduce(chains, request.reduce)

this would allow us to start composing and chaining, rather than stoping chains, and forking new ones in different places...

Now, given how that is a structural change, I would prefer to do that as its own standalone work; adds far too much risk to this patch...

[dcapwell]

it would be good to document why the code is written this way, and we can avoid the boolean success if we rely on return when a failure is thrown

// There are two different type of exceptions: user function throws, listener throws.  To make sure this is clear,
            // make sure to catch the exception from the user function and set as failed, and let the listener exceptions bubble up.
            V call;
            try
            {
                call = callable.call();
            }
            catch (Throwable t)
            {
                setResult(null, t);
                return;
            }
            setResult(call, null);

[dcapwell]

to avoid copy/paste maybe we should extends AsyncResults.RunnableResult?

public static class Task<T> extends AsyncResults.RunnableResult<T> implements Pending, RunnableFuture<T>
    {
        public Task(Callable<T> fn)
        {
            super(fn);
        }

we can avoid the run method in this case completely

[dcapwell]

I am +1 once ReadData is resolved

[dcapwell]

why this lock? all impl delegate to CommandStores so wouldn't need locking.

[aweisberg]

You are right it probably isn't necessary. I reflexively added it because the existing call to cancel is from a synchronized method, but the actual implementations of cancel don't require it to be synchronized and are probably also idempotent so it's fine if multiple responses and cancellations happen at once. I'll remove it.

[dcapwell]

looks like dead code now?

[dcapwell]

Suggested change

	node.reply(replyTo, replyContext, null, fail);
	state = State.RETURNED;
	node.reply(replyTo, replyContext, null, fail);

[aweisberg]

Shouldn't this also have the same illegal state logic as well? Maybe the fail handling should be in the switch below and handle obsolete as well.

[dcapwell]

Suggested change

	reply(replyingToNode, replyContext, new FailureReply(failure));
	lookup.apply(self).agent().onUncaughtException(failure);
	reply(replyingToNode, replyContext, new FailureReply(failure));