Add ExecuteMultiOperation

[stephanos]

What changed?

Adding a new "ExecuteMultiOperation" API that will support sending a combined Start + Update.

The first version of the Server implementation can be found here: temporalio/temporal#5577

Note that while the API has a working implementation, it might still change. It is therefore annotated with "NOTE: Experimental API".

Why?

New API for sending operations to the Server atomically.

Breaking changes

[cretz]

I think multi-operation failure should be a gRPC error.

I can't imagine a continue-on-error future (or even a non-transactional future) where this has value reported as a successful gRPC result with a failure embedded. I think for now we should consider multi-operation transactional (i.e. all fails or all succeeds) as that's where it has most value, and use gRPC errors to report failure. If we need to be more detailed about the error for the user, we can add error details which can include partial successes if we ever have that (I hope not, it's confusing and has limited value). Even if we did have the concept of partial completion at some point in the future, I think it'd be up to request options on whether error should be ignored, but it's important IMO to let gRPC failures be gRPC failures.

[stephanos]

I've alluded to it in the other comment; the issue here is that once the Update is admitted, the Workflow needs to start or the wait stage accepted/completed would never be reached. And once the Workflow has started, it cannot be undone (as of now).

We could send a gRPC failure back in that case. That would mean that the client cannot see the partial success anymore.

[cretz]

Update failure/rejection outcome is already on the successful update response. A gRPC failure would be failure to admit an update which should also fail to start the workflow (same as failure to admit a signal on signal with start).

Also, I don't believe that multi-operation update should wait until update reaches a certain state, or that a worker even needs to run. I believe update "durably admitted" (which I heard we're adding support for so update-with-start works) is when it should return.

[stephanos]

Update failure/rejection outcome is already on the successful update response. A gRPC failure would be failure to admit an update which should also fail to start the workflow (same as failure to admit a signal on signal with start).

You're right, I've come to see it the same way 👍

Also, I don't believe that multi-operation update should wait until update reaches a certain state, or that a worker even needs to run. I believe update "durably admitted" (which I heard we're adding support for so update-with-start works) is when it should return.

We are considering "durably admitted" now (approval pending) indeed.

It's my understanding that being able to wait for accepted/completed allows the client to receive the response in a single gRPC call instead of having to initiate a poll. Once we allow the user to pick wait stage "Admitted", they can choose whether to to unblock early and poll or receive everything in one go.

Or are there other concerns for you?

[cretz]

It's my understanding that being able to wait for accepted/completed allows the client to receive the response in a single gRPC call instead of having to initiate a poll.

This is my understanding too, I just think a supported wait-stage should be admitted with this call. One of the great aspects of Temporal is that when I submit something durably I know it will run, I don't have to wait for it to run. We may not be offering this for update, but since we're offering it for update-with-start, we should allow users to return without a worker running (i.e. basically immediately once made durable), same as signal-with-start. But I also support the ability to wait longer for update response for sure.

[cretz]

Can we specify the transactional expectations of this call? Basically guarantee it all fails or all succeeds atomically/transactionally?

[stephanos]

Okay, I took a stab at it. I'm not quite sure yet how to express the subtleties in error behavior: when the Update fails to be admitted, there actually won't be a new Workflow (akin to a transaction). However, if the Update doesn't reach Accepted/Completed, only the Update operation will fail but not the Start. So it actually has limited transactional guarantees.

[cretz]

However, if the Update doesn't reach Accepted/Completed, only the Update operation will fail but not the Start. So it actually has limited transactional guarantees.

This depends on how you define fail. It won't fail from a gRPC perspective. I don't consider an update being rejected as an update "failing" from the POV of this API. If an update doesn't reach accepted/completed, the update response is capable of representing that. Still, that the start and update were accepted by the server and the update reached a state of durability, then the gRPC call succeeded, even if a single update had an application-level failure (not to be confused with server/gRPC failure).

Also, I don't believe that multi-operation update should wait until update reaches a certain state, or that a worker even needs to run. I believe update "durably admitted" (which I heard we're adding support for so update-with-start works) is when it should return.

[cretz]

Looks good. May want to clarify idempotency behavior (i.e. that users should set start workflow request ID since they may repeatedly invoke this w/ fail-if-existing), but don't need to necessarily.

(also, I disagree that multi operation should ever need to be speculative/lossy as that is very confusing for users, but this was already decided internally)

[stephanos]

May want to clarify idempotency behavior (i.e. that users should set start workflow request ID since they may repeatedly invoke this w/ fail-if-existing)

@cretz I'll think about this more ... I wonder if we should require a request ID 🤔 If we're not, I'll add a follow-up comment in a separate PR.

[cretz]

I wonder if we should require a request ID

We should not require it IMO. It is a choice from the API caller whether they will ever re-invoke this, and request ID is only for that. Think about the user invoking this from the HTTP API. We really need to care about that developer's experience too (which is why lossy updates are rough on them).