Receiving notifications about transaction lifecycle

[jjmason]

Hi,

I'm looking for a way to send information about changes to grain state to an external service, but only after the transaction in which those changes were made is committed. To do this I'd like to be able to receive a notification (in each silo where the information was reported) after a transaction is confirmed or aborted.

It looks like there is some work around separating transaction management and participation, specifically in #4810, #4820 and #4860. From what I can understand, the idea is to add an ITransactionalResource or ITransactionalResourceExtension as a participant in transactions I'm interested in, but it's not clear what the "right" way to do this is.

I've run into a few issues going down this route - in particular when a reference to ITransactionalResourceExtension is a transaction participant, it is sometimes assumed to also be an ITransactionManagerExtension (for example when it is the first participant added to the transaction).

Is separation of transaction management and participation intended to support this kind of use case? If so, is that support ready in version 2.1.0, and could you point me to some resources as to how to use it?

[jason-bragg]

The transaction orchestration (interactions between transaction agents, managers, and resources) is extensible but the surface is still in flux. The manager logic and resource logic are setup by the transactional state, with the extensions being primarily bridging logic for distributed calls.

A transaction participant can act as a transaction manager, resource, or both. This is dictated by the SupportedRoles property in the participantId. For instance, if you wanted a transactional state that only performed as a resource, and not as a manager, the custom transactional state would need only set the supported roles to Role.Resource to indicate this.

For examples of how to provide your own transactional logic with custom manager or resource logic, please check out the TOC implementation. We added a TransactionCommitter which uses a service to perform the final commit operation rather than storage. To do this we added a new facet (ITransactionCommitter) and an implementation (TransactionCommitter) which is marked as a primary transaction manager (meaning there can be only one of these in a transaction and it will be selected as manager even if other participants support that behavior). Since the TransactionCommitter is not a resource no resource logic was configured, and the manager logic was customized for the desired behavior.

[jjmason]

@jason-bragg thanks! I'll take a look at the TOC implementation and let you know if I have further questions.

[jjmason]

@jason-bragg did you mean that the TOC implementation can provide this solution or that we should use it as a reference to build our own? I hooked up an ITransactionCommitter and it (mostly, see below) works, but ideally we'd be able to be notified of aborts as well. A workaround might be to use a timer to clear data associated with old transactions, but it feels a little dirty.

I did run into an interesting problem when the grain with the ITransactionCommitter facet was a StatelessWorker. Transactions started their prepare phase, but never completed it, leading to eventual prepare timeouts and aborted transactions. It does seem a bit odd for StatelessWorkers to participate in transactions, but in our case we'd like to avoid sending heavy records between silos and just gather them locally to be sent when the transaction is committed. Is this intended behavior? If not, let me know if you'd like me to open a separate issue.

Thanks!

[jason-bragg]

I mentioned the TOC as a reference implementation. I do not think it will address your needs as is, but can inform the developement of a custom solution.

TransactionalState and TransactionCommitter are both stateful. They are not intended to work within stateless workers. :/

[jjmason]

I've manged to get something (sort of) working along the lines of the TOC implementation.

I had to use a local build of Orleans with ContextResourceFactoryExtensions and ResourceFactoryRegistry modified to be public instead of internal. Are these planned to be made public at some later date, or am I completely off track looking at them?

Another, perhaps more straightforward, question: although I'm only adding resources to the transaction, I'm concerned that I might need to add a manager as well in case the the grain having the observation facet I added is nominated as the transaction manager (either because the transaction has no other participants or because the first participant is selected as the manager if no resources are have the priority manager role). Is this the case?

[jjmason]

@jason-bragg any thoughts on the above?

[jason-bragg]

Unfortunatly some of the transaction orchestration has changed. ContextResourceFactoryExtensions is gone in 2.2 beta. I suggest updating to that. You 'shouldn't' need access to any internals for what you're doing.

If you don't want the facet you're adding to be selected as a manager, set it's supported role only to Resource in it's ParticipantId.

[jsteinich]

@jason-bragg I work with @jjmason. We finally got a chance to revisit this on our side and have a few more findings and questions.

ContextResourceFactoryExtensions certainly appears to be a necessary part of implementing custom transactional resources. Both TransactionalState and TransactionCommitter register with that and both TransactionalResourceExtension and TransactionManagerExtension need to pull from it. While we may be able to use DI to replace the implementation class for ITransactionalResourceExtension, it doesn't seem like the correct way of doing things.
Another hidden piece is TransactionQueue which does a lot of the heavy lifting but not available for extension.

In general it would be great to not need to know about (or have access to) to the low level transaction pieces to simply register a hook into the transaction lifecycle. ITransactionCommitter comes close to that, but it is both missing other hooks and rather heavy handed.

Another question that isn't quite clear: Is it even possible to have a StatelessWorker be an ITransactionalResource? I know it doesn't work with the current implementations, but can the GrainReference that makes up most of the ParticipantId actually be used to get back to a particular instance of the grain?

I think we'll be able to use the ITransactionCommitter to meet our current needs, but it will require some creativity to work around the areas where it lacks.

[jason-bragg]

@jsteinich

Re: ContextResourceFactoryExtensions - I was in error. Yeah, this is still required and internal, sorry I missed that. Also,I agree that replacing the TransactionalResourceExtension and TransactionManagerExtension is not a good solution.

Re: TransactionQueue - Is the internals of the current implementation and is not really part of the extensibility pattern for transaction orchestration. IMO, it's very fragile and unintuitive code that I don't recommend playing with. We've plans to significantly refactor it to improve testability and maintainability.

“can the GrainReference that makes up most of the ParticipantId actually be used to get back to a particular instance of the grain?”

It does that for normal grains, but not stateless workers, as stateless workers are always considered local to the caller. I think the best one can do in the current system is to use PreferLocalPlacement. This will create a normal grain but try to place it on the same silo as the caller.

Thanks for the feedback, and I understand your frustrations. This is the first release of the transaction system and only the grain facing portions of the pattern (Transaction attribution and ITransactionalState) are being locked down this release. The orchestration patterns (ITransactionReasource, ITransactionManager, and related extensions) are still subject to change and the patterns we use there are not settled. I clarify this not to excuse the issues you’ve encountered, only to communicate that the extensibility points you’re working with have not been completely vetted yet.

For early adopter efforts like yours, I’d suggest forking the orleans repository, making changes to the transaction orchestration needed for you to implement your patterns, use custom nugets built from your fork, and create pull requests back to Orleans for the changes you’ve made. This will allow you to move forward faster, while integrating the changes you need back into Orleans. Once Orleans releases catch up with your fork, you can switch back to the released Orleans nugets.

[jsteinich]

We were able to get the ITransactionalStateStorage approach working for our needs. It does feel a bit hacky and was a bit more complicated than expected.

Thanks again for the suggestion.

[ghost]

Thanks for contacting us. We believe that the question you've raised has been answered. If you still feel a need to continue the discussion, feel free to reopen the issue and add your comments.