Decouple ServiceControl from the endpoints transport

[SzymonPobiega]

Why decouple?

Reduce the development testing effort by limiting the number of transports supported in ServiceControl. The adapter between the user transport and the SC would be used even when both sides use same infrastructure in order to avoid problems e.g. with incompatible routing technologies

What transports should be supported?

Initially we can limit the support to MSMQ only both on premise and in the cloud. The reason why is the fact that SC currently requires a custom VM anyway (because of embedded RavenDB) so installing MSMQ locally would not be a problem. Ultimately we should also support one of the cloud-native transports (e.g. ASB on Azure and SQS on AWS).

What are the immediate benefits?

There are numerous benefits when going with the adapter approach, some of them available immediately

• Support for SQL Server multi-instance topology by adjusting adapter configuration
• Support for ASB multi-namespace by adjusting adapter configuration
• Support for RabbitMQ non-standard topologies
• Can use single instance of ServiceControl/ServicePulse with multiple transports
• Easier and more robust configuration of SC integration events

What are the long-term benefits?

• Allows to redesign the structure SC input queues that live behind the adapter layer. E.g. the messages from the error queue can be forwarded not only to error queue but also to audit queue, allowing decoupling of audit message storage feature from the recoverability feature
• Allows using dedicated high-performance receive strategies in the adapter
• Allows hosting parts of ServiceControl directly in the adapter layer (e.g. audit ingestion) in order to achieve higher throughput

How would we go about releasing it?

Introducing adapters does not change the data format of SC so it does not require data migrations. Apart from the changes in the handling of integration events, the new version of SC is compatible in the sense it can also work directly, without the adapter in front of it.

• Users who don't have configured integrations would just upgrade and see no change.
• Users who have integrations would have to install an adapter and configure it in the SC UI.

Initially the new release would support all the currently supported transport in order to give users time to migrate. We would inform the users that in months we would stop supporting some transports (e.g. RabbitMQ) and that by this data they need to put in an adapter and switch their SC to one of the supported transports.

Issues solved

• ASB V7 recommended topology is incompatible ServiceControl
• ExternalIntegrationDispatchRequests documents not being processed after a restart

Show me da codez

• Working adapter code https://github.com/SzymonPobiega/ServiceControlAdapter
• SC adjusted to work with adapters [Reference] Prepare SC for transport adapters #870
• NServiceBus.Raw used to implement the adapter https://github.com/SzymonPobiega/NServiceBus.Raw

Can I actually see it?

There is a video showing it work. You can also run the projects in RabbitMq or SqlServer (or both simultaneously) in the adapter solution while using SC from the mentioned PR. Messages randomly fail processing and you can retry them in SC. Some endpoints also include specialized integration endpoints that process SC's notifications.

[SzymonPobiega]

@Particular/servicecontrol-maintainers consider this issue for the maintainer budget.

[johnsimons]

1. The first question we should be asking ourselves is whether ServiceControl actually needs a bus internally.
2. The second question is whether endpoints should really be using the transport to communicate information back and forth to ServiceControl.

Here is my take on the answers:

1. Looking at what ServiceControl does, and how chatty we ended up, I do not think we need a queuing system internally, majority of the actions performed by ServiceControl are background processing of data.
2. This one is definitely a NO, and I actually done some spiking on using WebSockets as the communication mechanism.

[SzymonPobiega]

Looking at what ServiceControl does, and how chatty we ended up, I do not think we need a queuing system internally

I agree. But I still think we'll need some queue in front of ServiceControl. That's why SC needs to use some kind of queue mechanism. When using the adapter approach, we initially have two buffers, on the user side and on the SC side. Then, gradually, we can eliminate the buffers on the SC side, leaving only the queues in front of the adapter

This one is definitely a NO

I don't think this is so black-and-white. While moving the messages directly to the endpoint where they failed has some advantages, it has also some problems e.g. a failed message would have to contain a stable TCP address of the process (IP + Port) or we need to maintain some kind of mapping between queue addresses and TCP addresses.

[mikeminutillo]

My take:

1. I agree. SC does not need a queue to talk to itself internally.
2. I disagree. We need a communication mechanism that is durable and recoverable in between SC and the endpoints. The endpoints are already sending audit and error messages to queues. I think it makes sense to use them to allow us to work with endpoints without any significant modification. For retries we can easily send a message back the input queue of the endpoint and it will ingest it and it will work. For heartbeats, endpoint control, and custom checks I can see an argument to drop the queue and do something else.

[SzymonPobiega]

@mikeminutillo I would like to start working on this. I've raised #872 as a first step.

[janovesk]

Pretty sure I like this :)

[SzymonPobiega]

Closing as outdated.