Multi partition support - move service bus out of the Data Partition Configuration.

[ashley-kelham]

Indexer queue needs to be triggered by the service bus. Storage needs to send the message to service bus. To support multiple partitions the easiest way to configure this for the time being is to have a single service bus rather than one per partition.

How messaging works in a multi partition setup will be rationalized when the notification service is deployed and integrated so this simpler solution for the current setup is preferred as it will need to be re-worked either way.

This service bus should live in the common resource group as if it was in services and it got recreated the connections to it would be broken.

[danielscholl]

@ashley-kelham -- So I'm trying to understand what exactly you are saying here. We are saying that until Notification Service comes online we will only use 1 Service Bus and there will be no planned changes for the indexer queue functionality and data partition service and all services will for now only use 1 Service Bus.

Based on that, what rational are you using that Common is the correct place for that. My preference would be to move Service Bus over to the ServiceResources group.

@daniakodeih & @imran-siddique - Please note this is a major architectural decision point being discussed here.

[ashley-kelham]

Correct.

The implementation of indexer queue is currently being reviewed but it shouldn't need changes as we see it because it uses the partition id header and only forwards the requests onto indexer service.

The service bus connection information would still be passed into Key Vault and retrieved by the partition service. This means storage service would change to pull the service bus connection information from the partition service. It would just pull the same connection info for every partition (but the service would be ignorant of this).

I was thinking of the use case of re-creating the service rg. If the service bus was recreated then the connection information (secret) would change but not updated in Key Vault. This would mean Storage service would be broken.

[danielscholl]

I believe that regardless of what configuration we put the Service Bus a configuration that creates something would have to own the responsibility of ensuring that information is in KV. We currently have components that we are doing this in the Service Resource Configuration (postgres, redis, storage) that is used for Airflow. I think the proper place for now actually is to move the Service Bus up into Service Resources. I'll note that and ensure our architecture diagram is updated.

[danielscholl]

@ashley-kelham Okay I'm getting ready to start adding this piece in. I think the approach that is best to take for now is the following. Leave SB as part of the partition resource group but while multiple exist only 1 is used. This will save me work later.

[ashley-kelham]

That's good thanks Daniel, how best do we document this not to confuse consumers? Or will the deployment diagram change to only show one service bus?

[danielscholl]

Closing this issue