One Single View can be updated by multiple Single View Creators to optimize performances

[flower-of-the-bridges]

Which area of this project could be most improved?

Fast Data

Description

Use Case

Consider the following mapping contained in an aggregation of a single view collection named users, that contains user information along with the orders that has been made on a e-commerce website:

{
  "userId:" "USER.ID", (PK)
  "field1": "PROJECTION_1.FIELD_1",
  / * other fields read by different projections */
  "fieldN": "PROJECTION_N.FIELD_N",
  "orders": [{
    "trackingNumber": "ORDERS.TRACK_NUMB",
    "price": "ORDERS.PRICE"
  }]
}

The problem

This single view is aggregated each time there are variations in the projections, which are PROJECTION_1 + PROJECTION_N + USERS + ORDERS = N + 2

Now imagine that the projection ORDERS has a very high rate of updates, e.g 1 update per second (yes, our user has a lot of money to do orders): this means that each time an ingestion message is received and the corresponding projection change is updated, N + 2 queries are performed to the MongoDB cluster. This means that, in a minute, 60 * (N +2) queries are performed. As N increases, the performance of the whole database can degrade.

Optimization

To optimize the number of queries, you can split the current Single View Creator (SVC) into two different ones and separate the aggregation fields by using a particular configuration for each of em (you can read the docs for the complete list of the environment variables available).

In this example, we can split the current single view into two different single view creators, namely SVC1 and SVC2:

• SVC1 is delegated to compute the aggregation for the fields related to PROJECTION_1 ... PROJECTION_N:
  • Configuration: the microservice should have the following environment variables
    • SINGLE_VIEWS_COLLECTION: users → the SVC will write aggregation result on a record in the collection users
    • TYPE: articolo → the SVC will read projection changes with the filter {type: users}
    • UPSERT_STRATEGY: update → once the aggregation is completed, the SVC won't replace the whole document as the default behaviour, but instead will keep the existing one and update just the fields computed by the aggregation
  • Aggregation: the SVC aggregation configuration will have the following mapping

    {
       "userId": "USERS.ID", (PK)
       "field1": "PROJECTION_1.FIELD_1",
      / * other fields read by different projections */
      "fieldN": "PROJECTION_N.FIELD_N",
    }

• SVC2 is delegated to compute just the aggregation of the orders:
  • Configuration: the microservice should have the following environment variables
    • SINGLE_VIEWS_COLLECTION: users → the aggregation result will be written on a record in the same collection used by SVC1, which is users
  • TYPE: users-orders → the SVC will read projection changes matching the filter {type: users-orders}
  • UPSERT_STRATEGY: update → once the aggregation is completed, the SVC won't replace the whole document as the default behaviour, but instead will keep the existing one and update just the orders array computed by the aggregation
  • Aggregation: the SVC aggregation configuration will have the following mapping

    {
       "userId": "USERS.ID", (PK)
       "orders": [{
         "trackingNumber": "ORDERS.TRACK_NUMB",
         "price": "ORDERS.PRICE"
       }]
    }

Following this approach:

• when updates from USERS, PROJECTION_1, ..., PROJECTION_N are received by the ingestion components, only N + 1 queries will be performed by the aggregation process of SVC1
• when updates from USERS, ORDER_1, ..., ORDER_N are received by the ingestion components, only 2 queries will be performed by the aggregation process of SVC2

Both SVCs must include the primary key of the single view to obtain the identifier

The single view can be divided further into different SVCs to improve performances of the remaining N + 1 queries performed by SVC_1

How to achieve this from MIA Console

This optimization can be obtained from the Mia Platform Console in the following way: (please note that I’m using v10.5.8, so without the no-code feature)

1. Create two different single views from the Single View's Fast Data section:
   • users → is the single view where the whole schema definition is included, SVC1 is linked and the Strategy section has linked the strategies for projections USERS, PROJECTION_1...PROJECTION_N. This means that for each projection involved, the RTU will produce a projection change having {type: users}
   • users-orders → an empty single view definition where only the SVC2 is linked and the strategy for projections USERS, ORDERS is linked. This means that for USERS and ORDERS projections, the RTU will produce a projection change having {type: users-orders}

Note that users-parallel will result in an empty collection.

Conclusions

• with the environment variable SINGLE_VIEWS_COLLECTION, you can tell SVC2 to write the result on the same collection of SVC1;
• with TYPE: <a different type> each SVC related to the same SINGLE_VIEWS_COLLECTION variable can read updates from different sources;
• with UPSERT_STRATEGY: update, you can tell the different SVCs that the content of the aggregation will be upserted and not replaced in the record contained in the output collection.

Possible Improvements

The Mia Console should enable this optimization by design by having the opportunity to link to one single view definition multiple SVCs, where each one can have the responsibility to update only certain fields.

[ianFar96]

Hi @flower-of-the-bridges, thank you for your suggestion, we have actually added a use case description in the documentation some weeks ago, you should be able to see it in the docs on the next release. For the configuration side, the console should already enable you to connect more than one Single View Creators to a Single View where you can associate the same er-schema but with different aggregation. Are we missing something or is it just a matter of console versions?

[flower-of-the-bridges]

Hi @ianFar96

Thanks for the information, I know that in newer versions you can have different SVCs linked to one single view. However, they must have the same TYPE environment variable.

This limit still leads to relying on a temporary single view to have different TYPE environment variables that update concurrently the same single view collection.

Also, this use case is not displayed properly in the Fast Data Section. When I link an SVC to a single view I can't see:

• the type configured
• upsert/delete strategies (if adopted)

To see this information, I have to go to the microservice and check them in the environment section and config map section.

IMO the TYPE variable should be managed only from the Single View section instead of the microservice section, since manual edit on the variable itself can change drastically the behaviour of the SVC.