ID Linking Functionality

[weberjm]

In order to support updating/upserting records through our system, a standard method of linking IDs is needed. It has been proposed and decided to provide this functionality with a call made by the Ferryman which will save IDs to the Data Hub and then add that mapping to future Flow Steps.

Description

After each step in which a recordUid is generated, it is transmitted to an external ID registry (implemented through the Data Hub). If appropriate, a oihUid is retrieved or generated, then returned, and injected into the message. This requires that the communication to the Data Hub is synchronous, with a response.

However, instead of having a dedicated step and component to handle this communication, it is directly included within the Ferryman. The Ferryman would be in some fashion notified/configured that ID linking is active for a certain step (e.g. through flow definition and interpreted by the orchestrator). Additionally, it would need some way to know the specific application ID of the current step.

If active, the Ferryman would then automatically attach the ID registry for the current object to the message in a reserved field. In future steps, the Ferryman would first look into this registry (if present), select the relevant recordUid for the current step’s application (if present), and inject it into the message itself before passing it into the connector. If a message with a recordUid is emitted by a connector, the Ferryman will post this new recordUid to the data hub and update the attached ID registry.

Passing along the registry is useful because each instance of the Ferryman only knows (and indeed only should know) about its current step. So if the Ferryman of Step 1 would not know which Application is served in Step 2, and would not be able to pick out the appropriate recordUid ahead of time. That means that picking out the current recordUid would need to happen inside Step 2 before it is passed into the connector. This requires either having the registry as part of the message right there, or fetching it from the Data Hub beforehand. Of these two options, the former would likely be quicker and require less overhead, as much as halving the average number of API calls necessary in a given flow.

Examples

There's a single flow with two connectors, fetching data from Application A and upserting it in application B. No further components are needed Thus, the overall flow simply looks like A -> B

First an object is created in A and inserted in B. Later that same object is updated in A and the update propagates to B.

Insert

1. A-connector fetches a new object from A, where it has the local recordUid abc.
2. The Ferryman checks the messages meta property and notices a present recordUid. It posts this recordUid to the Data Hub. As it is not yet in the registry, the Data Hub creates a new ID registry with the oihUid newObject, saves the record A: abc under this registry, and the whole thing.
3. The Ferryman then attaches this registry to the message for further steps. The registry would now look something like: oihUid: ‘newObject’, recordUids: { A: ‘abc’ }
4. In the next step, the Ferryman notices the presence of the ID registry, but finds no entry for its current Application B, and takes no further action.
5. B-connector receives object, inserts it into B, where it is assigned local recordUid 123
6. B-connector emits the new message with recordUid 123 together with the oihUid newObject that it received
7. The Ferryman posts the tuple of oihUid newObject and recordUid 123 to the Data Hub. The Data Hub looks up the oihUid in the registry, finds it, and adds the record B: 123 to it.

Update

1. A-connector fetches updated updated object with local recordUid abc
2. Ferryman finds recordUid abc and posts it to the Data Hub, which returns the entire ID registry for the associated oihUid.
3. The Ferryman attaches this registry to the message for further steps. The registry would now look something like: oihUid: ‘newObject’, recordUids: { A: ‘abc’, B: ‘123’ }
4. In the next step, Ferryman notices the presence of the ID registry. It finds an entry for its current Application B, with the recordUid 123
5. Ferryman injects recordUid 123 into the message
6. B-connector uses recordUid 123 to look up the correct object to update

Notes

• Connector logic would not have to be adjusted, as passing forward the created recordUid is already spec behaviour
• No additional steps or flows required
• Default behaviour of current flows would remain the same unless ID-linking is explicitly activated
• Overall flow structure remains intact (as opposed to Hub&Spoke, which creates several sub-flows)
• Does require a Data Hub or similar service to serve as registry
• Does require some implementation in both the Ferryman and Orchestrator to pass along new configuration fields
• Actual message content will be modified outside of the component itself, making things slightly less transparent for component developers. This will be handled through documentation and standardization of messaging formats (Validate and formalize connector message format. #1078).
• Message size would be slightly larger due to attached ID registries, but this is likely negligible

[SvenHoeffler]

Basic functionality is implemented as for ferryman version 1.3.1-beta.11 . There's still some outstanding QA and fine-tuning to be done before a proper release, but in the course of development two further questions became apparent. These probably ought to be separate issues, I'll just note them down here for posterity.

• How and where should applicationUids be determined and passed into the ferryman? Currently we just use the nodeSettings field in the flow definition to pass it into, which works well enough, but another solution may be more efficient to cover different use-cases.
• Currently, it's operating under the assumption that recordUids are always entirely unique. If they are not, it may possible that entries may end up unintentionally being linked across tenants.