Question: App-specific context metadata

[pierreneu]

Question Area

[ ] App Directory
[ ] API
[X] Context Data
[X] Intents
[ ] Use Cases
[ ] Other

Question

Right now Symphony raises intents such as ViewInstrument, with a fdc3.instrument context, allowing the action to be performed in another app that is configured to handle that intent and context pair.

However, we'd like to include more information (metadata about the interaction) in the context data, such as the chat room id, message id, user email/id etc. as this might be useful to the receiving app. However, adding it to the instrument object is not semantically correct (it doesn't relate to the instrument.

This additional metadata might be added not only to the ViewInstrument intent, but any outgoing intents coming from out application.

Options we think of:

1. Add custom properties to the fdc3.instrument e.g. com.symphony.originatingchatroomid. This is not semantically correct as custom properties in the standard are reserved to identifiers and the metadata relates to the interaction rather than the instument/security.
2. Add an optional object nested in the fdc3.instrument. It can be either something specific to the type of context such as fdc3.chatinfo or something much more generic such as fdc3.appcontext. However, this has the same issue as the first suggestion.
3. Change the context type to a new context type that would wrap a fdc3.instrument. This however breaks interoperability with apps that are only listening to ViewInstrument(fdc3.instrument)
4. Add properties to the ContextMetadata that the receiving app MAY receive (Originating App Metadata). ContextMetadata is currently supplied by the Desktop Agent, we would need a way to supply additional information to the Desktop Agent to augment it.

Based on a side discussion with Kris, it seems that the current standard may not provide a perfect solution.

[kriswest]

This seems like a use-case that others would run into (wanting to provide additional, optional metadata about an interaction). Wrapping context types to do this does indeed break the coupling between apps (which is based on specific intent/context type pairs) and adding the metadata into context types affects any that you use (including non-standard proprietary types) and does not seem practical.

Hence (as discussed off to the side), what I'd propose is that we extend ContextMetadata so this can contain some additional fields (we wrapped the only field it currently carries, AppIdentifier, in ContextMetadata for exactly this sort of future-proofing). We could then extend other functions with a new optional argument to include that metadata, e.g.
broadcast(context: Context): Promise<void>;
changes to:
broadcast(context: Context, sourceMetadata?: Context): Promise<void>;

raiseIntent(intent: string, context: Context, app?: AppIdentifier): Promise<IntentResolution>;
changes to:
raiseIntent(intent: string, context: Context, app?: AppIdentifier, sourceMetadata?: Context): Promise<IntentResolution>;

and ContextMetadata changes from:

interface ContextMetadata {
  /** Identifier for the app instance that sent the context and/or intent. 
   *  @experimental 
   */
  readonly source: AppIdentifier;
}

to

interface ContextMetadata {
  /** Identifier for the app instance that sent the context and/or intent. 
   *  @experimental 
   */
  readonly source: AppIdentifier;
  /** Additional optional metadata that an app may choose to send with a broadcast context or raised intent.
   *  For example to provide source information from within the source application (e.g. from which chat room
   *  (in a chat app) or order (in an OMS) did the message originate from.
   *  @experimental
   */
  readonly sourceMetadata: Context;
}

You could then pass in your ChatRoom context as the sourceMetadata.

ContextMetadata/Originating app metadata is currently an optional feature (there is an issue open to consider whether that should be required, see #735). But it looks like adding this wouldn’t break anything and could make it more useful… The additional §sourceMetadata§ would become available but the interaction is not dependent on it - making it optional.

However, it is an API change so it’d have to happen in an FDC3 2.1…

[nemery-flextrade]

+1 on this as a concept. Especially useful for case Pierre mentioned whereby downstream systems can pick up chat IDs for subsequent messaging but I can see it being useful in other situations as well.

[robmoffat]

It's not really clear from the description exactly what the use case is here. It just says that there might be extra data useful to the receiving app. It would be really handy to have an exact example of this so that I can think about it the same way. At the moment, I am wondering a bit why a charting application would want to know anything about chat rooms, for instance.

Leaving aside any changes to FDC3, I think this is what you could do at the moment:

setup

1. ChartIQ exports the ViewInstrument intent for a fdc3.instrument context.
2. Symphony exports a PublishToChat intent for the fdc3.chart context.

in action

user wants to turn a ticker into a chart and publish that chart into a chat room

3. Symphony is set up to broadcast context of the fdc3.chat.initSettings that the user is currently in, as the user navigates between chats.
4. A user is looking at an instrument ticker in a Symphony chat, and raises the ViewInstrument intent, with the fdc3.instrument context for that ticker.
5. ChartIQ shows the instrument's chart.
6. The user then raises the PublishToChat intent, which uses the fdc3.chart and also the last broadcast fdc3.chat.initSettings context.

There are a few problems with this:

• First, symphony is going to be broadcasting chat/room contexts information a lot. I'm not entirely sure this is a bad thing, since it would let other applications know which chat the user is engaged with.
• Second, there is no linkage from the chat room to the instrument, so you don't necessarily know if the instrument is coming from the latest broadcast chat.
• Third, the latest broadcast chat might change while you're viewing the instrument in ChartIQ. So, PublishToChat is going to do different things depending on which chat you're looking at when you call it.

Am I on the right track understanding this?

[kriswest]

@robmoffat @pierreneu - picking this thread up again (sorry for the long delay).
You are on the right track.

I think the way to describe the proposal is: when broadcasting a context or raising an intent/context pair, you should be able to attach an additional context that represents a source or similar reference for it. This reference may be used for a secondary purpose such as logging or taking follow-up actions.

Re: the @robmoffat foresaw:

First, symphony is going to be broadcasting chat/room contexts information a lot. I'm not entirely sure this is a bad thing, since it would let other applications know which chat the user is engaged with.

• Adding a little extra optional data to messages, to support further interop operations seems like a good thing - but leakage of information about user activity is obviously bad. However, the additional information is optional, so Symphony and similar apps don't have to broadcast it with every message - rather just add it when it`s useful. They may also use intents or app channels & private channels to prevent unintended leakage.

Second, there is no linkage from the chat room to the instrument, so you don't necessarily know if the instrument is coming from the latest broadcast chat.

• The linkage between the context message (for either broadcast/raiseIntent) and the metadata is provided by them being received in the same message: the source context is an additional property on the existing ContextMetadata:

type ContextHandler = (context: Context, metadata?: ContextMetadata) => void;
type IntentHandler = (context: Context, metadata?: ContextMetadata) => Promise<IntentResult> | void;

interface ContextMetadata {
  readonly source: AppIdentifier;
  readonly sourceMetadata: Context;  //<- this is the proposed new bit
}

Third, the latest broadcast chat might change while you're viewing the instrument in ChartIQ. So, PublishToChat is going to do different things depending on which chat you're looking at when you call it.

The metadata is tied to a specific broadcast, rather than a channel. Hence, if you want the sourceMetadata for a specific past message, you need to keep hold of it, rather than using whatever was the most recent one.

I think this is interesting because it adds a bit more (optional) nuance to context messages that can make follow-up actions easier/better.

[robmoffat]

I disagree with this proposal for the following reasons:

1. You're sending two items of context, one as the "regular" payload, the other buried in metadata. It's not clear why one should be in one place and the other in the other. I mean, it might be clear in this particular use case, but it could easily be abused.
2. Listeners should be allowed to ignore the metadata and just receive the context. However, if the context is buried in the metadata object, they will ignore it- but maybe the metadata context is the context data they're subscribed to, and interested in?
3. Somehow, you're expecting ChartIQ to honour the original metadata and pass this back through as metadata in the next intent. It needs to think: ok, when I was handed this fdc3.instrument there was metadata X, and so when I call an intent, I also need to supply metadata X going back again. Which seems like an imposition on other apps.
4. I'm not 100% convinced that the users aren't going to want to control which chat the chart gets sent to. If they are, shouldn't this be done with a chat picker when the intent comes back to Symphony?

I'm going to suggest different solutions to each of these in turn.

Issues 1 & 2

In actual fact I think there are lots of use cases which are similar to this that aren't supported by FDC3 and wouldn't be covered by this solution anyway. For example:

• A counterparty books a trade which changes aposition on a given instrument
• A contact posts a chart in a chat room

In both of these, there are multiple pieces of context data for the same event.

I think it would make more sense if the ContextMetadata could contain some kind of optional, app-supplied eventId which would allow other apps listening to a channel to correlate context data back into the event they came from, if this was needed:

• Were the app only subscribed to a single type of context data, it might not care about this.
• Were it subscribed to two or more types of context, it could perform this correlation.
• As with the examples above, we can support any number of context items being bound together.

Issue 3

Rather than ChartIQ being expected to handle the context data linking, Symphony could do it if It wants to. That is - it should be aware of the fact that an intent was raised in a given room with a given instrument. Since the Chart context item coming back also contains the same instrument it could figure out which room raised the chart in the first place.

Issue 4

But actually I think this is the simplest solution to the problem - just providing a picker list of chats with the most recent ones the user has engaged in being first on the list.

[kriswest]

(n.b. attempting to clarify the proposal here, not advocating for it)

@robmoffat Re: your issues 1&2, i think the easiest way to understand this is that the existing context is all you ever need for the initial action and the metadata can 100% be ignored. In your examples:

A counterparty books a trade which changes a position on a given instrument
A contact posts a chart in a chat room

All of this data would go in the 'primary' or existing context argument:

• In the first one you might have an order context, that embeds an instrument and counterparty. The position is updated later after the trade happens (trade might be returned as a result, but probably happens later)
• In the second example you raise an intent and pass a context that encodes the message, chat room, and any attachments.

I.e. I don't think these examples are affected by the proposal - but they are a good reminder that the design of contexts is important and that they should contain at least the minimum required detail for follow-up actions/responses from the resolving application..

Re: issues 3 & 4 - I don't think any sort of contract needs to be honoured. Rather the metadata could be used to inform a secondary action such as a reply. As you state, there are other ways that could be dealt with such as interaction history in the chat application and the user could/should absolutely be given control.

To sum up, I see this proposal as trying to provide some additional contextual information on the action/raised intent coming from the first application so that it can be used to better formulate or direct a response back from the second application.

As you point out, there are ways that the first application could itself try to internally direct or handle the response it receives from the second application using its own records (which could include recent chats, which apps resolved intents it sent outwards etc.) - although perhaps with a bit less precision.

I know we'll receive feedback that this adds complexity to FDC3 - which is fair. It might reduce complexity in apps that need this feature - but I doubt that will be enough of a sue case to see it approved.

An entirely alternative approach that could be used is to make no changes to the API at all, and instead add a sourceContext or similar element to individual context types or even Context itself (so all inherit it, as optional) to encourage including the desired source info in the primary context.

P.S. thanks for your analysis on this one @robmoffat - has certainly helped me come to a conclusion on how I feel about it!

[kriswest]

@pierreneu @nemery-flextrade Let us know if you get a chance to review this and have thoughts on the alternative proposal (add something to the primary context if you want to provide source metadata).

[robmoffat]

@kriswest said:

In the first one you might have an order context, that embeds an instrument and counterparty. The position is updated later after the trade happens (trade might be returned as a result, but probably happens later)

Yes, but that's the point. If I am an application just listening for instruments, then the instrument embedded in the order context wouldn't come to me.

Sometimes, that behaviour might be good. Other times (like, I think, in the examples I give) it would be better if it were possible to be able to subscribe to context items individually.

[kriswest]

For channels, you can subscribe to the individual types and the advice is for the app sharing context to broadcast all the different types you might want other apps to bind onto (see https://fdc3.finos.org/docs/api/spec#broadcasting-and-listening-for-multiple-context-types) - personally, I'm not a fan of this, and had an alternative proposal (and util function) for recursing through a context to find types you support and can act on - but the decision went the other way and picked something 'simpler' (which seems to actually create more complexity 😱).

For intents, it's a bit different as you can only send one thing. However, you can research the options before you do with findIntent (although you have to call it with each type you could send as AppIntent doesn't tell you what context is needed). That said, I think the use case is generally different for an intent as it's a specific action, rather than synchronization of context, and that will usually have specific data requirements for that action (although I can think of one counter-example, e.g. ViewChart with Chart or Instrument).

At any rate, I don't there's any suggestion that the proposed additional data should ever be needed to perform a requested action - only ever inform later subsidiary actions (logging, responses etc.).

If anything were to be done about this particular issue, I think it would need further advocacy and use cases from a number of participants.

As for your thoughts on extracting embedded contexts - great minds think alike ;-). I think it a separate issue, however.