You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As we're starting a new year and a new development cycle, I'd like to share and discuss the design ideas I have around 3 related topics:
OpenAPI callbacks support (as introduced in OAS 3.0),
OpenAPI webhooks support (as introduced in OAS 3.1),
Sync-to-Async triggers (as introduced as a personal idea 😉 )
These elements were on our roadmap for a long time (https://github.com/orgs/microcks/projects/1?pane=issue&itemId=26464135), and I saw some demands recently on those topics. As these topics are closely related, I spent some time thinking about the best way to add support and implementation in Microcks.
Please consider this discussion as an opportunity to comment and share what you'd like to see and your feelings about the proposed plan. Feel free to react to anything and propose an alternative approach if it sounds better to you!
I'll expose here my analysis on 3 different topics and then different options I have considered for representation and implementation.
Callbacks are defined with both request (from API owner to user) and response (from user server to API owner)
Is the response useful in the context of mocking? I don’t think so…
Callbacks unsubscribe method is not defined by the spec. It’s up to the implementation to provide an /unsubscribe operation and a subscription identification mechanism (based on id or whatever)
I think that we can ignore this point in Microcks and consider we only have ephemeral subscriptions that are consecutive to the /subscribe operation invocation and are not persisted. IMHO this is the sweet spot for OpenAPI callbacks vs AsyncAPI: callbacks are best suited for « events consecutive to an operation one on specific one entity » ; AsyncAPI is well suited for « stream of events not necessarily related to an operation » (this last one tends to be mitigated with « request-response » style integrated into AsyncAPI v3.0 though…)
Webhooks - even if closely related to the callbacks feature - describe requests initiated other than by an API call, for example, by an out-of-band registration. We can typically think here about the webhooks that can be registered via the GitHub administration console and allow third-party apps to receive HTTP notification for events (whatever the entity)
Webhooks sound much more like AsyncAPI event publication to me.
It's a very common pattern to have a synchronous interaction (ie. invocation of Rest/Soap/Graph/Grpc API operation) triggering the publication of an asynchronous event on a channel (and thus an AsyncAPI subscribe operation). So why not be able to define this triggering from the mocking step in API lifecycle?
I see 2 different areas to address, which could be easy to set up in Microcks. The first one is about how to define this link using API specs and in Microcks ; the second one is about how to propagate context from sync request to async message to get coherent content.
Example cases
These are simplified representations with some levels of complexity ignored to make things simple to understand.
Callback
/subscribepost:
requestBody:
examples:
EXAMPLE_1: {}response:
examples:
EXAMPLE_1: {}callbacks:
createEvent:
'{$request.body#/callbackUrl}':
requestBody:
examples:
EXAMPLE1: {}responses:
examples:
EXAMPLE1: {} # May actually be ignored.updateEvent:
'{$request.body#/callbackUrl}':
requestBody:
examples:
EXAMPLE1: {}responses:
examples:
EXAMPLE1: {} # May actually be ignored.
Webhook
webhooks:
# Each webhook needs a namenewPet:
post:
requestBody:
description: Information about a new pet in the systemexamples:
EXAMPLE_1: {}responses:
examples:
EXAMPLE_1: {} # May actually be ignored.
Sync-to-Async triggers
openapi: 3.1info:
title: User registration APIversion: 1.0.0paths:
/user:
post:
requestBody:
examples:
EXAMPLE_1: {"name": "John Doe"}response:
examples:
EXAMPLE_1: {"status": "registered"}x-microcks:
trigger: 'User signed-up API:0.0.1:PUBLISH /user/signups'# Exclusive with `triggeredBy` in AAS
Callbacks are defined as other pairs of Request/Response attached to the same operation and related to the origin example.
In that case, we need a new discriminator (or type of request and response) to tell Microcks it’s a callback-related request or response.
Operation POST /subscribe => 1 Request and 1 Response with operationId pointing to operation => 2 Requests (and 2 Responses) of type Callback with the reference of the event in the Spec + the endpoint expression ({$request.body#/callbackUrl})
Option 2 for callbacks
Callbacks are defined as additional EventMessages attached to the same operation and related to the origin example.
EventMessage already holds a dispatchCriteria that can be used to hold the endpoint expression ({$request.body#/callbackUrl}). We would need an additional property to keep track of event name (createEvent or updateEvent) for the sake of completeness.
Option 1 for webhooks
We can ignore them and simply argue that people have to consider AsyncAPI instead 😉
Option 2 for webhooks
Webhooks are defined similarly to AsyncAPI using EventMessages attached to an operation bound to a REST API in Microcks. In that case, the REST API may have a traditional HTTP-Verb operation + SUBSCRIBElike operation in AsyncAPI (meaning that the consumer will receive events)
Options for Sync-to-Async triggers
Mapping onto Microck’s internal model is obvious as we have 2 different APIs with already existing Request/Response/EventMessage types that can live independently. One thing we should add though is this concept of trigger:
As an independent association table (or collection in the DocumentDB world)
As an additional property on the operation definition.
Supporting both side’s specifications - trigger and triggerBy - can be tricky, and it may lead to inconsistencies. Things would be easier just supporting the OpenAPI side of things with trigger (it would also not trigger an update of unrelated API). The 2nd option seems more natural as it’s where the business logic of event triggering should be located.
Question: should we allow multi-values for trigger property? (Aka triggering multiple events from the same REST operation invocation). I would think so, but I would need confirmation.
Runtime implementation
Option 1 for callbacks
Implement sending back requests in the main web app component. We may use Spring internal ApplicationEvents to have temporal decoupling with the main mock invocation thread (like we’re already doing for metrics gathering or event publication to async-minion). We need a way to wait for a specified (short-)time before sending the event without locking threads or resources (see concern #2 below)
It fits whatever option is used for modeling.
Option 2 for callbacks
Implement sending back requests in the async-minion component. It fits whatever option is used for modeling, but it adds a hard dependency on async stuff, even for « simple » OpenAPI use cases. Here again, we need a way to wait for a specified (short-)time before sending the event without locking threads or resources (see concern #2 below). The scheduler we used on Quarkus is much more appropriate for long-running tasks, not for ephemeral and short-lived notifications.
Option 1 for webhooks
Implement sending back requests in the main web app component. As we just have a basic scheduler on the main component (just for import jobs), this would require duplicating the scheduling and long-running task logic we already have in the async-minion…
Option 2 for webhooks
Implement sending back requests in the async-minion component.
This option fits better with the modeling option 2 for webhooks as we can solely rely on the EventMessage definitions to send stream of messages. The process would be as follows:
At startup: gather all candidate AsyncAPI (as of today) + gather all webhooks EventMessage definitions
Depending on configured frequencies (see concern #4 below), attach AsyncAPI and webhooks via the same AsyncMockDefinition
One different thing would be how we retrieve the target endpoints (see concern #3 below):
Endpoints are static configuration in the case of AsyncAPI
Endpoints may be dynamically registered/un-registered in the case of REST webhooks
Options for Sync-to-Async triggers
Implementation of sending triggered event must be realized in the async-minion component. Triggering command will be materialized as an event sent from the Microcks main component to the async-minion (via Kafka or WebSocket for the uber images) and message reception (filtered by minion-supported bindings) will trigger a single message publication. ProducerManager will need refactoring to allow the reuse of single message publication outside of the scheduled jobs.
How do we reuse content from the incoming request into our triggered event messages? See concern #1 below.
Concerns
We need a way to transfer context from the original request to the callback of the triggered message payload. Template expressions ({{request.body/path}}) seem to fit well here. In the context of AsyncAPI triggering, we may use the presence of {{ (request|response).* }} expression as a filter to know that we should use this as a template for sending messages. These same messages would be ignored for regular AsyncAPI mock events publication.
We need a way to specify/know the delay between original request processing and callback submission. This can be a low random number (2-5 secs) in case of a single callback events but we need explicit information if there is more than 1 callback - in order to keep the temporal logic of events (createEvent must be sent back before updateEvent but Microcks can’t guess this based on event names).
Even if we model Webhooks following the option 2, we’d need a way to realize the out-of-band registration of target endpoints. Contrary to AsyncAPI brokers that can be typically shared by consumers (let’s say - most of the time), webhooks endpoints tend to be multiples (you know, the web nature…). We may need to provide a Microcks based API to allow registration/unregistration of endpoints for webhooks for an API and version.
We need a way to specify/know the frequency for webhook events publication. We can follow here the same logic as for AsyncAPI: default frequency or the one specified via a x-microcks/frequency property at the operation/webhook level.
Summary
Proposed priorities:
OAS Callbacks implementation (based on community requests)
Sync-to-Async triggers (based on experience and to ease implementation of webhooks later on)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hey Microcks community!
As we're starting a new year and a new development cycle, I'd like to share and discuss the design ideas I have around 3 related topics:
These elements were on our roadmap for a long time (https://github.com/orgs/microcks/projects/1?pane=issue&itemId=26464135), and I saw some demands recently on those topics. As these topics are closely related, I spent some time thinking about the best way to add support and implementation in Microcks.
Please consider this discussion as an opportunity to comment and share what you'd like to see and your feelings about the proposed plan. Feel free to react to anything and propose an alternative approach if it sounds better to you!
I'll expose here my analysis on 3 different topics and then different options I have considered for representation and implementation.
References:
Analysis
Callbacks are defined with both request (from API owner to user) and response (from user server to API owner)
Callbacks unsubscribe method is not defined by the spec. It’s up to the implementation to provide an
/unsubscribe
operation and a subscription identification mechanism (based on id or whatever)Webhooks - even if closely related to the callbacks feature - describe requests initiated other than by an API call, for example, by an out-of-band registration. We can typically think here about the webhooks that can be registered via the GitHub administration console and allow third-party apps to receive HTTP notification for events (whatever the entity)
It's a very common pattern to have a synchronous interaction (ie. invocation of Rest/Soap/Graph/Grpc API operation) triggering the publication of an asynchronous event on a channel (and thus an AsyncAPI subscribe operation). So why not be able to define this triggering from the mocking step in API lifecycle?
Example cases
These are simplified representations with some levels of complexity ignored to make things simple to understand.
Callback
Webhook
Sync-to-Async triggers
Model representation
Option 1 for callbacks
Callbacks are defined as other pairs of
Request
/Response
attached to the same operation and related to the origin example.In that case, we need a new discriminator (or type of request and response) to tell Microcks it’s a callback-related request or response.
Operation
POST /subscribe
=> 1 Request and 1 Response withoperationId
pointing to operation => 2 Requests (and 2 Responses) of typeCallback
with the reference of the event in the Spec + the endpoint expression ({$request.body#/callbackUrl}
)Option 2 for callbacks
Callbacks are defined as additional
EventMessage
s attached to the same operation and related to the origin example.EventMessage
already holds adispatchCriteria
that can be used to hold the endpoint expression ({$request.body#/callbackUrl}
). We would need an additional property to keep track of event name (createEvent
orupdateEvent
) for the sake of completeness.Option 1 for webhooks
We can ignore them and simply argue that people have to consider AsyncAPI instead 😉
Option 2 for webhooks
Webhooks are defined similarly to AsyncAPI using
EventMessage
s attached to an operation bound to aREST API
in Microcks. In that case, the REST API may have a traditional HTTP-Verb operation +SUBSCRIBE
like operation in AsyncAPI (meaning that the consumer will receive events)Options for Sync-to-Async triggers
Mapping onto Microck’s internal model is obvious as we have 2 different APIs with already existing
Request
/Response
/EventMessage
types that can live independently. One thing we should add though is this concept oftrigger
:Supporting both side’s specifications -
trigger
andtriggerBy
- can be tricky, and it may lead to inconsistencies. Things would be easier just supporting the OpenAPI side of things withtrigger
(it would also not trigger an update of unrelated API). The 2nd option seems more natural as it’s where the business logic of event triggering should be located.Runtime implementation
Option 1 for callbacks
Implement sending back requests in the main web app component. We may use Spring internal
ApplicationEvents
to have temporal decoupling with the main mock invocation thread (like we’re already doing formetrics
gathering or event publication to async-minion). We need a way to wait for a specified (short-)time before sending the event without locking threads or resources (see concern#2
below)It fits whatever option is used for modeling.
Option 2 for callbacks
Implement sending back requests in the async-minion component. It fits whatever option is used for modeling, but it adds a hard dependency on async stuff, even for « simple » OpenAPI use cases. Here again, we need a way to wait for a specified (short-)time before sending the event without locking threads or resources (see concern
#2
below). The scheduler we used on Quarkus is much more appropriate for long-running tasks, not for ephemeral and short-lived notifications.Option 1 for webhooks
Implement sending back requests in the main web app component. As we just have a basic scheduler on the main component (just for import jobs), this would require duplicating the scheduling and long-running task logic we already have in the async-minion…
Option 2 for webhooks
Implement sending back requests in the async-minion component.
This option fits better with the modeling option 2 for webhooks as we can solely rely on the
EventMessage
definitions to send stream of messages. The process would be as follows:EventMessage
definitions#4
below), attach AsyncAPI and webhooks via the sameAsyncMockDefinition
#3
below):Options for Sync-to-Async triggers
Implementation of sending triggered event must be realized in the async-minion component. Triggering command will be materialized as an event sent from the Microcks main component to the async-minion (via Kafka or WebSocket for the uber images) and message reception (filtered by minion-supported bindings) will trigger a single message publication.
ProducerManager
will need refactoring to allow the reuse of single message publication outside of the scheduled jobs.How do we reuse content from the incoming request into our triggered event messages? See concern
#1
below.Concerns
{{request.body/path}}
) seem to fit well here. In the context of AsyncAPI triggering, we may use the presence of{{ (request|response).* }}
expression as a filter to know that we should use this as a template for sending messages. These same messages would be ignored for regular AsyncAPI mock events publication.createEvent
must be sent back beforeupdateEvent
but Microcks can’t guess this based on event names).x-microcks/frequency
property at the operation/webhook level.Summary
Proposed priorities:
Proposed options:
trigger
custom property support only, as a multi-valued propertyBeta Was this translation helpful? Give feedback.
All reactions