Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message Queue Architecture and Options #300

Merged
merged 56 commits into from Nov 7, 2019
Merged
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
523d8c1
Create message-queue-architecture.md
tariqjawed83 Sep 26, 2019
92d90d8
Update message-queue-architecture.md
tariqjawed83 Sep 27, 2019
b558ddb
Update message-queue-architecture.md
tariqjawed83 Sep 27, 2019
4709442
formatting and images
tariqjawed83 Sep 27, 2019
82a44b4
minor update
tariqjawed83 Sep 27, 2019
acdc1ff
table fix
tariqjawed83 Sep 27, 2019
f4e8d27
legends
tariqjawed83 Sep 27, 2019
9b9ea2e
AWS EventBridge Evaluation
tariqjawed83 Sep 27, 2019
5682e86
AWS MQ added
tariqjawed83 Sep 27, 2019
d49085c
AWS SQS added
tariqjawed83 Sep 27, 2019
4ebbbd9
AWS Kinesis added
tariqjawed83 Sep 27, 2019
91f6e1f
link
tariqjawed83 Sep 27, 2019
7a1a83f
link fix
tariqjawed83 Sep 27, 2019
e929e85
link update
tariqjawed83 Sep 27, 2019
27457ad
chk
tariqjawed83 Sep 27, 2019
25f5fe9
chk 2
tariqjawed83 Sep 27, 2019
5516007
chk 3
tariqjawed83 Sep 27, 2019
b68b348
updating all links
tariqjawed83 Sep 27, 2019
590953f
added index
tariqjawed83 Sep 27, 2019
1fa019f
TOC added
tariqjawed83 Sep 27, 2019
df88a75
toc
tariqjawed83 Sep 27, 2019
6ea00c0
Manual Indexing
tariqjawed83 Sep 27, 2019
2391bbd
minor mistakes
tariqjawed83 Sep 28, 2019
b5039a4
legend image added
tariqjawed83 Sep 30, 2019
dbc9a6a
reducing size of legend image
tariqjawed83 Sep 30, 2019
9ec4de2
zoom adjustment
tariqjawed83 Sep 30, 2019
5c7cb97
AWS MQ architecture adjustments
tariqjawed83 Sep 30, 2019
801da78
adding Kafka to the list
tariqjawed83 Oct 15, 2019
6ad7a06
commit correction
tariqjawed83 Oct 15, 2019
ff9640d
added azure service bus
tariqjawed83 Oct 18, 2019
2aed356
few correction on availablity vs possiblity
tariqjawed83 Oct 18, 2019
2869602
language support
tariqjawed83 Oct 18, 2019
1d5fc2b
additional information for library support
tariqjawed83 Oct 23, 2019
0df916b
Added Enqueue library
tariqjawed83 Oct 24, 2019
4e89cab
sendEvent correction for Enqueue
tariqjawed83 Oct 24, 2019
b497bed
link correction
tariqjawed83 Oct 24, 2019
a148303
Poll Image Fix
tariqjawed83 Oct 24, 2019
97bfb29
table split for summary
tariqjawed83 Oct 24, 2019
a27e4e6
spell fixed
tariqjawed83 Oct 24, 2019
ce0533a
Added Adobe I/O to the list
tariqjawed83 Oct 25, 2019
4107f56
few fixes on links
tariqjawed83 Oct 25, 2019
739aa7f
minor fix
tariqjawed83 Oct 25, 2019
b487e17
spell fix
tariqjawed83 Oct 25, 2019
d857be8
Update design-documents/message-queue/message-queue-architecture.md
tariqjawed83 Oct 28, 2019
fa90073
updating class diagram for ConsumerRunner
tariqjawed83 Oct 28, 2019
0a6ef64
updated overview
tariqjawed83 Oct 30, 2019
7a41e8a
refactoring document and Use Case Summary
tariqjawed83 Oct 31, 2019
28443a8
image links fixed
tariqjawed83 Oct 31, 2019
bc95d8d
summary of technologies
tariqjawed83 Oct 31, 2019
5d4534b
digram correction
tariqjawed83 Oct 31, 2019
cfd5a87
correcting sync topic concept
tariqjawed83 Nov 4, 2019
708ca05
added proposal for implementation using Enqueue
tariqjawed83 Nov 6, 2019
8ac0bf7
added interrupt based consume process
tariqjawed83 Nov 6, 2019
efd26e3
minor spell corrections
tariqjawed83 Nov 6, 2019
9023374
added queue-interop library details
tariqjawed83 Nov 6, 2019
3e37546
Interop option detail
tariqjawed83 Nov 6, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -0,0 +1,26 @@
#### 1- AWS EventBridge

This comment has been minimized.

Copy link
@nuzil

nuzil Nov 4, 2019

Contributor

Btw there are already some project, maybe you can reference on it
https://github.com/magento-engcom/amazon-event-bridge


AWS EventBridge is a serverless event bus, it facilitates receving data from your application & third parties to AWS Services. Currently it seems like the Targets are specifically AWS Services. These targets are set using specialized rules. Following targets can be specified as of now

[EventBridge Targets](https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutTargets.html)

##### High Level Architecture

![product-page-diagram-EventBridge_How-it-works_V2@2x](AWSEventBridgeArchitecture.png)



##### Evaluation Table - Details

| Method | Evaluation | Implementation Readiness |
| ------------- | ------------------- | ------------------------------------------------------------ |
| dequeue() | Not Possible or N/A | Multiple **Targets** can be set, to receive the messages when they are available asynchronously. There is no concept of fetching the message from the Event Bus on-demand, its more of a serverless architecture. |
| acknowledge() | Not Possible or N/A | There is no need to acknowledge the message, AWS internally makesure that the Target receives the message. |
| subscribe() | Not Possible or N/A | AWS related Targets can be set or subscribed for the EventBus based on the Rules; but we cannot set PHP Callback as Functions. |
| reject() | Not Possible or N/A | The concept is not available or used. |
| push() | Available | PutEvents or PutPartnerEvents functions can be used for this purpose. |



<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
@@ -0,0 +1,24 @@
#### 4- AWS Kinesis

AWS Kinesis is a streaming based distributed messaging technology; it uses publish/subscribe mechanism for loose coupling between senders and receivers. It is designed for extremely high throughput for realtime applications.

##### High Level Architecture

Streaming and the concept of **Stream** itself is the central idea behind Kinesis. It is pretty similar to Apache Kafka with some differences. It is also suitable for implementing Event Sourcing and CQRS pattern, which is commonly used in Microservices Architecture because of the out-of-the-box support for high throughput messaging and publish/subscribe mechanism.

![product-page-diagram_Amazon-Kinesis-Data-Streams](AWSKinesisArchitecture.png)

##### Evaluation Table - Details

| Method | Evaluation | Implementation Readiness |
| ------------- | ---------- | ------------------------------------------------------------ |
| dequeue() | Possiblity | getRecords() - ShardIterator needs to be managed behind the scenes. Stream can be Queue name. It can use getShardIterator(), before the call. |
| acknowledge() | Possiblity | Need to maintain the ShardIterator & SequenceNumber associated with it, using getShardIterator(). We can save NextShardIterator from getRecords() to acknowledge the message. |
| subscribe() | Workaround | Workaround - batch reads with long polling can be implemented, example [Long Polling Subscribe Mechanism in AWS Kinesis](https://github.com/kaliop-uk/kueueingbundle-kinesis/blob/master/Adapter/Kinesis/Consumer.php) |
| reject() | Possiblity | If we don't move the ShardIterator to NextShardIterator, we are pretty much staying on the same message. |
| push() | Possiblity | Since you will have to provide stream, data & partition; we need to have some strategy to for partition selection; and need to maintain these values for Consumers. |



<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
@@ -0,0 +1,44 @@
#### 2- AWS MQ

AWS MQ is a Message Broker based on popular Apache ActiveMQ; it supports multiple protocols for connectivity for instance AMQP, JMS, STOMP, NMS, MQTT and WebSocket.

| # | Important Features |
| ---- | -------------------------------- |
| 1 | Queues & Topics with Ordering |
| 2 | Transient & persistent messaging |
| 3 | Local & distributed transactions |
| 4 | Request/reply |
| 5 | Message filtering |
| 6 | Scheduled messages delivery |
| 7 | Large message sizes |



##### High Level Architecture

Since its a managed service, it provides multi zone fault tolreance and resiliancy out of the box.

<img src="AWSMQArchitectureNew.png" alt="image-20190927212814114" width="50%" />



##### Evaluation Table - Details

Most of the features are available since Magento is also using AMQP protocol with RabbitMQ, but the protocol version is different, RabbitMQ uses 0.9 and Amazon MQ is using AMQP 1.0; so any migration would require porting of Queues from RabbitMQ to AWS MQ. There might be some changes and adjustments to the QueueInterface implementation code to accomodate the new protocol differences. AMQP 1.0 is a completely different protocol.

This comment has been minimized.

Copy link
@nuzil

nuzil Nov 4, 2019

Contributor

Not sure its easiest task, related to backward compatibility. Maybe even make sense to make its as separate module. This part have to be checked


Another challenge is that we have lack of any good implementation of AMQP 1.0 protocol for PHP. There is port of C library that is been recommended for Azure Message Bus, but technically it should work with AWS MQ as well.

[azure-uaqmp-c PHP Bindings for AMQP 1.0](https://github.com/norzechowicz/php-uamqp)

[Exported PHP Modules from native C Library](https://github.com/norzechowicz/php-uamqp/tree/master/ext/src/php)

| Method | Evaluation | Implementation Readiness |
| ------------- | ----------- | ------------------------------------------------------------ |
| dequeue() | Available | receive() |
| acknowledge() | Available | accept() / release() |
| subscribe() | *Workaround | Long Polling might need to be implemented, unless we find a good library that supports AMQP 1.0 in PHP; Java has full support though. |
| reject() | Available | reject() |
| push() | Available | sendMessage(Message, Destination) |

<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
@@ -0,0 +1,32 @@
#### 3- AWS SQS

AWS SQS is a distributed & fault tolerant Queuing Technology; it provides point to point connectivity. It can be used with SNS to add publish / subscribe mechanism as well. Single message gets replicated across different SQS Servers.

##### High Level Architecture



![image-20190927145647952](AWS_SQSArchitecture1.png)



SQS uses Visibility Timeout to prevent other consumers to receive the same message, during which a consumer has to Delete the message explicitly afrer processing it or the message will be available for others for reuse.

![image-20190927150306008](AWS_SQSArchitecture2.png)



##### Evaluation Table - Details

| Method | Evaluation | Implementation Readiness |
| ------------- | ---------- | ------------------------------------------------------------ |
| dequeue() | Available | ReceiveMessage() - possiblity with many available options for instance long & short polling. |
| acknowledge() | Possiblity | DeleteMessage() for positive acknowledge, by default message locked for **Visibility Timeout** period for other consumers. |
| subscribe() | Workaround | ReceiveMessage() - Using polling based mechanism, it can be implemented; but not exactly as true callback mechanism. |
| reject() | Possiblity | The messages are auto visible again for consumption, if explicit DeleteMessage() is not called before timeout, as explained above. |
| push() | Available | SendMessage() |



<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
Binary file not shown.
@@ -0,0 +1,36 @@
#### 8- Adobe I/O

Adobe I/O is a serverless event driven platform that allows you to quickly deploy custom functions/code in the cloud without any server setup. These functions executes via HTTP requests or Adobe I/O Events. These Events can be orchestrated with Sequences & Compositions. It is built on top of Apache OpenWhisk framework.

Events are triggered by Event Providers within Adobe Services, for instance Creative Cloud Assets, Adobe Experience Manager & Adobe Analytics. To start listening to events for your application, you need to register a Webhook (URL endpoint) specifying which Event Types from which Event Providers it wants to receive; Adobe pushes events to your webhook via HTTP POST messages.

*"Magento SaaS based next generation platform can push it's Events on Adobe I/O Events like other Adobe Services, to be consumed by Developers through Adobe I/O Runtime for custom functionality and integrations. But it is not a right candidate for Magento Event Bus"*

[Adobe I/O Runtime Docs](https://www.adobe.io/apis/experienceplatform/runtime/docs.html)

[Apache OpenWhisk](https://openwhisk.apache.org/)

[Adobe I/O Events](https://www.adobe.io/apis/experienceplatform/events.html)

##### Example Architecture

Here is a nice example of Slack integration with Adobe Experience Manager (AEM) for asset change notification,

<img src="https://miro.medium.com/max/1920/1*ajkz4p7Q8Dc0BbrcOo6TpQ.jpeg" alt="Adobe I/O Events" width="80%" height="80%" />

[For more details follow the link](https://medium.com/adobetech/monitoring-aem-asset-updates-with-adobe-i-o-events-9c2a8395880d)



##### Evaluation Table - Details

| Method | Evaluation | Implementation Readiness |
| ------------- | ------------------- | ------------------------------------------------------------ |
| dequeue() | Not Possible or N/A | There is not a concept of explicit fetching of event, rather you define a trigger/event and the actions associated with it. |
| acknowledge() | Not Possible or N/A | This concept is not used, the architecture is funadementally different |
| subscribe() | Not Possible or N/A | A PHP callback function is not possible, although a custom webhook (http endpoint) can be configured to be triggered for a particular Event. |
| reject() | Not Possible or N/A | This concept is not used, the architecture is funadementally different |
| push() | Not Possible or N/A | Events are triggered by Adobe SaaS Services in the Adobe Cloud as discussed above. |

<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

@@ -0,0 +1,30 @@
#### 5- Apache Kafka

Apache Kafka is a popular open-source stream-processing / messaging platform; its by design distributed, replicated & resilient (or fault tolerent) which can acheive very high throughput.

##### High Level Architecture

Topic and Publish / Subscribe mechanism is at the core of Kafka. Effective for implementing Event Sourcing and CQRS pattern, which is commonly used in Microservices Architecture. It is also used for variety of streaming use cases, which requires near real-time processing of records.

<img src="Apache_Kafka_HLD.png" alt="image-20190927212814114" width="60%" />



##### Evaluation Table - Details

Consumer is usually part of Consumer Group, it ensures that each group receives a copy of the message from the topic. Consumer needs to know its offset and partition, although parition can be automatically assigned when you begin consuming data from the topic, but you can also choose to manually assign parition, but these two cannot be mixed up. Consumer first needs to subscribe itself to the list of topics.

After you read the message(s), you can either configure auto-commit or allow manual commits for the offsets, in real-life use-cases you probably would want to control manually based on your strategy.

| Method | Evaluation | Implementation Readiness |
| ------------- | ---------- | ------------------------------------------------------------ |
| dequeue() | Possiblity | Initiate **poll () or consume()**. If there are records available, the call will immediately returns, otherwise it will wait for specified timeout which can be passed as parameter. |
| acknowledge() | Possiblity | There are several ways to commit the offset, which indicates that a particular consumer has consumed those messages. The way you call commit API controls the delivery semantics. |
| subscribe() | Possiblity | There are multiple ways in which the subscribtion mechanism can be implemented, the default Kafka subscribtion is telling Kafka which topics a consumer is interested in. But we can also subscribe a callback function; and we can use Kafka Stream API to receive messages in near realtime. |
| reject() | Possiblity | If we don't auto-commit or manually commit the offset, then we are not moving the needle. |
| push() | Possiblity | Since you will have to provide topic, data & partition; we need to have some strategy to for partition selection; and need to maintain these values for Consumers. |



<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
@@ -0,0 +1,36 @@
#### 6- Azure Service Bus

Microsoft Azure Service Bus is a fully managed enterprise integration message broker. It support familiar concepts like Queues, Topics, Rules/Filters and much more.

##### High Level Architecture



<img src="AzureServiceBusQueue.png" alt="Legend" width="70%" height="70%" />



<img src="AzureServiceBusTopic.png" alt="Legend" width="70%" height="70%" />



##### Evaluation Table - Details

Azure Service Bus supports AMQP 1.0, and couple of languages, PHP support is again limited for the protocol

[AMQP Azure Service Bus Overview](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview)

[azure-uaqmp-c PHP Bindings for AMQP 1.0](https://github.com/norzechowicz/php-uamqp)

[Exported PHP Modules from C Native Library](https://github.com/norzechowicz/php-uamqp/tree/master/ext/src/php)

| Method | Evaluation | Implementation Readiness |
| ------------- | ----------- | ------------------------------------------------------------ |
| dequeue() | Available | receive() |
| acknowledge() | Available | accept() / release() |
| subscribe() | *Workaround | Long Polling might need to be implemented, unless we find a good library that supports AMQP 1.0 for PHP; Java has full support for required features. |
| reject() | Available | reject(errorCondition, errorDescription) |
| push() | Available | sendMessage(message, destination) |

<img src="legend_img.png" alt="Legend" width="70%" height="70%" />

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.