[MessageQueue]: add support for first class queue configuration. #30111

Nolan-Arnold · 2020-09-18T20:45:52Z

Co-authored-by: Damien Retzinger damienwebdev@gmail.com

Description (*)

The Message Broker, RabbitMQ, supports arguments that are defined at queue creation time that dictate certain behaviors of the resulting queue. Currently (as of v2.4.0), Magento infers what queues will be created from properties of the exchange.binding defined in any given module's queue_topology.xml.

Unfortunately, the current implementation of queue_topology.xml does not expose a queue object to specify arguments at queue creation time, preventing utilization of the configurable behavior of RabbitMQ queues.

Related Pull Requests

https://github.com/magento/architecture/pull/432/files

Manual testing scenarios (*)

https://github.com/magento/architecture/blob/master/design-documents/message-queue/first-class-queue-configuration.md#sample-configurations

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

CC: @damienwebdev @nrkapoor @paliarush @nuzil @maghamed

m2-assistant · 2020-09-18T20:45:55Z

Hi @Nolan-Arnold. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

m2-assistant · 2020-09-18T20:46:52Z

Hi @Nolan-Arnold, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

m2-assistant · 2020-09-18T20:46:56Z

Hi @Nolan-Arnold. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

damienwebdev · 2020-09-19T02:56:44Z

lib/internal/Magento/Framework/MessageQueue/Topology/Config/Xml/Converter.php

+            $autoDelete = $this->getAttributeValue($exchange, 'autoDelete', false);
+            $result[$name . '--' . $connection] = [
+                'name' => $name,
+                'type' => $this->getAttributeValue($exchange, 'type'),


Given that we're inside a loop over the queues, this property can be inferred without the extra attribute on the config node, correct?

damienwebdev · 2020-09-19T02:58:34Z

app/code/Magento/Ui/etc/di.xml

@@ -256,6 +256,7 @@
                <item name="array" xsi:type="object">arrayArgumentInterpreterProxy</item>
                <item name="boolean" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Boolean</item>
                <item name="number" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Number</item>
+                <item name="int" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Integer</item>


I don't precisely know the full scope of how why this interpreter injection seems to occur in various hierarchies, but potentially this isn't in the scope of this PR?

cc: @paliarush

damienwebdev · 2020-09-19T02:59:05Z

app/etc/di.xml

@@ -479,6 +479,7 @@
                <item name="array" xsi:type="object">layoutArrayArgumentReaderInterpreterProxy</item>
                <item name="boolean" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Boolean</item>
                <item name="number" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Number</item>
+                <item name="int" xsi:type="object">Magento\Framework\Data\Argument\Interpreter\Integer</item>


I don't precisely know the full scope of how why this interpreter injection seems to occur in various hierarchies, but potentially this isn't in the scope of this PR?

cc: @paliarush

damienwebdev · 2020-09-19T03:01:05Z

lib/internal/Magento/Framework/MessageQueue/Topology/Config/Data.php

@@ -23,4 +23,20 @@ public function __construct(
    ) {
        parent::__construct($reader, $cache, $cacheId, $serializer);
    }
+
+    public function getQueues(): array {


Since ->get() loses backward compatibility (it returns both exchanges and queues when it used to only return exchanges), how should we consider the compatibility of this API?

cc: @paliarush

damienwebdev · 2020-09-19T03:02:56Z

lib/internal/Magento/Framework/MessageQueue/Topology/Config/Validator/FieldsTypes.php

+     * @throws \LogicException
+     */
+    private function validateQueueTypes($queueName, $queueConfig)
+    {


What validation should be provided here?

damienwebdev · 2020-09-19T03:03:34Z

lib/internal/Magento/Framework/MessageQueue/Topology/Config/Validator/Format.php

                foreach ($diff as $field) {
-                    $errors[] = sprintf('Missing [%s] field for binding %s in exchange config.', $field, $name);
+                    $errors[] = sprintf('Really Missing [%s] field for queue %s.', $field, $name);


@Nolan-Arnold is the emphatic Really necessary?

damienwebdev · 2020-09-19T03:04:38Z

lib/internal/Magento/Framework/MessageQueue/Topology/Config/Xml/Converter.php

@@ -99,6 +99,41 @@ public function convert($source)
                'arguments' => $exchangeArguments,
            ];
        }
+
+        foreach ($source->getElementsByTagName('queue') as $exchange) {


$exchange should be $queue

nuzil · 2021-03-23T15:50:25Z

Hey @Nolan-Arnold are you still working on this? Can you check comments and also backmerge 2.4 into your branch?

tdgroot · 2023-05-03T07:18:54Z

lib/internal/Magento/Framework/Data/Argument/Interpreter/Integer.php

+{
+    /**
+     * {@inheritdoc}
+     * @return int|float


Since you're returning an int, the return value should just be int right?

Oh I thought this was new stuff, but clearly confused 😄

ghost added this to Pending Review in Pull Requests Dashboard Sep 18, 2020

ghost added the Progress: pending review label Sep 18, 2020

magento-engcom-team added Component: App Component: Data Component: MessageQueue Component: Ui Release Line: 2.4 labels Sep 18, 2020

Nolan-Arnold closed this Sep 18, 2020

ghost removed this from Pending Review in Pull Requests Dashboard Sep 18, 2020

Nolan-Arnold reopened this Sep 18, 2020

ghost added this to Pending Review in Pull Requests Dashboard Sep 18, 2020

damienwebdev suggested changes Sep 19, 2020

View reviewed changes

sidolov added Priority: P3 May be fixed according to the position in the backlog. Severity: S3 Affects non-critical data or functionality and does not force users to employ a workaround. labels Sep 21, 2020

[MessageQueue]: add support for first class queue configuration

7762e5a

Nolan-Arnold force-pushed the 2.4-develop branch from 7b096aa to 7762e5a Compare October 30, 2020 16:15

nuzil self-requested a review March 23, 2021 15:50

tdgroot reviewed May 3, 2023

View reviewed changes

m2-community-project bot assigned tdgroot May 3, 2023

m2-community-project bot moved this from Pending Review to Review in Progress in Pull Requests Dashboard May 3, 2023

m2-community-project bot added Progress: review and removed Progress: pending review labels May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MessageQueue]: add support for first class queue configuration. #30111

[MessageQueue]: add support for first class queue configuration. #30111

Nolan-Arnold commented Sep 18, 2020

m2-assistant bot commented Sep 18, 2020

m2-assistant bot commented Sep 18, 2020

m2-assistant bot commented Sep 18, 2020

damienwebdev Sep 19, 2020

damienwebdev Sep 19, 2020 •

edited

damienwebdev Sep 19, 2020 •

edited

damienwebdev Sep 19, 2020

damienwebdev Sep 19, 2020

damienwebdev Sep 19, 2020

damienwebdev Sep 19, 2020

damienwebdev Sep 19, 2020

nuzil commented Mar 23, 2021

tdgroot May 3, 2023

tdgroot May 3, 2023

[MessageQueue]: add support for first class queue configuration. #30111

Are you sure you want to change the base?

[MessageQueue]: add support for first class queue configuration. #30111

Conversation

Nolan-Arnold commented Sep 18, 2020

Description (*)

Related Pull Requests

Manual testing scenarios (*)

Contribution checklist (*)

m2-assistant bot commented Sep 18, 2020

m2-assistant bot commented Sep 18, 2020

m2-assistant bot commented Sep 18, 2020

Choose a reason for hiding this comment

damienwebdev Sep 19, 2020 • edited

Choose a reason for hiding this comment

damienwebdev Sep 19, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuzil commented Mar 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damienwebdev Sep 19, 2020 •

edited

damienwebdev Sep 19, 2020 •

edited