Add manage message queues instructions #5569

brunoroeder · 2019-10-04T15:00:06Z

Purpose of this pull request

Inserted infos about Manage message queues. This is a suggestion base on #5468 'issue'.

Affected DevDocs pages

https://devdocs.magento.com/guides/v2.3/rest/bulk-endpoints.html

devops-devdocs · 2019-10-04T15:01:03Z

An admin must run tests on this PR before it can be merged.

guides/v2.2/rest/bulk-endpoints.md

….2/rest/bulk-endpoints.md

hostep · 2019-10-06T17:49:24Z

guides/v2.3/rest/bulk-endpoints.md

+By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).
+
+{:.bs-callout .bs-callout-info}
+Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).


Correct me if I'm wrong, but I'm almost 100% convinced that cron should always be used, even if you use RabbitMQ, otherwise there won't be any consumers running and messages will only end up in the queue but will not get processed.
There seems to be a lot of misinformation out there around this fact.

The exception is if you use something like supervisor to manage consumers, but that has nothing to do with RabbitMQ vs MysqlMQ...

Hi @hostep base on @rogyar previous explanation and first paragraph of
Manage message queues
- "If you don’t want to implement the RabbitMQ solution, you can manage message queues with cron jobs (or an external process manager) and the command line to ensure that consumers are retrieving messages." - Is not necessary have the job 'consumers_runner' enabled if you use RabbitMQ to manage queues.

Well, that's what I mean by

There seems to be a lot of misinformation out there around this fact.

😉

Anyway, just to make sure I know what I'm talking about, I just tested this again to be very sure, here are some graphs from the RabbitMQ management interface:

Between 22:06 and 22:10, I send in 3 REST calls to the bulk endpoints. No cronjobs or message queue consumers were running at that time. Notice that messages are added to the queue but stay in there and aren't getting processed.

At around 22:11, I executed a bin/magento cron:run which spawned some message queue consumer processes in the background. Notice that suddenly all messages are getting processed and the queue is empty again.

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

Is this enough information which you guys can use to fix all incorrect information on devdocs around this? Thanks! 🙂

Thanks @hostep. I'll investigate it. Probably the misinformation is caused by de description of the Manage message queues that explains:

If you don’t want to implement the RabbitMQ solution, you can manage message queues with cron jobs ...

But at RabbitMQ explains that Crons are necessary to run consumers:

After you have connected Magento and RabbitMQ, you must start the message queue consumers. See Configure message queues for details.

Probably the mistake is in the Manage message queues description that cause this confusion :(

Hi there!

Consumers can use RabbitMQ or the DB for messages. It depends on the implementation of the consumer.
But to run consumers you need to run the command bin/magento queue:consumers:start ...
This command can be run by the cron job or some worker (for example, you can use workers on Magento Cloud Pro instances)

@brunoroeder I provided updated comments in the PR. If you can make the suggested changes, then we can merge this PR. Short story: Make suggested edits to note referring to start:consumers command, and remove note about RabbitMQ.

Let me know if you have any questions. Thx.

Hi @hostep , @meker12

Correct me if I'm wrong, but I'm almost 100% convinced that cron should always be used

Its not now. CRON really still CAN be used. But you can also disable your consumer to run by using cron. And them align your server to run consumer as a service, without basically letting Magento know that its running.

So Cronjob is really optional on this step, same us direct running on consumer process. This decision is up to developers

Hi @nuzil, thanks for the feedback!

I agree with your comment, sorry that was a bad wording from me.
But I don't agree with the current proposal as it seems to suggest that you need to run RabbitMQ or need to run cronjobs for the messages to get processed.

This is not the case. You need a place to store the messages (RabbitMQ or MysqlMQ) and consumer processes to do something with those messages. You need both, one of them by itself is not going to work.

For bulk api, you always need RabbitMQ (at the moment of writing) and using cronjobs as an alternative makes no sense. The cronjob which is being referred to here, is the consumer_runner one which spawns these consumer processes.

You are correct that cronjobs aren't necessary to spawn consumer processes, you can manually run them or have some other system manage them. But you do need consumer processes running to have the messages being processed.
So they are optional, but are not an alternative as what is being suggested.

I already tried to explain it a bit better previously a bit higher up, let me quote myself:

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

I'm not seeing ghosts right? I did test this many times now already ... 🙂

Right, I guess I skip this part of your message :-)
No you are totally correct described situation 👍 , with this amount of discussions can be wrongly to define what is the last point

Hi @nuzil, thanks for the feedback!

I agree with your comment, sorry that was a bad wording from me.
But I don't agree with the current proposal as it seems to suggest that you need to run RabbitMQ or need to run cronjobs for the messages to get processed.

This is not the case. You need a place to store the messages (RabbitMQ or MysqlMQ) and consumer processes to do something with those messages. You need both, one of them by itself is not going to work.

For bulk api, you always need RabbitMQ (at the moment of writing) and using cronjobs as an alternative makes no sense. The cronjob which is being referred to here, is the consumer_runner one which spawns these consumer processes.

You are correct that cronjobs aren't necessary to spawn consumer processes, you can manually run them or have some other system manage them. But you do need consumer processes running to have the messages being processed.
So they are optional, but are not an alternative as what is being suggested.

I already tried to explain it a bit better previously a bit higher up, let me quote myself:

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

I'm not seeing ghosts right? I did test this many times now already ... 🙂

dobooth · 2019-10-16T20:59:14Z

Thanks for the work here. Is everyone in agreement with this now? Ready to merge?

keharper · 2019-10-16T21:08:26Z

@meker12 Are you working on this too?

guides/v2.2/rest/bulk-endpoints.md

guides/v2.3/rest/bulk-endpoints.md

meker12 · 2019-10-22T12:47:44Z

guides/v2.3/rest/bulk-endpoints.md

+By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).
+
+{:.bs-callout .bs-callout-info}
+Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).


Suggested change

Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).

So I would leave it

@hostep and @nuzil Looking at an issue opened in Magento code repo: magento/magento2#25142 (comment)

In that issue, it looks like customer gets an error if they don't use RabbitMQ with Bulk API. Adding the note about using alternatives for processing message seems to add confusion. Even if it is possible to use something besides RabbitMQ to process messages with the Bulk API, isn't the advantage of the Bulk API is that it is pre-configured with the functionality that you need?

My suggestion is to delete the "Alternatively, you can use ... " note from this topic for now. And to provide information about the custom configuration that does not use RabbitMQ as a separate topic, perhaps a KB article or blog. That is assuming that the best practice and easiest way to process message queues using the Bulk API is to use the default configuration with RabbitMQ as the message broker.

Additionally, after we close this PR, it might be helpful to update the Bulk API tutorial to include RabbitMQ configuration in the Before you begin section.

meker12 · 2019-10-22T12:50:51Z

guides/v2.2/rest/bulk-endpoints.md


 {:.bs-callout .bs-callout-tip}
-Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing.
+Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing, after configure the message queues. For more details see: [Manage message queues]({{ page.baseurl }}config-guide/mq/manage-message-queues.html).


Suggested change

Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing, after configure the message queues. For more details see: [Manage message queues]({{ page.baseurl }}config-guide/mq/manage-message-queues.html).

Use the [`bin/magento queue:consumers:start async.operations.all`](https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-queue.html) command to start the consumer that handles asynchronous and bulk API messages. You must also ensure that you have configured RabbitMQ, which is the message broker that the Bulk API uses to process messages. See [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).

I agree on this point

I edited the section above to include the information about RabbitMQ. The intent is to avoid having too many separate notes in the topic.

guides/v2.3/rest/bulk-endpoints.md

nuzil · 2019-10-24T14:12:58Z

guides/v2.3/rest/bulk-endpoints.md

+By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).
+
+{:.bs-callout .bs-callout-info}
+Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).


Hi @hostep , @meker12

Correct me if I'm wrong, but I'm almost 100% convinced that cron should always be used

Its not now. CRON really still CAN be used. But you can also disable your consumer to run by using cron. And them align your server to run consumer as a service, without basically letting Magento know that its running.

So Cronjob is really optional on this step, same us direct running on consumer process. This decision is up to developers

nuzil · 2019-10-24T14:13:10Z

guides/v2.3/rest/bulk-endpoints.md

+By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).
+
+{:.bs-callout .bs-callout-info}
+Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).


So I would leave it

meker12 · 2019-10-28T15:41:25Z

guides/v2.2/rest/bulk-endpoints.md


 Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.

 {:.bs-callout .bs-callout-tip}


Suggested change

{:.bs-callout .bs-callout-tip}

{:.bs-callout .bs-callout-info}

meker12 · 2019-10-28T15:41:43Z

guides/v2.3/rest/bulk-endpoints.md

 Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.

+{: .bs-callout .bs-callout-warning}
+By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).


Suggest incorporating this information in the first note instead of having a second note immediately following the first note.

meker12 · 2019-10-29T21:26:05Z

This PR getting complicated, and I haven't heard back from @brunoroeder. Created #5867 incorporating review feedback and suggested changes. Closing this one.

ghost · 2019-10-29T21:26:06Z

Hi @brunoroeder, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

brunoroeder added 2 commits October 4, 2019 11:55

insert Manage message queues instructions to enable bulk

8dc0259

insert Manage message queues instructions to enable bulk

4b3c964

magento-engcom-team added Partner: Trezo partners-contribution PR created by Magento partner labels Oct 4, 2019

brunoroeder added 2 commits October 4, 2019 12:05

fix typo

912e028

fix typo

0446645

rogyar self-assigned this Oct 4, 2019

m2-community-project bot added the Progress: review label Oct 4, 2019

rogyar suggested changes Oct 4, 2019

View reviewed changes

guides/v2.2/rest/bulk-endpoints.md Outdated Show resolved Hide resolved

rogyar added 2.2.x 2.3.x Magento 2.3 related changes labels Oct 4, 2019

warnings and tips hot to configure message queues processingguides/v2…

45bf9a3

….2/rest/bulk-endpoints.md

brunoroeder requested a review from rogyar October 4, 2019 17:17

rogyar approved these changes Oct 6, 2019

View reviewed changes

hostep reviewed Oct 6, 2019

View reviewed changes

meker12 self-requested a review October 9, 2019 19:53

formatting

342b6bc

dobooth requested review from BaDos and hostep October 16, 2019 20:58

m2-community-project bot assigned meker12 Oct 16, 2019

meker12 reviewed Oct 16, 2019

View reviewed changes

guides/v2.2/rest/bulk-endpoints.md Show resolved Hide resolved

meker12 self-requested a review October 16, 2019 22:00

meker12 suggested changes Oct 16, 2019

View reviewed changes

guides/v2.3/rest/bulk-endpoints.md Show resolved Hide resolved

guides/v2.3/rest/bulk-endpoints.md Show resolved Hide resolved

hostep mentioned this pull request Oct 19, 2019

Unknown connection name amqp magento/magento2#25142

Closed

meker12 suggested changes Oct 22, 2019

View reviewed changes

m2-community-project bot assigned nuzil Oct 24, 2019

nuzil reviewed Oct 24, 2019

View reviewed changes

meker12 reviewed Oct 28, 2019

View reviewed changes

meker12 closed this Oct 29, 2019


		Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.

		{:.bs-callout .bs-callout-tip}

	{:.bs-callout .bs-callout-tip}
	{:.bs-callout .bs-callout-info}

Add manage message queues instructions #5569

Add manage message queues instructions #5569

Uh oh!

Conversation

brunoroeder commented Oct 4, 2019

Purpose of this pull request

Affected DevDocs pages

Uh oh!

devops-devdocs commented Oct 4, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meker12 Oct 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dobooth commented Oct 16, 2019

Uh oh!

keharper commented Oct 16, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meker12 Oct 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meker12 Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meker12 Oct 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meker12 commented Oct 29, 2019

Uh oh!

ghost commented Oct 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

meker12 Oct 21, 2019 •

edited

Loading

meker12 Oct 28, 2019 •

edited

Loading

meker12 Oct 22, 2019 •

edited

Loading

meker12 Oct 28, 2019 •

edited

Loading