Skip to content
This repository was archived by the owner on Nov 19, 2024. It is now read-only.

Add manage message queues instructions #5569

Closed
wants to merge 6 commits into from
Closed

Add manage message queues instructions #5569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8dc0259
insert Manage message queues instructions to enable bulk
brunoroeder Oct 4, 2019
4b3c964
insert Manage message queues instructions to enable bulk
brunoroeder Oct 4, 2019
912e028
fix typo
brunoroeder Oct 4, 2019
0446645
fix typo
brunoroeder Oct 4, 2019
45bf9a3
warnings and tips hot to configure message queues processingguides/v2…
brunoroeder Oct 4, 2019
342b6bc
formatting
dobooth Oct 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion guides/v2.2/rest/bulk-endpoints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ functional_areas:
Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.

{:.bs-callout .bs-callout-tip}
Copy link
Contributor

@meker12 meker12 Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{:.bs-callout .bs-callout-tip}
{:.bs-callout .bs-callout-info}

Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing.
Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing, after configure the message queues. For more details see: [Manage message queues]({{ page.baseurl }}config-guide/mq/manage-message-queues.html).
Copy link
Contributor

@meker12 meker12 Oct 22, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing, after configure the message queues. For more details see: [Manage message queues]({{ page.baseurl }}config-guide/mq/manage-message-queues.html).
Use the [`bin/magento queue:consumers:start async.operations.all`](https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-queue.html) command to start the consumer that handles asynchronous and bulk API messages. You must also ensure that you have configured RabbitMQ, which is the message broker that the Bulk API uses to process messages. See [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).

Copy link
Contributor

@nuzil nuzil Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on this point

Copy link
Contributor

@meker12 meker12 Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I edited the section above to include the information about RabbitMQ. The intent is to avoid having too many separate notes in the topic.


### Routes

Expand Down
6 changes: 6 additions & 0 deletions guides/v2.3/rest/bulk-endpoints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ functional_areas:

Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.

{: .bs-callout .bs-callout-warning}
By default, Magento uses RabbitMQ (AMQP) for processing message queues. Before using Bulk operations, you should see [RabbitMQ]({{ page.baseurl }}/install-gde/prereq/install-rabbitmq.html).
Copy link
Contributor

@meker12 meker12 Oct 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest incorporating this information in the first note instead of having a second note immediately following the first note.


{:.bs-callout .bs-callout-info}
Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).
Copy link
Contributor

@hostep hostep Oct 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong, but I'm almost 100% convinced that cron should always be used, even if you use RabbitMQ, otherwise there won't be any consumers running and messages will only end up in the queue but will not get processed.
There seems to be a lot of misinformation out there around this fact.

The exception is if you use something like supervisor to manage consumers, but that has nothing to do with RabbitMQ vs MysqlMQ...

Copy link
Contributor Author

@brunoroeder brunoroeder Oct 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @hostep base on @rogyar previous explanation and first paragraph of
Manage message queues
 - "If you don’t want to implement the RabbitMQ solution, you can manage message queues with cron jobs (or an external process manager) and the command line to ensure that consumers are retrieving messages." - Is not necessary have the job 'consumers_runner' enabled if you use RabbitMQ to manage queues.

Copy link
Contributor

@hostep hostep Oct 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that's what I mean by

There seems to be a lot of misinformation out there around this fact.

😉

Anyway, just to make sure I know what I'm talking about, I just tested this again to be very sure, here are some graphs from the RabbitMQ management interface:
Screenshot 2019-10-07 at 22 11 30

Between 22:06 and 22:10, I send in 3 REST calls to the bulk endpoints. No cronjobs or message queue consumers were running at that time. Notice that messages are added to the queue but stay in there and aren't getting processed.

At around 22:11, I executed a bin/magento cron:run which spawned some message queue consumer processes in the background. Notice that suddenly all messages are getting processed and the queue is empty again.

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

Is this enough information which you guys can use to fix all incorrect information on devdocs around this? Thanks! 🙂

Copy link
Contributor Author

@brunoroeder brunoroeder Oct 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hostep. I'll investigate it. Probably the misinformation is caused by de description of the Manage message queues that explains:

If you don’t want to implement the RabbitMQ solution, you can manage message queues with cron jobs ...

But at RabbitMQ explains that Crons are necessary to run consumers:

After you have connected Magento and RabbitMQ, you must start the message queue consumers. See Configure message queues for details.

Probably the mistake is in the Manage message queues description that cause this confusion :(

Copy link
Contributor

@BaDos BaDos Oct 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi there!

Consumers can use RabbitMQ or the DB for messages. It depends on the implementation of the consumer.
But to run consumers you need to run the command bin/magento queue:consumers:start ...
This command can be run by the cron job or some worker (for example, you can use workers on Magento Cloud Pro instances)

Copy link
Contributor

@meker12 meker12 Oct 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brunoroeder I provided updated comments in the PR. If you can make the suggested changes, then we can merge this PR. Short story: Make suggested edits to note referring to start:consumers command, and remove note about RabbitMQ.

Let me know if you have any questions. Thx.

Copy link
Contributor

@nuzil nuzil Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @hostep , @meker12

Correct me if I'm wrong, but I'm almost 100% convinced that cron should always be used

Its not now. CRON really still CAN be used. But you can also disable your consumer to run by using cron. And them align your server to run consumer as a service, without basically letting Magento know that its running.

So Cronjob is really optional on this step, same us direct running on consumer process. This decision is up to developers

Copy link
Contributor

@hostep hostep Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nuzil, thanks for the feedback!

I agree with your comment, sorry that was a bad wording from me.
But I don't agree with the current proposal as it seems to suggest that you need to run RabbitMQ or need to run cronjobs for the messages to get processed.

This is not the case. You need a place to store the messages (RabbitMQ or MysqlMQ) and consumer processes to do something with those messages. You need both, one of them by itself is not going to work.

For bulk api, you always need RabbitMQ (at the moment of writing) and using cronjobs as an alternative makes no sense. The cronjob which is being referred to here, is the consumer_runner one which spawns these consumer processes.

You are correct that cronjobs aren't necessary to spawn consumer processes, you can manually run them or have some other system manage them. But you do need consumer processes running to have the messages being processed.
So they are optional, but are not an alternative as what is being suggested.

I already tried to explain it a bit better previously a bit higher up, let me quote myself:

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

I'm not seeing ghosts right? I did test this many times now already ... 🙂

Copy link
Contributor

@nuzil nuzil Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I guess I skip this part of your message :-)
No you are totally correct described situation 👍 , with this amount of discussions can be wrongly to define what is the last point

Copy link
Contributor

@meker12 meker12 Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nuzil, thanks for the feedback!

I agree with your comment, sorry that was a bad wording from me.
But I don't agree with the current proposal as it seems to suggest that you need to run RabbitMQ or need to run cronjobs for the messages to get processed.

This is not the case. You need a place to store the messages (RabbitMQ or MysqlMQ) and consumer processes to do something with those messages. You need both, one of them by itself is not going to work.

For bulk api, you always need RabbitMQ (at the moment of writing) and using cronjobs as an alternative makes no sense. The cronjob which is being referred to here, is the consumer_runner one which spawns these consumer processes.

You are correct that cronjobs aren't necessary to spawn consumer processes, you can manually run them or have some other system manage them. But you do need consumer processes running to have the messages being processed.
So they are optional, but are not an alternative as what is being suggested.

I already tried to explain it a bit better previously a bit higher up, let me quote myself:

So using RabbitMQ by itself is not enough, you need consumers running within Magento so they can process the pending messages in the queue. Those consumers either get spawned using Magento's cronjob, you can start them manually using bin/magento queue:consumers:start ..., or you can use something like supervisor to manage them.

I'm not seeing ghosts right? I did test this many times now already ... 🙂

Copy link
Contributor

@meker12 meker12 Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Alternatively you can use CRON for processing message queues without RabbitMQ installed, see [Manage message queues]({{ page.baseurl }}/config-guide/mq/manage-message-queues.html).

Copy link
Contributor

@nuzil nuzil Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I would leave it

Copy link
Contributor

@meker12 meker12 Oct 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hostep and @nuzil Looking at an issue opened in Magento code repo: magento/magento2#25142 (comment)

In that issue, it looks like customer gets an error if they don't use RabbitMQ with Bulk API. Adding the note about using alternatives for processing message seems to add confusion. Even if it is possible to use something besides RabbitMQ to process messages with the Bulk API, isn't the advantage of the Bulk API is that it is pre-configured with the functionality that you need?

My suggestion is to delete the "Alternatively, you can use ... " note from this topic for now. And to provide information about the custom configuration that does not use RabbitMQ as a separate topic, perhaps a KB article or blog. That is assuming that the best practice and easiest way to process message queues using the Bulk API is to use the default configuration with RabbitMQ as the message broker.

Additionally, after we close this PR, it might be helpful to update the Bulk API tutorial to include RabbitMQ configuration in the Before you begin section.


{:.bs-callout .bs-callout-tip}
Use the `bin/magento queue:consumers:start async.operations.all` command to enable bulk endpoint processing.

Expand Down