Best practice for running consumer?

[stevenmusumeche]

How should the Symfony consumer command be run to execute without stopping? Via a cron job that runs every so often?

[tiborb]

I suggest having a look at Supervisor: http://supervisord.org/, it can make sure your command runs, restarts after a crash and so on.

Here is an example of how it can be configured:

[program:FOO]
command = /vagrant/app/console --env=prod some_command
numprocs = 1
autostart = false
autorestart = true

There is a symfony bundle for it too: https://github.com/supervisorphp/supervisor

[stevenmusumeche]

🙇

[stevenmusumeche]

Have you had success running the same consumer in parallel?

[tiborb]

Yes, you can have multiple instances of the same consumer (numprocs=x). What are you trying to achieve?

[stevenmusumeche]

The consumer is doing cache invalidation work and then fetching a URL, so each job takes 1-2 seconds. The queue gets filled in big chunks, so I'm trying to parallelize the work. I didn't know if running multiple Symfony commands in parallel would cause issues.

By the way, thank you for this bundle, the documentation is great and the API is easy to work with.

[tiborb]

Thanks, parallel symfony commands are fine.

PS: Beware that long running symfony commands could cause a memory leak (symfony/monolog-bundle#118). I could not reproduce it ever since, but still, keep an eye open for that ;)

[Sander-Toonen]

Just for future reference: I have about 30 consumers running in parallel. This works fine. Together, they process a few million messages day for different purposes.

Try to avoid memory usage accumulation. Don't keep appending to arrays without ever clearing them. This means for instance that you shouldn't use FingersCrossedHandler in Monolog since this keeps a buffer of log messages. Fine for a single request, not for a daysworth of debug messages.

Even when you are careful, PHP might leak memory. What can you say, PHP... (That could indeed be a catchphrase) I have a cronjob installed that restarts the workers every night, but in theory the consumers could run about a month before they run out of memory.

[rburkovsky]

Maybe I am bad at using search engines, but I wasn't able to find a working configuration and instructions to make daemon of single consumer. So that's what I came up with:

$ apt-get install supervisor

Put it into root of my project with name supervisord.conf

[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 10MB
directory = %(here)s
pidfile = /tmp/supervisord.pid
[program:todo]
command = php /var/www/symfony/bin/console rabbitmq:consumer -w sync
autostart = true
autorestart = true

From the same place:

$ supervisord

[NikitaKharkov]

I found more one bundle for symfony: Supervisor for symfony with Yaml configuration. I think It will be useful for somebody

[dirkwinkhaus]

I would recommend using systemd services instead of supervisord. For me systemd is much more reliable than supervisord

[irzhywau]

@dirkwinkhaus could you tell more about your motivation?

I actually use supervisord in a container to handle a dozen of consumers. Unfortunately some of them lose connection after some time and rabbitmq server returns error like {inet_error,etimedout}

[chucky2305]

I am new to RabbitMq: Just to be clear: @tiborb:
"Yes, you can have multiple instances of the same consumer (numprocs=x). What are you trying to achieve?"

What part makes sure, that one message in the queue is beeing processed by only one consumer?