Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable RabbitMQ mirrored queues #1226

Merged
merged 21 commits into from
Sep 25, 2019
Merged

Enable RabbitMQ mirrored queues #1226

merged 21 commits into from
Sep 25, 2019

Conversation

vincenzodnp
Copy link
Contributor

@vincenzodnp vincenzodnp commented Sep 5, 2019 •
edited

Checklist

  • Affected Issues have been mentioned in the Closing issues section
  • Documentation has been written/updated.
  • Changelog entry has been written

Currently the RabbitMQ cluster is not set to use native HA mechanism called "Mirrored Queues".

It allows to create queues mirrored on 2 or more nodes, making the cluster fully High Availability.
In fact, at the moment, if the node (pod) where the queue is declared dies, queue's consumer are not able to use the it.

With mirrored queues, not only queue is still available for consumers but the mirror node is automatically elected new queue master (writable).

When the mirror node(s) is(are) online again, the queue is automatically synced all over the nodes defined in the policy.

Changelog Entry

Improvement - Set RabbitMQ policy to enable Mirrored Queue by default for all "lagoon-" queues (#1098 )

Closing issues

closes #1098

@vincenzodnp
Added definitions.json file to define and set RabbitMQ policies for HA
894975e 
(Mirrored Queues) queues.

Added logic to parse environment variables to set defult username,
password and patter for Mirrored Queues at container startup
@vincenzodnp
- Changed RabbitMQ default ha-policy to "exactly" instead of "all"
dcd1824 
- Set number of mirrors per queue to 2
- Modified broker lagoon template by adding the RABBITMQ_DEFAULT_HA_PATTER variable
Schnitzel
Schnitzel requested changes Sep 6, 2019
View reviewed changes
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
services/broker/.lagoon.app.yml Outdated Show resolved Hide resolved
images/rabbitmq-cluster/cluster-rabbit.sh Outdated Show resolved Hide resolved
images/rabbitmq/definitions.json Outdated Show resolved Hide resolved
@vincenzodnp
Update Dockerfile
a80cc57 
Use of COPY for consistence in Dockerfile
Changed RABBITMQ_DEFAULT_HA_PATTERN default value
@vincenzodnp
Added REPLACE_VHOST replace patter for vhosts variable
644f46b
@vincenzodnp
New ENV variables logic
6bd1652 
Added RABBITMQ_DEFAULT_VHOST variable
Added envplate binary to substitute variables
@vincenzodnp
RabbitMQ image consistent ENTRYPOINT
dc5d8aa 
Same ENTRYPOINT for rabbitmq and rabbitmq-cluster images
Defined ENV variables for Lagoon broker service
vincenzodnp
vincenzodnp commented Sep 6, 2019
View reviewed changes
Copy link
Contributor Author

@vincenzodnp vincenzodnp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed last commit to fix it.
Please review.
Thanks

dasrecht
dasrecht requested changes Sep 6, 2019
View reviewed changes
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
services/broker/Dockerfile Outdated Show resolved Hide resolved
Schnitzel
Schnitzel requested changes Sep 6, 2019
View reviewed changes
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
vincenzodnp
vincenzodnp commented Sep 6, 2019
View reviewed changes
Copy link
Contributor Author

@vincenzodnp vincenzodnp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready for review

Schnitzel
Schnitzel requested changes Sep 6, 2019
View reviewed changes
images/rabbitmq/definitions.json Show resolved Hide resolved
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
Schnitzel
Schnitzel requested changes Sep 7, 2019
View reviewed changes
images/rabbitmq/Dockerfile Outdated Show resolved Hide resolved
@vincenzodnp
Copy link
Contributor Author

Added documentation about image

Schnitzel
Schnitzel requested changes Sep 9, 2019
View reviewed changes
docs/using_lagoon/docker_images/rabbitmq.md Outdated Show resolved Hide resolved
docs/using_lagoon/docker_images/rabbitmq.md Outdated Show resolved Hide resolved
@Schnitzel
Copy link
Contributor

@vincenzodnp
the testing currently fails:

Step 1/13 : FROM ${IMAGE_REPO:-lagoon}/commons as commons
pull access denied for lagoon/commons, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Makefile:149: recipe for target 'build/rabbitmq' failed

make: *** [build/rabbitmq] Error 1

this is because it tries to load lagoon/commons, but the env variable IMAGE_REPO would overwrite this to something like jenkinslagoonpr122616.
We missed the ARG on top of the rabbitmq Dockerfile, see like https://github.com/amazeeio/lagoon/blob/master/images/nginx/Dockerfile#L1

Can you add that? thanks

@vincenzodnp
Copy link
Contributor Author

@Schnitzel added the ARG IMAGE_REPO.
Can you review?
Thanks

@Schnitzel Schnitzel merged commit 31842ad into master Sep 25, 2019
@Schnitzel Schnitzel added this to the v1.1.0 milestone Sep 27, 2019
@dasrecht dasrecht deleted the 1098-rabbitmq-mirrored-queues branch January 24, 2020 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dasrecht dasrecht Awaiting requested review from dasrecht

@Schnitzel Schnitzel Awaiting requested review from Schnitzel

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.1.0
Development

Successfully merging this pull request may close these issues.

Set RabbitMQ queues in HA mode (Mirrored queues)
3 participants
@vincenzodnp @Schnitzel @dasrecht