S01E07: How to monitor RabbitMQ? #18
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
k3sup will work with any Linux host, simply specify a K3SUP_IP variable when running the make target @displague there are a few areas that the Equinix Metal CLI needs obvious improvements. Most obvious ones are getting the instance IP, ID & deleting an instance not picking up the PROJECT_ID env var. Let me know if you want to continue this conversation. @alexellis why is k3sup installing traefik by default? It felt awkward to have to disable it so that I would end up with a more vanilla K3S install. Everything else worked as advertised, thanks for a smooth experience. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
To force rebuild, run: make -B kubeconfig Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
There seems to be a permissions issue in k3s:
Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"default\""
I found this which seems to be a simpler way of getting Prometheus &
Grafana onto k3s: https://github.com/cablespaghetti/k3s-monitoring
Going to try that next as the current approach feels like a rabbit hole
based on what we care about. Committing it to capture what we have so
far, even though it may end up discarded.
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Most likely an intermediary step... Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
https://grafana.com/orgs/rabbitmq/dashboards Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Otherwise they will use the system hostname which is the first part of the FQDN. More info: https://github.com/rabbitmq/rabbitmq-website/blob/stream-queue/site/stream.md#advertised-host-port re https://github.com/rabbitmq/rabbitmq-server/issues/2486 Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
3-4 CPUs & 1-2 CPUs are enough to push just over 1mil msg/s with this setup. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
That results in 1.4mil msg/s Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
+ share a single server + do not create server if it exists Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
This makes the benefits of minimal or prometheus metrics more obvious. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
If rates_mode is anything other than none, collect_statistics gets set to fine: https://github.com/rabbitmq/rabbitmq-management-agent/blob/77aac8f4985559b2a660610938ee0567588e0422/src/rabbit_mgmt_db_handler.erl#L57-L72 Because we don't use detailed rates_mode, so the setting is irrelevant. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Otherwise RabbitMQ will fail to boot due to missing sample_retention_policies. By the way, this property is not used because we don't have detailed rates_mode set, but there must be a map merge happening in Cuttlefish that results in an incomplete config which RabbitMQ fails to handle and crashes. cc @lukebakken @michaelklishin Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Memory was the problem, CPUs are sufficient. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Otherwise descriptions will be misaligned for tagets that have longer names. Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
`kubectl example rmq` would have been a sweet idea, but the currently available version of the plugin doesn't support generating examples from CRD schemas. seredot/kubectl-example#2 Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
|
Scheduled to go live in a number of hours, after it gets converted to HD and maybe even 4K. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TGIR S01E07: How to monitor RabbitMQ?
You have a few RabbitMQ deployments running (on Kubernetes). How do you monitor them?
You have heard of the great Grafana dashboards that team RabbitMQ maintains, maybe from this RabbitMQ Summit 2019 talk or from the official Monitoring with Prometheus & Grafana guide. But how do you actually set them up?
For speed and convenience, we spin up a K3S instance on a Linux host and do the following:
You may follow along on any Linux host, including a VM running on your macOS or Windows host.
We had some credits with Equinix Metal that we wanted to put to good use.
Closes #17