Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S01E01: How to upgrade from RabbitMQ 3.7 to 3.8? #3

Merged
merged 16 commits into from
Jan 31, 2020
Merged

S01E01: How to upgrade from RabbitMQ 3.7 to 3.8? #3

merged 16 commits into from
Jan 31, 2020

Conversation

gerhard
Copy link
Contributor

@gerhard gerhard commented Jan 6, 2020 •
edited
Loading

image

@gerhard
Start working on Episode 2, expected to ship 2020-01-31
26f948d 
How to upgrade RabbitMQ 3.7 to 3.8 in prod?

Proposed by @dlresende via #2

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard gerhard changed the title How to upgrade RabbitMQ 3.7 to 3.8 in prod? 2020-01-31 How to upgrade from RabbitMQ 3.7 to 3.8 in prod? 2020-01-31 Jan 6, 2020
@gerhard gerhard changed the title How to upgrade from RabbitMQ 3.7 to 3.8 in prod? 2020-01-31 How to upgrade from RabbitMQ 3.7 to 3.8 in prod? Jan 6, 2020
@gerhard gerhard changed the title How to upgrade from RabbitMQ 3.7 to 3.8 in prod? How to upgrade from RabbitMQ 3.7 to 3.8 in prod? - 2020-01-31 Jan 6, 2020
@gerhard gerhard changed the title How to upgrade from RabbitMQ 3.7 to 3.8 in prod? - 2020-01-31 How to upgrade from RabbitMQ 3.7 to 3.8? - 2020-01-31 Jan 6, 2020
@gerhard gerhard changed the title How to upgrade from RabbitMQ 3.7 to 3.8? - 2020-01-31 S01E01 How to upgrade from RabbitMQ 3.7 to 3.8? Jan 6, 2020
@gerhard gerhard changed the title S01E01 How to upgrade from RabbitMQ 3.7 to 3.8? S01E01: How to upgrade from RabbitMQ 3.7 to 3.8? Jan 17, 2020
@gerhard
Restructure episodes, add more content for S01E01
84a066d 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Simulate 3 nodes production cluster + message load
871f149 
[#170545354]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Declare queues & create message backlog separately from load
354e251 
We want to start with backlog & focus on upgrading the cluster while
consumers are making progress on draining the queues. Ideally, we would
like to see queues with very little message backlog by the time we
finish upgrading all nodes.

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Capture PerfTest java.util.concurrent.RejectedExecutionException
78ddb79 
To reproduce, start a broker in one session by running:

    make 3-nodes-production-rmq1-server

And run the following command in a different session:

    make 3-nodes-production-rmq1-backlog

PerfTest crashes with the following exception within 10 seconds:

    /usr/local/bin/docker run --rm --interactive --tty \
      --hostname tgir-s01e01-rmq1-backlog \
      --name tgir-s01e01-rmq1-backlog \
      --network tgir-s01e01 \
      pivotalrabbitmq/perf-test:2.10.0-ubuntu \
      --auto-delete false \
      --confirm 100 \
      --confirm-timeout 10 \
      --consumers 0 \
      --flag persistent \
      --pmessages 1000 \
      --producers 1 \
      --queue-args 'x-max-length=1000' \
      --queue-pattern 'tgir-s01e01-rmq1-q%d' \
      --queue-pattern-from 1 \
      --queue-pattern-to 2000 \
      --servers-startup-timeout 60 \
      --size 1000 \
      --type 'fanout' \
      --uri "amqp://guest:guest@tgir-s01e01-rmq1:5672/%2f"
    id: test-102849-029, starting producer #0
    id: test-102849-029, starting producer #0, channel #0
    id: test-102849-029, time: 7.178s, sent: 0.14 msg/s, confirmed: 0 msg/s, nacked: 0 msg/s, min/median/75th/95th/99th confirm latency: 0/0/0/0/0 ?s
    test stopped (Error in producer)
    id: test-102849-029, sending rate avg: 4.9 msg/s
    id: test-102849-029, receiving rate avg: 0 msg/s
    Exception in thread "AMQP Connection 172.26.0.2:5672" java.util.concurrent.RejectedExecutionException: Task com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable@394d1b4f rejected from java.util.concurrent.ThreadPoolExecutor@1314c64d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
            at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
            at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
            at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
            at java.base/java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:687)
            at com.rabbitmq.client.impl.ConsumerWorkService.addWork(ConsumerWorkService.java:81)
            at com.rabbitmq.client.impl.ConsumerDispatcher.execute(ConsumerDispatcher.java:214)
            at com.rabbitmq.client.impl.ConsumerDispatcher.handleShutdownSignal(ConsumerDispatcher.java:173)
            at com.rabbitmq.client.impl.ChannelN.broadcastShutdownSignal(ChannelN.java:283)
            at com.rabbitmq.client.impl.ChannelN.finishProcessShutdownSignal(ChannelN.java:301)
            at com.rabbitmq.client.impl.ChannelN.processShutdownSignal(ChannelN.java:317)
            at com.rabbitmq.client.impl.ChannelManager$1.run(ChannelManager.java:117)
            at com.rabbitmq.client.impl.ChannelManager.handleSignal(ChannelManager.java:121)
            at com.rabbitmq.client.impl.AMQConnection.finishShutdown(AMQConnection.java:982)
            at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:956)
            at com.rabbitmq.client.impl.AMQConnection.handleFailure(AMQConnection.java:759)
            at com.rabbitmq.client.impl.AMQConnection.access$400(AMQConnection.java:48)
            at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:654)
            at java.base/java.lang.Thread.run(Thread.java:834)

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Create 1mil messages backlog slowly & reliably
ac3d253 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Give RabbitMQ nodes more memory, increase publishing rate
7d7778b 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Limit the number of messages each publisher publishes
c141443 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Add more make targets to make it easy to upgrade from 3.7 to 3.8
790cd9f 
Disable publisher confirms, publish 5 msg/s/producer to backfill quicker

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Use the latest PerfTest connection recovery improvements
e0b9390 
rabbitmq/rabbitmq-perf-test#236

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Update any rabbitmq-server container to 3.8.x with one command
cdd9568 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Deploy all RabbitMQ nodes & clients as single-container GCP VMs
ba170d6 
They are quick to deploy & upgrade and only depend on the gcloud CLI -
no K8S, BOSH, Ansible, Terraform etc. Another nice feature is that they
can sustain production loads with no degradation, not even NAT penalty,
as observed when using Docker's bridged network, the default in K8S 🤔

Adding DNS would be nice, as would better memory & disk limits for
RabbitMQ nodes + clustering.

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Make progress on the final content for the 3.7 to 3.8 upgrade
73ca2a7 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Add message drain & move onto 3-node cluster
5870df8 
[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Finish TGIR S01E01 content
f2fd127 
Need to edit & upload the video, then add the outline before this can be
finished.

[#170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard
Finish TGIR S01E01
655b379 
Closes #2

[finishes #170544206]

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
@gerhard gerhard merged commit 4299d25 into master Jan 31, 2020
@gerhard gerhard deleted the upgrade-rabbitmq-37x-38x branch January 31, 2020 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gerhard