TECH-3227: Modernizing cloud-init.yml #22

fjoeaz · 2023-08-18T21:48:51Z

Description

This makes it so we can go from 3.8.0 -> 3.12.* with hot rolling upgrades.

Steps to Test

https://github.com/smartrent/control.smartrent.com/pull/11782

Links

JIRA Task: https://smartrent.atlassian.net/browse/TECH-3227

fjoeaz · 2023-08-18T21:58:07Z

cloud-init.yaml

+  - path: /root/conf/advanced.config
+    content: |
+      [
+        {rabbit, [
+          {forced_feature_flags_on_init, [
+            drop_unroutable_metric, empty_basic_get_metric, implicit_default_bindings, quorum_queue, virtual_host_metadata
+          ]}
+        ]}
+      ].


Because newer versions of RabbitMQ have feature flags that are automatically enabled and not backwards compatible nor can you disable them once they are on. We need to disable them on boot and then enable them once the cluster is upgraded to the new version.

NOTE: Unfortunately this can only be done using the old config file format.
AND: Yes is auto loaded and yes its advanced.config not *.conf like the new format ><

It might be nice to have this be a var but I haven't gotten there yet.

fjoeaz · 2023-08-18T21:59:37Z

cloud-init.yaml

+  - path: /root/conf/rabbitmq.conf
    content: |
-        [ { rabbit, [
-          { loopback_users, [ ] } ] }
-         ].
+      loopback_users = none


The new config format is much simpler, easier for humans to read and machines to generate. It is also relatively limited compared to the classic config format used prior to RabbitMQ 3.7.0.

NOTE: file name did change from rabbitmq.config -> rabbitmq.conf

REF: https://www.rabbitmq.com/configure.html#config-file-formats

fjoeaz · 2023-08-18T22:00:45Z

cloud-init.yaml

  - yum update -y
  - yum install -y docker jq
  - pip3 install boto3
-  - DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_API_KEY=$(aws ssm get-parameter --name ${dd_api_key} --with-decryption --region ${region} | jq -r '.Parameter.Value') DD_SITE="${dd_site}" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
+  - DD_AGENT_MAJOR_VERSION=7 DD_INSTALL_ONLY=true DD_API_KEY=$(aws ssm get-parameter --name ${dd_api_key} --with-decryption --region ${region} | jq -r '.Parameter.Value') DD_SITE="${dd_site}" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"


install_script.sh - Install script that uses DD_AGENT_MAJOR_VERSION=6 by default and also emits a deprecation warning when run

REF: https://github.com/DataDog/agent-linux-install-script#working-with-this-repository

fjoeaz · 2023-08-18T22:02:11Z

cloud-init.yaml

@@ -196,12 +213,14 @@ runcmd:
  - cp /root/datadog-agent/conf.d/rabbitmq.d/conf.yaml /etc/datadog-agent/conf.d/rabbitmq.d/conf.yaml && chown dd-agent /etc/datadog-agent/conf.d/rabbitmq.d/conf.yaml
  - cp /root/datadog-agent/secrets.py /etc/datadog-agent/secrets.py && chown dd-agent /etc/datadog-agent/secrets.py && chmod 0700 /etc/datadog-agent/secrets.py
  - systemctl start datadog-agent
-  - $(aws ecr get-login --no-include-email --region ${region} --registry-ids ${ecr_registry_id})
-  - docker run -d --name rabbitmq --hostname $HOSTNAME --log-driver=local --log-opt max-size=10m -p 4369:4369 -p 5672:5672 -p 15672:15672 -p 25672:25672 -e RABBITMQ_ERLANG_COOKIE=$(aws ssm get-parameter --name ${secret_cookie} --with-decryption --region ${region} | jq -r '.Parameter.Value') -v /root/data:/var/lib/rabbitmq -v /root/conf/:/etc/rabbitmq -v /root/bin:/tmp/bin ${rabbitmq_image}
+  - $(aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin ${ecr_registry_id}.dkr.ecr.${region}.amazonaws.com)


Security risk:

This command displays docker login commands to stdout with authentication credentials. Your credentials could be visible by other users on your system in a process list display or a command history. If you are not on a secure system, you should consider this risk and login interactively. For more information, see get-authorization-token.

REF: https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login.html

fjoeaz · 2023-08-18T22:07:41Z

cloud-init.yaml

-  - docker run -d --name rabbitmq --hostname $HOSTNAME --log-driver=local --log-opt max-size=10m -p 4369:4369 -p 5672:5672 -p 15672:15672 -p 25672:25672 -e RABBITMQ_ERLANG_COOKIE=$(aws ssm get-parameter --name ${secret_cookie} --with-decryption --region ${region} | jq -r '.Parameter.Value') -v /root/data:/var/lib/rabbitmq -v /root/conf/:/etc/rabbitmq -v /root/bin:/tmp/bin ${rabbitmq_image}
+  - $(aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin ${ecr_registry_id}.dkr.ecr.${region}.amazonaws.com)
+  - bash /root/erlang_cookie.sh
+  - docker run -d --name rabbitmq --hostname $HOSTNAME --log-driver=local --log-opt max-size=10m -p 4369:4369 -p 5672:5672 -p 15672:15672 -p 25672:25672 -v /root/data:/var/lib/rabbitmq -v /root/conf/:/etc/rabbitmq -v /root/bin:/tmp/bin ${rabbitmq_image}


RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.

There is not much mentioned about this other than its usage and the deprecated warning found in 3.8.10.

REF: https://www.rabbitmq.com/clustering.html#cookie-file-locations

fjoeaz · 2023-08-18T22:10:15Z

cloud-init.yaml

+  - path: /root/erlang_cookie.sh
+    content: |
+        #!/usr/bin/env bash
+        mkdir /root/data/
+        aws ssm get-parameter --name ${secret_cookie} --with-decryption --region ${region} | jq -r '.Parameter.Value' > /root/data/.erlang.cookie
+        chmod 400 /root/data/.erlang.cookie
+


Other parts of preexisting docker run will handle the rest of this through volume mounting. But first we need the correct file in the correct location with the correct perms:

mkdir /root/data/

/root/data:/var/lib/rabbitmq

chmod 400 /root/data/.erlang.cookie

this info would be great to include in a comments!

fjoeaz · 2023-08-18T22:14:12Z

cloud-init.yaml

  - path: /root/conf/enabled_plugins
    content: |
-        [prometheus_rabbitmq_exporter,rabbitmq_management].
-  - path: /root/conf/rabbitmq.config
+      [
+        rabbitmq_management
+      ].


RabbitMQ versions prior to 3.8 used a separate plugin, prometheus_rabbitmq_exporter, to expose metrics to Prometheus.

NOTE: We are and have been on 3.8.* for a long time no need for this removed plugin.

REF: https://www.rabbitmq.com/prometheus.html#3rd-party-plugin

@alexraskin FYI

fjoeaz · 2023-08-19T07:57:53Z

Don't hate the programmer hate the language :)

terraform-aws-rabbitmq/main.tf

Lines 1 to 29 in c7e6518

    
           locals { 
        
             feature_flag_template = ( 
        
               # if a minor version number we know is passed then populate and use the template 
        
               contains([8, 11, 12], var.upgrading_minor_version_number) 
        
               ? "{forced_feature_flags_on_init, [${local.ffs}]}" 
        
               : "" # otherwise leave it blank so it does not cause errors 
        
             ) 
        
             ffs = join(",", flatten([                        # finally flatten and join with commas 
        
               for ff_key in keys(local.ff_version)           # for each key 
        
               : ff_key <= var.upgrading_minor_version_number # if its <= pass minor version 
        
               ? local.ff_version[ff_key] : []                # return it otherwise return empty 
        
             ])) 
        
             # an object containing key pairs for the correct minor version feature flags 
        
             # https://www.rabbitmq.com/feature-flags.html#core-feature-flags 
        
             ff_version = { 
        
               8 = [ 
        
                 "drop_unroutable_metric", "empty_basic_get_metric", "implicit_default_bindings", 
        
                 "quorum_queue", "virtual_host_metadata", 
        
               ], 
        
               11 = ["user_limits", "maintenance_mode_status", ], 
        
               12 = [ 
        
                 "classic_mirrored_queue_version", "classic_queue_type_delivery_support", 
        
                 "direct_exchange_routing_v2", "feature_flags_v2", "listener_records_in_ets", 
        
                 "stream_queue", "stream_single_active_consumer", "tracking_records_in_ets", 
        
               ], 
        
             } 
        
           }

All parts tested from 3.8.* to 3.12.* all work as intended even if it looks a little sketchy. Unfortunately TF does NOT give us much options other that gnarly strings with nested if's.

fjoeaz · 2023-08-21T23:39:01Z

cloud-init.yaml

+  - |
+    TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
+    sed -i "s/replace_ip_address_here/$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/local-ipv4)/g" /root/datadog-agent/conf.d/rabbitmq.d/conf.yaml


v1 wasn't returning anything so needed v2: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

All better RabbitMQ QA Cluster Degraded: https://app.datadoghq.com/monitors/48134887

Dashboard: https://app.datadoghq.com/dashboard/kpa-i8c-nqn/rabbitmq-cmw-qa

rraub

There are some huge gains in this PR! thank you for all the attention to detail

rraub · 2023-08-22T19:21:11Z

cloud-init.yaml

+  - path: /root/erlang_cookie.sh
+    content: |
+        #!/usr/bin/env bash
+        mkdir /root/data/
+        aws ssm get-parameter --name ${secret_cookie} --with-decryption --region ${region} | jq -r '.Parameter.Value' > /root/data/.erlang.cookie
+        chmod 400 /root/data/.erlang.cookie
+


this info would be great to include in a comments!

rraub · 2023-08-22T19:26:17Z

main.tf

-    # https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-linux-2-ami-kernel-5-10/
-    name   = "name"
-    values = ["amzn2-ami-kernel-5.*"]
+  feature_flag_template = (


TECH-3227: Modernizing cloud-init.yml

ce85154

fjoeaz commented Aug 18, 2023

View reviewed changes

ops

4dfff69

fjoeaz commented Aug 18, 2023

View reviewed changes

fjoeaz self-assigned this Aug 18, 2023

fjoeaz added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Aug 18, 2023

fjoeaz commented Aug 18, 2023

View reviewed changes

fjoeaz added 4 commits August 18, 2023 15:50

clean up dd install script output

3b0b2a4

Um?

64fc587

lol

cc49249

notes because well you will see

c7e6518

fjoeaz requested review from rraub and a team August 19, 2023 08:21

fjoeaz marked this pull request as ready for review August 19, 2023 08:21

fix for imds

29c3989

fjoeaz commented Aug 21, 2023

View reviewed changes

rraub approved these changes Aug 22, 2023

View reviewed changes

comments

7cf2c24

fjoeaz merged commit 03c0b67 into smartrent-master Aug 22, 2023
4 checks passed

fjoeaz deleted the TECH-3227-Modernizing-cloud-init.yml branch August 22, 2023 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TECH-3227: Modernizing cloud-init.yml #22

TECH-3227: Modernizing cloud-init.yml #22

fjoeaz commented Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023

rraub Aug 22, 2023

fjoeaz Aug 18, 2023 •

edited

Loading

rraub Aug 18, 2023

fjoeaz commented Aug 19, 2023

fjoeaz Aug 21, 2023

rraub left a comment

rraub Aug 22, 2023

rraub Aug 22, 2023

TECH-3227: Modernizing cloud-init.yml #22

TECH-3227: Modernizing cloud-init.yml #22

Conversation

fjoeaz commented Aug 18, 2023 • edited Loading

Description

Steps to Test

Links

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023

Choose a reason for hiding this comment

rraub Aug 22, 2023

Choose a reason for hiding this comment

fjoeaz Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

rraub Aug 18, 2023

Choose a reason for hiding this comment

fjoeaz commented Aug 19, 2023

fjoeaz Aug 21, 2023

Choose a reason for hiding this comment

rraub left a comment

Choose a reason for hiding this comment

rraub Aug 22, 2023

Choose a reason for hiding this comment

rraub Aug 22, 2023

Choose a reason for hiding this comment

fjoeaz commented Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading

fjoeaz Aug 18, 2023 •

edited

Loading