cdk: start the CloudWatch agent on every instance, not just ollama

[jucor]

What

Moves the CloudWatch agent's config download + systemctl start from the ollama user-data block into the shared usrdata(), so every instance starts it — not just the GPU box.

The agent is already installed everywhere; only ollama ever started it. Result today: no mem_used_percent on the math, delphi or web tiers. Memory is what we need to right-size them — CPU sits at 0.5–1.3% and tells us nothing.

Added commands are guarded with || true (the function runs under set -e), so a failing agent can never abort an instance boot.

---

Rollout

Deploying this on its own gives you zero memory data. User-data only applies to instances at launch, and none of the ASGs has a rolling-update policy, so nothing gets replaced on deploy. Instances pick it up only when they're replaced — and this fleet is idle, so that won't happen on its own.

Two steps. Do both: step 1 gets data now, step 2 keeps it working after instances cycle.

Run these in AWS CloudShell — console → the >_ icon in the top bar. It's a Linux shell in the browser, already signed in as you, so this works the same from Windows, Mac or Linux and needs nothing installed. These are bash scripts; they will not run in PowerShell or cmd. (WSL or Git Bash work too if you'd rather stay local, but then you need the AWS CLI configured yourself.)

Both scripts find the ASGs themselves and skip ollama (it already publishes memory, and its boot does the most work — nothing to gain, most to break). Nothing to fill in.

Step 1 — turn it on now, on the instances already running

Doesn't need this PR merged. Nothing is replaced. Data arrives in ~1 minute. Safe to re-run.

#!/usr/bin/env bash
set -euo pipefail
REGION=${AWS_REGION:-us-east-1}
CWA=/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

# Every ASG except the GPU box.
ASGS=$(aws autoscaling describe-auto-scaling-groups --region "$REGION" \
  --query 'AutoScalingGroups[].AutoScalingGroupName' --output text \
  | tr '\t' '\n' | grep -vi ollama || true)
[ -n "$ASGS" ] || { echo "no ASGs found"; exit 1; }

# The agent config lives in S3. Read its URL out of the ollama launch
# template's user-data, where it is already baked in.
OLLAMA_LT=$(aws autoscaling describe-auto-scaling-groups --region "$REGION" \
  --query 'AutoScalingGroups[].[AutoScalingGroupName,LaunchTemplate.LaunchTemplateId]' \
  --output text | grep -i ollama | awk '{print $2}' | head -1)
UD_B64=$(aws ec2 describe-launch-template-versions --region "$REGION" \
  --launch-template-id "$OLLAMA_LT" --versions '$Latest' \
  --query 'LaunchTemplateVersions[0].LaunchTemplateData.UserData' --output text)
UD=$(printf '%s' "$UD_B64" | base64 -d 2>/dev/null || printf '%s' "$UD_B64" | base64 -D)
CFG_URL=$(printf '%s' "$UD" | grep -om1 's3://[^ "]*amazon-cloudwatch-agent.json')
[ -n "$CFG_URL" ] || { echo "could not find the agent config URL"; exit 1; }
echo "config: $CFG_URL"

for ASG in $ASGS; do
  ID=$(aws ssm send-command --region "$REGION" --output text \
    --query 'Command.CommandId' --cli-input-json "$(cat <<JSON
{
  "DocumentName": "AWS-RunShellScript",
  "Comment": "start cloudwatch agent",
  "Targets": [{"Key": "tag:aws:autoscaling:groupName", "Values": ["$ASG"]}],
  "Parameters": {"commands": [
    "aws s3 cp $CFG_URL /tmp/cwa.json",
    "sudo mkdir -p $(dirname $CWA)",
    "sudo mv /tmp/cwa.json $CWA",
    "sudo chmod 644 $CWA",
    "sudo chown root:root $CWA",
    "sudo systemctl enable amazon-cloudwatch-agent",
    "sudo systemctl start amazon-cloudwatch-agent"
  ]}
}
JSON
)")
  echo "$ASG -> $ID"
done

Any command that failed:

aws ssm list-command-invocations --details \
  --query 'CommandInvocations[?Status!=`Success`].[InstanceId,Status,StatusDetails]' --output table

Step 2 — after this PR is deployed, refresh the ASGs

This is what makes it stick. Step 1's fix is lost the moment an instance is replaced.

#!/usr/bin/env bash
set -euo pipefail
REGION=${AWS_REGION:-us-east-1}

ASGS=$(aws autoscaling describe-auto-scaling-groups --region "$REGION" \
  --query 'AutoScalingGroups[].AutoScalingGroupName' --output text \
  | tr '\t' '\n' | grep -vi ollama || true)
[ -n "$ASGS" ] || { echo "no ASGs found"; exit 1; }

for ASG in $ASGS; do
  ID=$(aws autoscaling start-instance-refresh --region "$REGION" \
    --auto-scaling-group-name "$ASG" \
    --preferences '{"MinHealthyPercentage":100,"MaxHealthyPercentage":200}' \
    --query 'InstanceRefreshId' --output text)
  echo "$ASG -> $ID"
done

MaxHealthyPercentage: 200 launches the replacement before terminating the old one, so capacity never dips. Every ASG has room (desired/max): math 1/5, delphi-small 2/7, delphi-large 1/3, web 2/10.

Watch it:

for ASG in $(aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].AutoScalingGroupName' --output text \
  | tr '\t' '\n' | grep -vi ollama); do
  echo "$ASG: $(aws autoscaling describe-instance-refreshes \
    --auto-scaling-group-name "$ASG" --max-records 1 \
    --query 'InstanceRefreshes[0].[Status,PercentageComplete]' --output text)"
done

Check it worked

aws cloudwatch list-metrics --namespace CWAgent --metric-name mem_used_percent \
  --query 'Metrics[].Dimensions[?Name==`AutoScalingGroupName`].Value' --output text

You should see every ASG listed, not just ollama. Metrics land at 60s intervals.

🤖 Generated with Claude Code