Tool for diagnosing what spinnaker service are up/down

[frankcarey]

I was having some issues with services dying and I came up with this to get a general cli dashboard of what services were up and what was down. Ended up being a lot faster than doing it one by one.. Thoughts?

$ bash spinnaker-status.sh
| SERVICE              |          - |       PORT |          - |
| spinnaker (deck)     |          - |       9000 |         OK |
| redis                |          - |       6379 |         OK |
| cassandra            |          - |       7199 |         OK |
| thrift (cassandra)   |          - |       9160 |         OK |
| clouddriver          |         OK |       7002 |         OK |
| echo                 |         OK |       8089 |         OK |
| front50              |         OK |       8080 |         OK |
| gate                 |         OK |       8084 |         OK |
| igor                 |         OK |       8088 |         OK |
| orca                 |         OK |       8083 |         OK |
| rosco                |       FAIL |       8087 |       FAIL |
| rush                 |         OK |       8085 |         OK |

check_port() {
  out="$1"
  if [ "$(netstat -el --numeric-ports | grep $1)" ]; then
    out="$out OK"
  else
    out="$out FAIL"
  fi
  printf "$out"
}

check_upstart() {
  out="$1"
  if [ "$(status $1 | grep running)" ]; then
    out="$out OK"
  else
    out="$out FAIL"
  fi
  printf "$out"
}

row() {
  printf "| %-20s | %10s | %10s | %10s |\n" "$1" $2 $3 $4
}

row "SERVICE" "-" "PORT" "-"
row 'spinnaker (deck)' "-"              `check_port 9000`
row "redis" "-"                         `check_port 6379`
row "cassandra" "-"                     `check_port 7199`
row "thrift (cassandra)" "-"            `check_port 9160`
row `check_upstart clouddriver`         `check_port 7002`
row `check_upstart "echo"`              `check_port 8089`
row `check_upstart front50`             `check_port 8080`
row `check_upstart gate`                `check_port 8084`
row `check_upstart igor`                `check_port 8088`
row `check_upstart orca`                `check_port 8083`
row `check_upstart rosco`               `check_port 8087`
row `check_upstart rush`                `check_port 8085`

[cfieber]

pretty cool!

for all the java services in spinnaker there is a /health endpoint - using cURL to hit that would actually give a better signal of app health (should return HTTP 200 and a JSON response {"status": "UP"} (possibly with some extra info depending on the service, so I'd pipe it through jq to test for status)

[cfieber]

here's something from a script we have in one of our deploys

UT=`cat /proc/uptime | cut -f1 -d.`
if [[ $UT -le 600 ]]
then
  echo "skipping healthcheck, system uptime currently only $UT seconds"
  exit 0
fi

function checkUrl {
    echo "checking $2"
    res=`curl -sL -w "%{http_code}" $1 -o /dev/null`
    if [ $res -ne 200 ]
    then
     # whatever you want to do here on DOWN
    fi
}

checkUrl "http://localhost:7002/health" clouddriver
checkUrl "http://localhost:8083/health" orca
# ... other services here

[ajordens]

Closing due to lack of activity.

[durgeshworks]

@frankcarey Hi very use full thank you.