Degradation of warm up time for algod docker image since 3.16.X

[daniel-makerx]

Subject of the issue

The warm up time (that is the time it takes for endpoints to start responding successfully after the container is first started) for the official algod docker images when using START_KMD=1 has significantly increased between versions 3.15.1 and 3.16.x (tested on 3.16.0-beta and 3.16.2-stable)

Observation: There is a 30s delay between the following two algod log messages, which may imply kmd is taking 30s to start

2023-06-26 14:13:15 Telemetry logging disabled: Name = , Guid = a10be9a2-6e22-421a-8f03-f2686fec4a29
2023-06-26 14:13:45 Successfully started kmd

Your environment

• Software version: First replicated in 3.16.0-beta, still present in 3.16.2-stable
• Operating System: Replicated on multiple OS's, exact times differ but the increase in warm up seems to be consistently around 30s

Steps to reproduce

The following script can be used to see the difference in start up times, docker and curl are assumed to be available.

#!/bin/bash
export TOKEN=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

wait_for_status() {
   echo -n "Waiting for /v2/status "
   for i in {1..60}; do
      curl --fail http://localhost:8080/v2/status --header "X-Algo-API-Token: $TOKEN" &>/dev/null
      if [ $? -eq 0 ]; then
         echo 
         echo "Took ${i}s"
         return
      fi
      echo -n .
      sleep 1
   done
}

get_warmup() {
   version=$1
   echo "Starting algod:$version"
   docker run \
      --name algod_warm \
      -d \
      --env START_KMD=1 \
      --env TOKEN=$TOKEN \
      -p 8080:8080 \
      algorand/algod:$version >/dev/null
   
   wait_for_status
   echo "Cleaning up ..."   
   docker stop algod_warm >/dev/null
   docker rm algod_warm >/dev/null
   echo
}

get_warmup 3.15.1-stable
get_warmup 3.16.2-stable

Expected behaviour

Warm up time of 3.16.x is comparable to warm up time of 3.15.1

Actual behaviour

Warm up time is significantly longer in 3.16.x

Sample output of above script from a MacBoox Pro M1 running Ventura 13.2

Starting algod:3.15.1-stable
Waiting for /v2/status ....
Took 5s
Cleaning up ...

Starting algod:3.16.2-stable
Waiting for /v2/status ..................................
Took 35s
Cleaning up ...

[winder]

Resolved by #5514