Add support for the init daemon

When executing docker-compose up, all services listed in your docker-compose file will start immediately. As a consequence, some containers may fail as they depend on other containers which are not fully started or require (manual) actions to be executed first. We have resolved this by integrating an init daemon service in the application pipeline. The support for the init daemon service should be added by the component responsible.

The init daemon keeps track of the dependencies between containers/actions and orchestrates the startup of the services. Each service must therefore report its status to the init daemon and check with the init daemon service whether it can start.

To this end the init daemon service provides 3 REST calls:

• GET /canStart Validate if a given step (specified by its code) of a pipeline can be started. The step is specified through the step query param. Returns "true" or "false" as text/plain on success. E.g. /canStart?step="compute_aggregations"

• PUT /execute
  Start the execution of a step in a pipeline. The step is specified through the step query param. Returns 204 on success.
  E.g. /execute?step="compute_aggregations"

• PUT /finish Finish the execution of a step in a pipeline. The step is specified through the step query param. Returns 204 on success. E.g. /finish?step="compute_aggregations"

Each Docker image (or one of its parent images) used in your pilot case must support the init daemon service. I.e.:

• the container must check with the init daemon service whether it can start before starting
• the container must report to the init daemon service when it has started
• the container must report to the init daemon service when it has finished

E.g. the Spark submit image, which is a parent of the demo-spark-sensor-data image, supports the init daemon in its startup script. The relevant lines are highlighted in the code snippet below:

#!/bin/bash
export SPARK_MASTER_URL=spark://${SPARK_MASTER_NAME}:${SPARK_MASTER_PORT}

/wait-for-step.sh

echo "Submit application ${SPARK_APPLICATION_JAR_LOCATION} with main class    ${SPARK_APPLICATION_MAIN_CLASS} to Spark master ${SPARK_MASTER_URL}"
echo "Passing arguments ${SPARK_APPLICATION_ARGS}"

/execute-step.sh
/spark/bin/spark-submit \
    --class ${SPARK_APPLICATION_MAIN_CLASS} \
    --master ${SPARK_MASTER_URL} \
    ${SPARK_APPLICATION_JAR_LOCATION} ${SPARK_APPLICATION_ARGS}
/finish-step.sh

The bash script used can be found in the Spark base image folder. You can reuse these bash scripts in your Docker images. Each bash script executes an HTTP call to the init daemon service until the request succeeds. The scripts assume the following environment variables to be configured:

• ENABLE_INIT_DAEMON: flag to enable/disable the init daemon support. This should be enabled by default and can be configured in the Docker image as done in the Spark base image.
• INIT_DAEMON_BASE_URI: URL of the init daemon REST service. This should be configured in the Docker image with default value 'http://identifier/init-daemon' as done in the Spark base image.
• INIT_DAEMON_STEP: name of the step the service corresponds with. I.e. the service cannot start before the steps before this step have finished. This value needs to be set by the pilot case partner in the docker-compose.yml as explained later.

In some cases it might be that start and finish of a service cannot easily be reported to the init daemon service automatically. We can fallback to manual intervention in those cases, i.e. the executor of the pipeline needs to report manually whether a service has started successfully. However, we intend to automate as much as possible and reduce manual intervention to a minimum.

Using the init-daemon with the delta-service and the mu-swarm-logger-service

When using the init-daemon, the delta service and the mu-swarm-logger-service, it is also possible to update its step status automatically when a healthcheck is marked as healthy or unhealthy.

To do so, the init-daemon should subscribe to the effective deltas in order to be warned when a new healthcheck is detected. Modify the subscribers.json used to configure your delta as follows (where "initdaemon" is the name used for your init-daemon microservice):

{
  "potentials":[
  ],
  "effectives":[
    "http://initdaemon/process_delta"
  ]
}

Your mu-swarm-logger-service must also be configured to communicate with the delta service instead of the database directly, in order to warn any listeners when a new docker event is produced. Modify your docker-compose.yml file as follows (where "logger" is the name used for your mu-swarm-logger-service):

logger:
  links:
    - delta:database

You must then provide a few configurations on your microservice. Please check the example below (where "my-service" is the name of your microservice):

my-service:
  labels:
    - "LOG=1"
  environment:
    INIT_DAEMON_STEP: step_1
    INIT_DAEMON_STEP_STATUS_WHEN_HEALTHY: done
    INIT_DAEMON_STEP_STATUS_WHEN_UNHEALTHY: failed

Last but not least, your microservice needs to provide a healthcheck which will automatically be taken up by the swarm-logger:

my-service:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:4567"]
    interval: 30s
    timeout: 10s
    retries: 5

That's it! Your microservice should now be watched by the init-daemon for any changes to its health_status.

---

Implementing Pilot on BDE Stack

1. List the required images and dependencies
2. Add support for the init daemon
3. Build a pipeline flow
4. Create a configuration for the Integrator UI
5. Construct a docker-compose.yml file