Skip to content

App Definition Authoring for KubeDirector

Jump to bottom
Joel Baxter edited this page Aug 17, 2022 · 21 revisions

The application expert preparing an application for KubeDirector deployment is responsible for providing up to three kinds of components:

  • application metadata in the form of a KubeDirectorApp resource
  • a container image (or multiple images) encapsulating the application software
  • optionally, a package of setup scripts/artifacts (or multiple packages)

The KubeDirectorApp resource identifies the other components. Together, these components form an entry in the KubeDirector catalog of application types.

Because of this referencing, it is quite possible for example for a given container image or setup package to be used by multiple different KubeDirectorApp resources. However the application author must always ensure that the referenced container image(s) and setup package(s) work together to implement the application behavior promised by the KubeDirectorApp.

The sections below describe these components in more detail.

KubeDirectorApp

The KubeDirectorApp resource describes the services the cluster will provide and the "roles" that members (virtual hosts) can take on within the application cluster.

Each role is associated with a container image, an optional setup package, and a set of service endpoints that each role member will provide. A role can also have other characteristics such as an allowed number of role members or minimum memory size per member.

A sample KubeDirectorApp for defining a Cassandra application instance is provided below. This is a custom resource provided to the k8s API, but keep in mind that this document is not interpreted by any k8s service -- it is only used by KubeDirector.

This example describes an arrangement that has two or more "seed" role members and a variable number of "worker" role members. Two services are defined, both of which run on the members of either role. The final few properties declare that extra deployment steps must be taken to support systemd within the container and provide certain additional container privileges required by the app; also, if persistent storage is used to deploy this app, the "/data" directory should be placed there (along with other default persisted directories "/usr", "/opt", "/var", and "/etc").

This is a simple app type definition in many respects. Roles can differ from each other in several ways, not only in the services that run on a role member but also in the container image and setup package used within the role. To keep this example straightforward, such role features are not used here.

apiVersion: kubedirector.hpe.com/v1beta1
kind: KubeDirectorApp
metadata:
  name: cassandra311
spec:
  label:
    name: Cassandra 3.11
    description: Cassandra 3.11 on centos 7.x
  distroId: bluedata/cassandra311
  version: '3.1'
  defaultImageRepoTag: docker.io/bluedata/cassandra:3.0
  configSchemaVersion: 7
  defaultConfigPackage:
    packageURL: https://s3.amazonaws.com/bluek8s/kubedirector/cassandra311/appconfig-3.1.tgz
  services:
  - id: ssh
    label:
      name: SSH
    endpoint:
      port: 22
      isDashboard: false
  - id: cassandra
    label:
      name: cql
    endpoint:
      port: 9042
      urlScheme: cql
      isDashboard: false
  roles:
  - id: seed
    cardinality: 2+
  - id: worker
    cardinality: 0+
  config:
    selectedRoles:
    - seed
    - worker
    roleServices:
    - roleID: seed
      serviceIDs:
      - ssh
      - cassandra
    - roleID: worker
      serviceIDs:
      - ssh
      - cassandra
  defaultPersistDirs:
  - "/usr"
  - "/opt"
  - "/var"
  - "/data"
  capabilities:
  - SYS_RESOURCE
  - IPC_LOCK
  systemdRequired: true

For more details, see the complete spec for KubeDirectorApp.

Container Images

A container image for a role contains the application software already installed and ready to start after some final runtime configuration. Images can be shared among roles, but if the roles provide widely divergent services then it is better to use unique per-role images to minimize image size.

Inside a KubeDirectorApp role spec, the imageRepoTag property can be used to specify the image for that role. The top-level defaultImageRepoTag property of the KubeDirectorApp spec can also be used to specify an image for any role that does not include its own imageRepoTag.

Note that each container in the cluster will be an independent network entity, likely running multiple services on their usual ports. Preparing the image will be very similar to installing the application on a bare-metal host.

The images must be hosted on a container image repository, such as Docker Hub or quay.io, that will be accessible to the k8s nodes.

Images must have bash installed and available in the standard executables path for the container user. If the app will use a setup package (as per below), the image must also have python (2 or 3), curl, and tar available.

Finally, if you want to enable detailed storage-initialization progress reporting in the "storageInitProgress" field of a kdcluster's MemberStateDetail, the app image must provide the rsync utility. This specifically must be a version of rsync that supports these command-line arguments: --log-file=filename --info=progress2 --relative -ax --no-inc-recursive

Setup Package

At runtime, an application will usually need to be configured with information about the identities of the virtual cluster members and other configuration or state. This is accomplished through an app setup package.

Inside a KubeDirectorApp role spec, the configPackage property can be used to specify the setup package (or absence of one) for that role. The top-level defaultConfigPackage property of the KubeDirectorApp spec can also be used to determine this for any role that does not include its own configPackage.

If a setup package is specified for a role, then KubeDirector will automatically do the following steps inside each container created for a virtual cluster:

  • Create a KubeDirector config file tailored for that container's role and specific identity, derived from the KubeDirectorCluster, the relevant KubeDirectorApp, and runtime state.
  • Fetch the application setup package designated by the KubeDirectorApp resource (see below for a description of possible package source locations).
  • Unpack it.
  • Run a script from the extracted package contents, with arguments that specify an initial configuration. (This script will also be used later to notify when other members are added to or removed from the cluster.)

If the virtual cluster is later resized/reconfigured, KubeDirector will update the in-container config files as necessary, and in some cases invoke the startscript again to allow the app configuration to change in response.

Therefore if this kind of runtime configuration is required, the author of the app definition must provide that setup package. The setup package must be a tgz archive. That archive can either be hosted at a webserver accessible by the k8s nodes (in the future, other methods of package hosting may be possible) or a file in the filesystem of the container image. The value of the relevant packageURL property (within the config package object) must be a URL with the "http", "https", or "file" protocol.

That archive must contain a directory, and within that directory an executable script named "startscript". It may also contain any other scripts or artifacts that will be in turn used by "startscript".

The startscript will be invoked at three different lifecycle events, with the following command-line arguments. In the cases where a list of FQDNs is provided, the list is a single string composed of comma-separated FQDNs.

Event Arguments
this member created --configure
new members in some role have been added --addnodes --role <role_id> --fqdns <list_of_added>
members in some role will be deleted --delnodes --role <role_id> --fqdns <list_to_be_deleted>

The "created" event will also be invoked after member reboot in the following two cases:

  • the member does not use persistent storage
  • the previous attempt to configure the member was unsuccessful

It is the responsibility of the startscript to determine whether the current node, in its defined role, has anything that it needs to do in response to these events. The script can query the KubeDirector-created config file to determine its own role and other information that may be necessary to handle to the lifecycle event. Its bottom-line responsibility is to provide the service endpoints advertised by the KubeDirectorApp for its role.

An edited example of a startscript for the Cassandra application is provided below. Note that the setup package does not only contain this script; it will also contain some standard shell macro/utility definitions for reading the KubeDirector config file, and any application config file templates or other artifacts that the startscript will make use of.

#!/bin/bash

# The script flow will initially diverge depending on whether arguments
# indicate that it was invoked for initial configuration, or as a notification
# of other members entering/leaving the cluster. We'll assume initial
# configuration in this example.

source macros.sh # utilities common for all app setups

# Edit various app config file templates. The all-caps macros such as
# GET_FQDN_LIST provide info from this container's KubeDirector-provided
# configuration info.

REPLACE_PATTERN @@@@SEEDS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_FQDN_LIST seed

REPLACE_PATTERN @@@@LISTEN_ADDRESS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_NODE_FQDN

REPLACE_PATTERN @@@@CONCURRENT_COMPACTORS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_TOTAL_VCORES

# etc. ...

# Start services appropriate to this member's role and the deployment choices.
# This could use macros to determine the current member's role and the
# appropriate services for that role; in this case however we know that both
# the seed role and the worker role always want to run the cassandra service.
# The sshd service is already started by default.

REGISTER_START_SERVICE_SYSCTL cassandra cassandra

Detailed documentation for startscript authoring is not yet available. See the "macros.sh" file in the example app setup packages for the available macros and comments on their functionality. Note that the current example app setup packages have been adapted from existing setup scripts and may still contain vestigial code or artifacts not related to KubeDirector; this is being addressed.

Clone this wiki locally