An Ubuntu based Chronos container with the capability of logging to both standard and json format. It comes packaged with Logstash-Forwarder and is managed via Supervisord.
- Container Release: 1.1.3
- Mesos: 0.26.0-0.2.145.ubuntu1404
- Chronos: 2.4.0-0.1.20151007110204.ubuntu1404
Services Include
- Chronos - A Mesos Framework that provides a distributed and fault tolerant 'cron'.
- Logstash-Forwarder - A lightweight log collector and shipper for use with Logstash.
- Redpill - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
- Usage
- Example Run Command
- Example Marathon App Definition
- Modification and Anatomy of the Project
- Important Environment Variables
- Service Configuration
- Chronos
- Logstash-Forwarder
- Redpill
- Troubleshooting
When running the Chronos container in any deployment; the container does require several environment variables to be defined to function correctly.
-
ENVIRONMENT
-ENVIRONMENT
will enable or disable services and change the value of several other environment variables based on where the container is running (prod
,local
etc.). Please see the Environment section under Important Environment Variables. -
LIBPROCESS_IP
- The ip in which libprocess will bind to. (defaults to0.0.0.0
) -
LIBPROCESS_PORT
- The port used for libprocess communication (defaults to9000
) -
LIBPROCESS_ADVERTISE_IP
- If set, this will be the 'advertised' or 'externalized' ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication. -
LIBPROCESS_ADVERTISE_PORT
- If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication. -
CHRONOS_MASTER
- The zk url of Mesos Masters. -
CHRONOS_ZK_HOSTS
- A comma delimited list of Zookeeper servers used for storing Chronos state. Note: Does not need to be prefixed withzk://
.
The libprocess variables are not necessarily required if using host networking (as long as the default ip and port are available). However, you will quickly run into problems if attempting to run it alongside another container attempting to do the same thing. This is where running with an alternate LIBPROCESS_PORT
or running the container with standard bridge networking and using the two LIBPROCESS_ADVERTISE_*
variables is ideal.
A supplied sample seed script is available at /opt/scripts/marathon_env_init.sh
. This will assign 1 to 1 mappings of the 2 exposed ports needed for Chronos + Mesos to their associated variables.
PORT0
- The Chronos WebUIPORT1
- port used for bothLIBPROCESS_PORT
andLIBPROCESS_ADVERTISED_PORT
.
For further configuration information, please see the Chronos service section.
docker run -d \
--name chronos \
-e ENVIRONMENT=production \
-e PARENT_HOST=$(hostname) \
-e LOG_STDOUT_THRESHOLD=WARN \
-e LIBPROCESS_PORT=9200 \
-e LIBPROCESS_ADVERTISE_PORT=9200 \
-e LIBPROCESS_ADVERTISE_IP=10.10.0.11 \
-e CHRONOS_MASTER=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos \
-e CHRONOS_ZK_HOSTS=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181 \
-e CHRONOS_HOSTNAME=10.10.0.11 \
-e CHRONOS_HTTP_PORT=4400 \
-e CHRONOS_MESOS_FRAMEWORK_NAME=chronos \
-p 4400:4400 \
-p 9200:9200 \
chronos
{
"id": "/chronos",
"instances": 1,
"cpus": 1,
"mem": 512,
"container": {
"type": "DOCKER",
"docker": {
"image": "registry.address/mesos/chronos",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 31114,
"hostPort": 31114,
"protocol": "tcp"
},
{
"containerPort": 31115,
"hostPort": 31115,
"protocol": "tcp"
}
]
}
},
"env": {
"ENVIRONMENT": "production",
"ENVIRONMENT_INIT": "/opt/scripts/marathon_env_init.sh",
"CHRONOS_MASTER": "zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos",
"CHRONOS_ZK_HOSTS": "zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181",
"CHRONOS_LOG_STDOUT_THRESHOLD": "WARN",
"CHRONOS_HOSTANME": "$HOST",
"CHRONOS_MESOS_FRAMEWORK_NAME": "chronos"
},
"healthChecks": [
{
"protocol": "HTTP",
"portIndex": 0,
"path": "/",
"gracePeriodSeconds": 30,
"intervalSeconds": 20,
"maxConsecutiveFailures": 3
}
],
"uris": [
"file:///docker.tar.gz"
]
}
- Note: The example assumes a v1.6+ version of docker or a v2 version of the docker registry. For information on using an older version or connecting to a v1 registry, please see the private registry section of the Marathon documentation.
#!/bin/bash
##### Sample environment init script #####
# PORT0 = Chronos Web
# PORT1 = libprocess bind port
##########################################
local local_ip="$(ip addr show eth0 | grep -m 1 -P -o '(?<=inet )[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')"
export CHRONOS_HTTP_PORT="$PORT0"
export LIBPROCESS_IP="$local_ip"
export LIBPROCESS_PORT="$PORT1"
export LIBPROCESS_ADVERTISE_IP="$HOST"
export LIBPROCESS_ADVERTISE_PORT="$PORT1"
echo "[$(date)][env-init][CHRONOS_HTTP_PORT] $PORT0"
echo "[$(date)][env-init][LIBPROCESS_IP] $local_ip"
echo "[$(date)][env-init][LIBPROCESS_PORT] $PORT1"
echo "[$(date)][env-init][LIBPROCESS_ADVERTISE_IP] $HOST"
echo "[$(date)][env-init][LIBPROCESS_ADVERTISE_PORT] $PORT1"
File Structure
The directory skel
in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.
Init
The init script (./init.sh
) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.
Chronos
The chronos configuration will automatically be generated at runtime, however logging options are specified in /etc/chronos/log4j.properties
.
Supervisord
All supervisord configs can be found in /etc/supervisor/conf.d/
. Services by default will redirect their stdout to /dev/fd/1
and stderr to /dev/fd/2
allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.
In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.
Logstash-Forwarder
The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder
. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt
, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS
environment variable.
In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.
Variable | Default |
---|---|
ENVIRONMENT_INIT |
|
APP_NAME |
chronos |
ENVIRONMENT |
local |
PARENT_HOST |
unknown |
JAVA_OPTS |
|
LIBPROCESS_IP |
0.0.0.0 |
LIBPROCESS_PORT |
9000 |
LIBPROCESS_ADVERTISE_IP |
|
LIBPROCESS_ADVERTISE_PORT |
|
CHRONOS_LOG_DIR |
/var/log/chronos |
CHRONOS_LOG_FILE |
chronos.log |
CHRONOS_LOG_FILE_LAYOUT |
json |
CHRONOS_LOG_FILE_THRESHOLD |
|
CHRONOS_LOG_STDOUT_LAYOUT |
standard |
CHRONOS_LOG_STDOUT_THRESHOLD |
|
SERVICE_LOGSTASH_FORWARDER |
|
SERVICE_LOGSTASH_FORWARDER_CONF |
/opt/logstash-forwarder/chronos.log |
SERVICE_REDPILL |
|
SERVICE_REDPILL_MONITOR |
chronos |
-
ENVIRONMENT_INIT
- If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables. -
APP_NAME
- A brief description of the container. If Logstash-Forwarder is enabled, this will populate theapp_name
field in the Logstash-Forwarder configuration file. -
ENVIRONMENT
- Sets defaults for several other variables based on the current running environment. Please see the environment section for further information. If logstash-forwarder is enabled, this value will populate theenvironment
field in the logstash-forwarder configuration file. -
PARENT_HOST
- The name of the parent host. If Logstash-Forwarder is enabled, this will populate theparent_host
field in the Logstash-Forwarder configuration file. -
JAVA_OPTS
- The Java environment variables that will be passed to Marathon at runtime. Generally used for adjusting memory allocation (-Xms
and-Xmx
). -
LIBPROCESS_IP
- The ip in which libprocess will bind to. (defaults to0.0.0.0
) -
LIBPROCESS_PORT
- The port used for libprocess communication (defaults to9000
) -
LIBPROCESS_ADVERTISE_IP
- If set, this will be the 'advertised' or 'externalized' ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication. -
LIBPROCESS_ADVERTISE_PORT
- If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication. -
CHRONOS_LOG_DIR
- The directory in which the Chronos log files will be stored. -
CHRONOS_LOG_FILE
- The name of the chronos log file. -
CHRONOS_LOG_FILE_LAYOUT
- The log format or layout to be used for the file logger. There are two available formats,standard
andjson
. Thestandard
format is more humanly readable and is the chronos default. Thejson
format is easier for log processing by applications such as logstash. (Options:standard
orjson
). -
CHRONOS_LOG_FILE_THRESHOLD
- The log level to be used for the file logger. (Options:ERROR
,WARN
,INFO
, andDEBUG
) -
CHRONOS_LOG_STDOUT_LAYOUT
- The log format or layout to be used for console output. There are two available formats,standard
andjson
. Thestandard
format is more humanly readable and is the chronos default. Thejson
format is easier for log processing by applications such as logstash. (Options:standard
orjson
). -
CHRONOS_LOG_STDOUT_THRESHOLD
The log level to be used for console output. (Options:ERROR
,WARN
,INFO
, andDEBUG
) -
SERVICE_LOGSTASH_FORWARDER
- Enables or disables the Logstash-Forwarder service. Set automatically depending on theENVIRONMENT
. See the Environment section below. (Options:enabled
ordisabled
) -
SERVICE_LOGSTASH_FORWARDER_CONF
- The path to the logstash-forwarder configuration. -
SERVICE_REDPILL
- Enables or disables the Redpill service. Set automatically depending on theENVIRONMENT
. See the Environment section below. (Options:enabled
ordisabled
) -
SERVICE_REDPILL_MONITOR
- The name of the supervisord service(s) that the Redpill service check script should monitor.
local
(default)
Variable | Default |
---|---|
CHRONOS_HOSTNAME |
<first ip bound to eth0> |
CHRONOS_LOG_FILE_THRESHOLD |
WARN |
CHRONOS_LOG_STDOUT_THRESHOLD |
WARN |
JAVA_OPTS |
-Xmx256m |
SERVICE_LOGSTASH_FORWARDER |
disabled |
SERVICE_REDPILL |
enabled |
prod
|production
|dev
|development
Variable | Default |
---|---|
CHRONOS_LOG_FILE_THRESHOLD |
WARN |
CHRONOS_LOG_STDOUT_THRESHOLD |
WARN |
JAVA_OPTS |
-Xms384m -Xmx512m |
SERVICE_LOGSTASH_FORWARDER |
enabled |
SERVICE_REDPILL |
enabled |
debug
Variable | Default |
---|---|
CHRONOS_LOG_FILE_THRESHOLD |
DEBUG |
CHRONOS_LOG_STDOUT_THRESHOLD |
DEBUG |
JAVA_OPTS |
-Xms384m -Xmx512m |
SERVICE_LOGSTASH_FORWARDER |
disabled |
SERVICE_REDPILL |
disabled |
Chronos is a highly-available 'distributed cron' Mesos Framework developed by the folks at Airbnb. They've made the code and documentation available over at github.
By default, Chronos does not supporting providing start parameters as environment variables; however the init script will translate environment variables to parameters as long as they follow the form CHRONOS_<COMMAND_LINE_OPTION>
e.g. CHRONOS_MASTER=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos
.
A list of the Chronos command line flags can be found in their configuration docs. Alternatively, you can execute the following command to print the available options with the container itself:
docker run -it --rm chronos java -cp /usr/share/java:/usr/bin/chronos org.apache.mesos.chronos.scheduler.Main --help
In addition to the above Chronos configuration, some specific logging options have been added via the following variables:
Variable | Default |
---|---|
CHRONOS_LOG_DIR |
/var/log/chronos |
CHRONOS_LOG_FILE |
chronos.log |
CHRONOS_LOG_FILE_LAYOUT |
json |
CHRONOS_LOG_FILE_THRESHOLD |
|
CHRONOS_LOG_STDOUT_LAYOUT |
standard |
CHRONOS_LOG_STDOUT_THRESHOLD |
-
CHRONOS_LOG_DIR
- The directory in which the Chronos log files will be stored. -
CHRONOS_LOG_FILE
- The name of the chronos log file. -
CHRONOS_LOG_FILE_LAYOUT
- The log format or layout to be used for the file logger. There are two available formats,standard
andjson
. Thestandard
format is more humanly readable and is the chronos default. Thejson
format is easier for log processing by applications such as logstash. (Options:standard
orjson
). -
CHRONOS_LOG_FILE_THRESHOLD
- The log level to be used for the file logger. (Options:ERROR
,WARN
,INFO
, andDEBUG
) -
CHRONOS_LOG_STDOUT_LAYOUT
- The log format or layout to be used for console output. There are two available formats,standard
andjson
. Thestandard
format is more humanly readable and is the chronos default. Thejson
format is easier for log processing by applications such as logstash. (Options:standard
orjson
). -
CHRONOS_LOG_STDOUT_THRESHOLD
The log level to be used for console output. (Options:ERROR
,WARN
,INFO
, andDEBUG
)
Logstash-Forwarder is a lightweight application that collects and forwards logs to a logstash server endpoint for further processing. For more information see the Logstash-Forwarder project.
Variable | Default |
---|---|
SERVICE_LOGSTASH_FORWARDER |
|
SERVICE_LOGSTASH_FORWARDER_CONF |
/opt/logstash-forwadrer/chronos.conf |
SERVICE_LOGSTASH_FORWARDER_ADDRESS |
|
SERVICE_LOGSTASH_FORWARDER_CERT |
|
SERVICE_LOGSTASH_FORWARDER_CMD |
/opt/logstash-forwarder/logstash-forwarder -config=”$SERVICE_LOGSTASH_FORWARDER_CONF” |
-
SERVICE_LOGSTASH_FORWARDER
- Enables or disables the Logstash-Forwarder service. Set automatically depending on theENVIRONMENT
. See the Environment section. (Options:enabled
ordisabled
) -
SERVICE_LOGSTASH_FORWARDER_CONF
- The path to the logstash-forwarder configuration. -
SERVICE_LOGSTASH_FORWARDER_ADDRESS
- The address of the Logstash server. -
SERVICE_LOGSTASH_FORWARDER_CERT
- The path to the Logstash-Forwarder server certificate. -
SERVICE_LOGSTASH_FORWARDER_CMD
- The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
Redpill is a small script that performs status checks on services managed through supervisor. In the event of a failed service (FATAL) Redpill optionally runs a cleanup script and then terminates the parent supervisor process.
Variable | Default |
---|---|
SERVICE_REDPILL |
|
SERVICE_REDPILL_MONITOR |
chronos |
SERVICE_REDPILL_INTERVAL |
|
SERVICE_REDPILL_CLEANUP |
-
SERVICE_REDPILL
- Enables or disables the Redpill service. Set automatically depending on theENVIRONMENT
. See the Environment section. (Options:enabled
ordisabled
) -
SERVICE_REDPILL_MONITOR
- The name of the supervisord service(s) that the Redpill service check script should monitor. -
SERVICE_REDPILL_INTERVAL
- The interval in which Redpill polls supervisor for status checks. (Default for the script is 30 seconds) -
SERVICE_REDPILL_CLEANUP
- The path to the script that will be executed upon container termination.
root@c90c98ae31e1:/# /opt/scripts/redpill.sh --help
Redpill - Supervisor status monitor. Terminates the supervisor process if any specified service enters a FATAL state.
-c | --cleanup Optional path to cleanup script that should be executed upon exit.
-h | --help This help text.
-i | --interval Optional interval at which the service check is performed in seconds. (Default: 30)
-s | --service A comma delimited list of the supervisor service names that should be monitored.
In the event of an issue, the ENVIRONMENT
variable can be set to debug
. This will stop the container from shipping logs and prevent it from terminating if one of the services enters a failed state.
For further support, please see either the Chronos Github Project or post to the Chronos Google Group.