Container Linux automates machine provisioning with a specialized system for applying initial configuration. This system implements a process of (trans)compilation and validation for machine configs, and an atomic service to apply validated configurations to machines.
Container Linux admins define these configurations in a format called the Container Linux Config. Container Linux Configs are structured as YAML and intended to be human-readable. The Container Linux Config is specific to Container Linux, as the name implies. It has features devoted to configuring Container Linux services such as etcd, rkt, Docker, flannel, and locksmith. The defining feature of the Container Linux Config is that it cannot be sent directly to a Container Linux provisioning target. Instead, it is first validated and transformed into a machine-readable and wire-efficient form. Before examining why and how that process is conducted, a few example Container Linux Configs will give a flavor of the format:
This extremely simple Container Linux Config will fetch and run the current release of etcd:
etcd:
Adding onto this, the desired version of etcd can also be specified. The following example Container Linux Config will provision a new Container Linux machine to fetch and run the etcd service, version 3.1.6:
etcd:
version: 3.1.6
The next example Container Linux Config is more realistic. It uses variable replacement to configure the etcd service with the provisioning target's public and private IPv4 addresses, making it repeatable across a group of machines:
etcd:
advertise_client_urls: http://{PUBLIC_IPV4}:2379
initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
listen_client_urls: http://0.0.0.0:2379
listen_peer_urls: http://{PRIVATE_IPV4}:2380
discovery: https://discovery.etcd.io/<token>
But what are PUBLIC_IPV4 and PRIVATE_IPV4? These variables are populated from the environment in which Container Linux runs (e.g. EC2, Azure, GCE), assuming this metadata exists. In the default EC2 case, the public_ipv4 and local_ipv4 metadata would be used. On bare metal, this information cannot be reliably derived in a general manner, so these variables couldn't be used. In Azure, either the virtual IP or public IP could be used for the PUBLIC_IPV4 and the dynamic IP would be used for the PRIVATE_IPV4. These are just a few examples. Given the many different environments in which Container Linux can run, it's difficult if not impossible to accurately determine these variables in every scenario.
On top of this variable-expansion problem, it is very common for users to inadvertently write invalid configs. Instead of forcing users to waste time testing potentially invalid configs, the use of a transformation tool is instead encouraged. The default tool recommended for this task is the Config Transpiler (ct for short). It is responsible for validating and transforming a Container Linux Config into the format that Container Linux can consume: the Ignition Config.
The Ignition Config is the configuration format consumed directly by Container Linux. Ignition, the utility in Container Linux responsible for provisioning the machine, actually fetches and executes the Ignition Config.
Ignition Configs are mostly static, distro-agnostic, and meant to be generated by a machine rather than a human. While they can be written directly by users, it is highly discouraged due to the easy that which errors can be introduced. Instead of writing Ignition Configs directly, users are encouraged to use provisioning tools like Matchbox, which transparently translate Container Linux Configs to Ignition Configs, or use the Config Transpiler directly.
As can be seen above, ct will typically only be manually invoked when users are manually provisioning machines. If a provisioning tool like Matchbox is used, ct will transparently be incorporated into the deployment pipeline. In this scenario, the user only needs to worry about preparing a Container Linux Config - Ignition and the Ignition Config are merely an implementation detail.
The Container Linux Config Transpiler abstracts the details of configuring Container Linux. It's responsible for transforming a Container Linux Config written by a user into an Ignition Config to be consumed by instances of Container Linux.
To go back to the previous example of configuring etcd, the following config will configure an etcd cluster using the machine's public and private IP addresses:
etcd:
advertise_client_urls: http://{PUBLIC_IPV4}:2379
initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
listen_client_urls: http://0.0.0.0:2379
listen_peer_urls: http://{PRIVATE_IPV4}:2380
discovery: https://discovery.etcd.io/<token>
It was alluded to earlier that ct requires information about the target environment before it can transform configs which use templating. If this config is passed to ct without any other arguments, ct fails:
$ bin/ct < example.yml
error: platform must be specified to use templating
This error message states that because the config takes advantage of templating (e.g. PUBLIC_IPV4), ct requires being invoked with the --platform argument. This extra information is used by ct to make the platform-specific customizations necessary. Keeping the Container Linux Config and the invocation arguments separate allows the Container Linux Config to remain largely platform independent.
CT can be invoked again and given Amazon EC2 as an example:
$ bin/ct --platform=ec2 < example.yml
{"ignition":{"version":"2.0.0","config"...
This time, ct successfully runs and produces the following Ignition Config:
{
"ignition": { "version": "2.0.0" },
"systemd": {
"units": [{
"name": "etcd-member.service",
"enable": true,
"dropins": [{
"name": "20-clct-etcd-member.conf",
"contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n --listen-peer-urls=\"http://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n --listen-client-urls=\"http://0.0.0.0:2379\" \\\n --initial-advertise-peer-urls=\"http://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n --advertise-client-urls=\"http://${COREOS_EC2_IPV4_PUBLIC}:2379\" \\\n --discovery=\"https://discovery.etcd.io/\u003ctoken\u003e\""
}]
}]
}
}
This Ignition Config enables and configures etcd as specified in the above Container Linux Config. This can be more easily seen if the contents of the etcd drop-in are formatted nicely:
[Unit]
Requires=coreos-metadata.service
After=coreos-metadata.service
[Service]
EnvironmentFile=/run/metadata/coreos
ExecStart=
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \
--listen-peer-urls="http://${COREOS_EC2_IPV4_LOCAL}:2380" \
--listen-client-urls="http://0.0.0.0:2379" \
--initial-advertise-peer-urls="http://$${COREOS_EC2_IPV4_LOCAL}:2380" \
--advertise-client-urls="http://${COREOS_EC2_IPV4_PUBLIC}:2379" \
--discovery="https://discovery.etcd.io/<token>"The details of these changes are covered in depth in Ignition's metadata documentation, but the gist is that coreos-metadata is used to fetch the IP addresses from the Amazon APIs and then systemd is leveraged to substitute the IP addresses into the invocation of etcd. The result is that even though Ignition only runs once, coreos-metadata fetches the IP addresses whenever etcd is run, allowing etcd to use IP addresses that have the potential to change.
Previously, the recommended way to provision a Container Linux machine was with a cloud-config. These configs would be given to a Container Linux machine and a utility called coreos-cloudinit would read this file and apply the configuration on every boot.
For a number of reasons, coreos-cloudinit has been deprecated in favor of Container Linux Configs and Ignition. For help migrating from these legacy cloud-configs to Container Linux Configs, refer to the migration guide.
Now that the basics of Container Linux Configs have been covered, a good next step is to read through the examples and start experimenting. The troubleshooting guide is a good reference for debugging issues.