Environment substitution for cloud-config #801
Comments
|
I think the variable names will be wrong. /etc/environment uses "COREOS_{PUBLIC,PRIVATE}_IPV4", but you are using "$private_ipv4" in your service ( fleet:
public-ip: $COREOS_PUBLIC_IPV4 # used for fleetctl ssh command
metadata: bandwidth_limit=$BANDWIDTH_LIMIT,cpus=$PROCESSORS,host=$HOSTNAME,memory=$MEMORY,provider=$PROVIDER,region=$REGION,storage_size=$INTERNAL_STORAGE,storage_type=$INTERNAL_STORAGE_TYPE,role=$ROLE,block_storage_size=$BLOCK_STORAGE_SIZE,block_storage_type=$BLOCK_STORAGE_TYPEYou shouldn't need to escape the variable because we don't do variable expansion sanely. The variable is only replaced if it exists (and this one doesn't). |
|
Hi, I need to escape it as it's part of a double-init.. The cloud config is delivered to the system within a writefiles inside another cloud-config - this, $public_ipv4 has to be escaped or else it's erased (replaced with an uninitialized value)! #205 states this, maybe you didn't read it but that's part of why I referenced it. Also, AFAIK none of the variables are being substituted - $public_ipv4 is supposed to be done as part of cloud-init. I'm pretty sure this functionality just doesn't exist - though, I think you are saying it does? - coreos-cloudinit does not seem use the environment to replace variables inside a cloud config. systemctl cat fleet.service after the second init provides this: So, $public_ipv4 is being replaced - no other variables are. Is it perhaps because of how I'm doing the double-init? I'm passing the environment in the unit and running |
|
I misunderstood. You want to:
I accomplish this a bit differently, but you can apply the same concept to a cloud-config: https://gist.github.com/crawford/ed7e954bfddd3394c775. To answer you original question (this is what threw me off), systemd does not do any of these substitutions. They are all performed by coreos-cloudinit. It is looking for COREOS_{PUBLIC,PRIVATE}_IPV{4,6}. |
|
Hi, I'm trying to do substitutions for MORE than just the IP address - maybe I didn't make that clear? See the metadata line. So you're saying.. coreos-cloudinit substitutes COREOS_{PUBLIC,PRIVATE}_IPV{4,6} with it's variable wherever ${public,private}_ipv{4,6} appears? Actually, the IP address IS substituted as you can see from my I'm suggesting that you should be able to provide OTHER environment variables and have them replaced in your cloud-config as well. I need to set more than IP dynamically, hostname, metadata (which will eventually pull from a metadata service) Is there a way at all for even OEMs to achieve this? |
|
Yes. Only these variables are substituted. If you need other variables, you'll need to leverage systemd for variable substitution. I think it would be easiest if you ignored the magic that coreos-cloudinit provides with its partial address substitution. You can use coreos.units to add a drop-in for fleet, which is what coreos-cloudinit does for you. This will allow you to source any environment files you need and let systemd do the substitutions. |
|
Hi, Thanks, that makes perfect sense and is probably easier than loading & substituting the file myself, though that's entirely possible since I'm scripting this all anyway - I just thought somehow, it would magically work - I was hoping the variable substitution in coreos-cloudinit was dynamic I guess. However, I'm substituting variables throughout the entire cloud-config, not just the metadata for the node, creating files based off the host name and such. I'm not sure that option would work in this scenario. I could see some advantages though if it were made that way, would you be against considering that? I could fork, test & maybe submit a PR. In the meantime, I may try what you suggested, if you think it will work in my use case. If you wouldn't mind an example would be appreciated, though your time is anyway! Thanks. |
|
What I should have done, is posted my gist of exactly what I'm doing - I don't know why I didn't. https://gist.github.com/ForbiddenEra/ee2d49b661c3066f4efd #cloud-config
ssh_authorized_keys:
- ssh-rsa key
write_files:
- path: /opt/cloud-config.yml
permissions: 0644
content: |
#cloud-config
ssh_authorized_keys:
- ssh-rsa key
hostname: $HOSTNAME
coreos:
etcd:
name: $HOSTNAME
discovery: $DISCOVERY
addr: \$private_ipv4:4001
peer-addr: \$private_ipv4:7001
election_timeout: 1500
heartbeat_interval: 500
fleet:
public-ip: \$private_ipv4 # used for fleetctl ssh command
metadata: bandwidth_limit=$BANDWIDTH_LIMIT,cpus=$PROCESSORS,host=$HOSTNAME,memory=$MEMORY,provider=$PROVIDER,region=$REGION,storage_size=$INTERNAL_STORAGE,storage_type=$INTERNAL_STORAGE_TYPE,role=$ROLE,block_storage_size=$BLOCK_STORAGE_SIZE,block_storage_type=$BLOCK_STORAGE_TYPE
units:
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: rpc-statd.service
command: start
enable: true
- name: example-setup.service
command: start
enable: true
content: |
[Unit]
Requires=network-online.target
Requires=docker.service
After=network-online.target
After=docker.service
Description=Shaped Setup
Documentation=http://shaped.ca
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin \
https://gist.githubusercontent.com/ForbiddenEra/../raw/.../volume-setup
ExecStartPre=/usr/bin/chmod +x /opt/bin/volume-setup
ExecStartPre=/usr/bin/wget -N -P /opt/bin \
https://gist.githubusercontent.com/ForbiddenEra/.../volume-unmount
ExecStartPre=/usr/bin/chmod +x /opt/bin/volume-unmount
ExecStartPre=/usr/bin/wget -N -P /opt/bin \
https://gist.githubusercontent.com/ForbiddenEra/../raw/.../volume-wait
ExecStartPre=/usr/bin/chmod +x /opt/bin/volume-wait
ExecStartPre=/usr/bin/update_engine_client -check_for_update
ExecStart=/bin/bash -c 'echo x.x.x.x docker-registry > /etc/hosts'
- name: volume-storage-online-wait.service
command: start
enabled: true
content: |
[Unit]
Requires=shaped-setup.service
Requires=network-online.target
After=shaped-setup.service
After=network-online.target
Before=volume-storage.service
Description=Wait for External Attached Volumes (for GlusterFS)
Documentation=http://shaped.ca/
[Service]
Type=oneshot
ExecStart=/opt/bin/volume-wait
RemainAfterExit=yes
- name: volume-storage.service
command: start
enabled: true
content: |
[Unit]
Requires=shaped-setup.service
Requires=network-online.target
Requires=volume-storage-online-wait.service
After=shaped-setup.service
After=network-online.target
After=volume-storage-online-wait.service
Description=Setup External Attached Volumes (for GlusterFS)
Documentation=http://shaped.ca/
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/volume-setup
ExecStop=/opt/bin/volume-unmount
[X-Fleet]
MachineMetadata=block_storage_type=attached
- name: swap-auto.service
command: start
enabled: true
content: |
[Unit]
Description=Setup Swap Service
Documentation=http://shaped.ca
[Service]
Type=oneshot
Environment="SWAP_PATH=/swap" "SWAP_FILE=swap.fs"
ExecStartPre=-/usr/bin/rm -rf ${SWAP_PATH}
ExecStartPre=/usr/bin/mkdir ${SWAP_PATH}
ExecStartPre=/usr/bin/touch ${SWAP_PATH}/${SWAP_FILE}
ExecStartPre=/usr/bin/chattr +C ${SWAP_PATH}/${SWAP_FILE}
ExecStartPre=/bin/bash -c "fallocate -l $(free -h --si|awk '/^Mem:/{print $2}') ${SWAP_PATH}/${SWAP_FILE}"
ExecStartPre=/usr/bin/chmod 600 ${SWAP_PATH}/${SWAP_FILE}
ExecStartPre=/usr/sbin/mkswap ${SWAP_PATH}/${SWAP_FILE}
ExecStartPre=/usr/sbin/losetup -f ${SWAP_PATH}/${SWAP_FILE}
ExecStart=/usr/bin/sh -c "/sbin/swapon $(/usr/sbin/losetup -j ${SWAP_PATH}/${SWAP_FILE} | /usr/bin/cut -d : -f 1)"
ExecStop=/usr/bin/sh -c "/sbin/swapoff $(/usr/sbin/losetup -j ${SWAP_PATH}/${SWAP_FILE} | /usr/bin/cut -d : -f 1)"
ExecStopPost=/usr/bin/sh -c "/usr/sbin/losetup -d $(/usr/sbin/losetup -j ${SWAP_PATH}/${SWAP_FILE} | /usr/bin/cut -d : -f 1)"
ExecStopPost=-/usr/bin/rm -rf ${SWAP_PATH}
RemainAfterExit=true
- name: install-busybox.service
command: start
enable: true
content: |
[Unit]
Requires=docker.service
Requires=shaped-setup.service
Requires=network-online.target
After=network-online.target
After=docker.service
After=shaped-setup.service
Description=Install BusyBox (for weave)
Documentation=http://zettio.github.io/weave/
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker pull busybox:latest
- name: install-pipework.service
command: start
enable: true
content: |
[Unit]
Requires=docker.service
Requires=shaped-setup.service
Requires=network-online.target
After=network-online.target
After=docker.service
After=shaped-setup.service
Description=Install Pipework
Documentation=https://github.com/jpetazzo/pipework
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin \
https://raw.github.com/jpetazzo/pipework/master/pipework
ExecStartPre=/usr/bin/chmod +x /opt/bin/pipework
ExecStart=/bin/echo Pipework Installed
- name: install-weave.service
command: start
enable: true
content: |
[Unit]
Requires=docker.service
Requires=shaped-setup.service
Requires=network-online.target
After=network-online.target
After=docker.service
After=shaped-setup.service
Description=Install Weave
Documentation=http://zettio.github.io/weave/
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin \
https://raw.github.com/zettio/weave/master/weave
ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
ExecStartPre=/usr/bin/docker pull zettio/weave:latest
ExecStart=/bin/echo Weave Installed
- name: 50-docker.network
runtime: false
content: |
[Match]
Type=bridge
Name=docker*
[Network]
Address=10.0.0.1/16
- name: dhcp.network
runtime: false
content: |
[Match]
Name=eth*
[Network]
DHCP=v4
- name: docker.netdev
runtime: false
content: |
[NetDev]
Name=docker0
Kind=bridge
- name: ethdocker.network
runtime: false
content: |
[Match]
Name=eth*
[Network]
Bridge=docker0
write_files:
- path: /home/core/.ssh/id_rsa.pub
permissions: 0600
content: |
- ssh-rsa key
- path: /home/core/.ssh/id_rsa
permissions: 0600
content: |
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
- path: /etc/conf.d/nfs
permissions: 0644
content: |
OPTS_RPC_MOUNTD=""
- path: /home/core/nfs/example.mount
permissions: 0644
content: |
[Mount]
What=nfshost.domain.com:/vol2/data
Where=/mnt/data
Type=nfs
- path: /etc/weave.$HOSTNAME.env
permissions: 0644
owner: core
content: |
WEAVE_LAUNCH_ARGS="-password key ip"
- path: /home/core/units/weave.service
permissions: 0644
owner: core
content: |
[Unit]
Description=Weave Network
Documentation=http://zettio.github.io/weave/
After=install-weave.service
Requires=install-weave.service
[Service]
EnvironmentFile=/etc/weave.%H.env
ExecStartPre=/opt/bin/weave launch $WEAVE_LAUNCH_ARGS
ExecStart=/usr/bin/docker logs -f weave
SuccessExitStatus=2
ExecStop=/opt/bin/weave stop
[X-Fleet]
Global=true
- path: /run/setup-environment.sh
permissions: 0755
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_iface_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
function get_nat_ip () {
while [ 1 ]; do
_out=$(curl -s icanhazip.com)
if [ -z "$_out" ]; then
sleep 1
else
echo $_out
exit
fi
done
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo getting private ipv4 from iface eth0..
echo COREOS_PUBLIC_IPV4=$(get_iface_ip eth0) >> $ENV # Also assigned to same IP
echo getting public-facing ipv4 from icanhazip.com..
echo COREOS_PRIVATE_IPV4=$(get_nat_ip) >> $ENV #eno1 should be changed to your device name
- path: /opt/bin/node-setup-environment
permissions: 0755
owner: root
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# we should be able to get most of this info from the
# open-stack meta-data service, however, not every providor has that
# so a. we can run our own meta-data service (lame)
# b. we can script as much meta-data as we can (current, lame)
# c. use openstack meta data (not so lame) but create our own elsewhere? (meh)
# d. figure out something better
mkdir -p /opt/bin
cd /opt
curl -s example.com/coreos/host.tgz | tar xvz
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/lib
# get list of subdomains matching core from our domain
host -l example.com example.com | grep ^core | cut -d ' ' -f 1,4 > /tmp/dns
while read l; do
_HOST=$(echo $l | cut -d ' ' -f 1)
_IP=$(echo $l | cut -d ' ' -f 2)
if [ $public_ipv4 == $_IP ]; then
echo Found my hostname: $_HOST
echo HOSTNAME=$_HOST >> $ENV
fi
done < /tmp/dns
rm /tmp/dns
export CLUSTER='dev001'
echo CLUSTER='dev001' >> $ENV
echo DISCOVERY=$(curl -s http://example.com/coreos/$CLUSTER.discovery)
echo PROCESSORS=$(nproc) >> $ENV
echo MEMORY=$(expr $(free -h --si|awk '/^Mem:/{print $2}' | cut -d '.' -f 1) \* 1024) >> $ENV
#echo INTERNAL_STORAGE=$(df -h /dev/vda1 | grep vda1 | cut -d ' ' -f 9 | cut -d 'G' -f 1) >> $ENV
echo INTERNAL_STORAGE=$(expr $(blockdev --getsize64 /dev/vda) / 1024 / 1024 / 1024) >> $ENV
echo INTERNAL_STORAGE_TYPE=ssd >> $ENV
echo BANDWIDTH_LIMIT=0 >> $ENV
echo BLOCK_STORAGE_SIZE=10 >> $ENV
echo BLOCK_STORAGE_TYPE=attached >> $ENV
echo PROVIDER=auro.io >> $ENV
echo REGION=ca-west >> $ENV
- path: /etc/resolv.conf
permissions: 0644
owner: root
content: |
nameserver 8.8.8.8
nameserver 8.8.4.4
domain example.com
hostname: intermediate
coreos:
units:
- name: setup-environment.service
command: start
runtime: true
content: |
[Unit]
Description=Setup configuration environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/run/setup-environment.sh
ExecStart=/opt/bin/node-setup-environment
- name: second-stage-cloudinit.service
runtime: true
command: start
content: |
[Unit]
Description=Run coreos-cloudinit with actual cloud-config after environment has been set up
Requires=setup-environment.service
After=setup-environment.service
#Requires=user-cloudinit-proc-cmdline.service
#After=user-cloudinit-proc-cmdline.service
[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/environment
ExecStart=/usr/bin/true
ExecStop=/usr/bin/coreos-cloudinit --from-file=/opt/cloud-config.yml |
|
I've written my own app to do this, but would still be awesome if I could just pass an environment to coreos-cloudinit and and it and it would handle replacement. |
|
Long term, I want to move people off of variable substitution in cloudinit and have them just use systemd instead. systemd is much better suited to do the substitution and it simplifies everything. |
|
I don't disagree with that idea, but how's that going to work? If I need variables replaced in my Metadata for example.. I suppose I could make a drop in for fleet with my environment, but can I specify things like Metadata in the environment? It would be nice to not have to double init, but I'm not sure I see a way around that in this situ. -------- Original message -------- Long term, I want to move people off of variable substitution in cloudinit and have them just use systemd instead. systemd is much better suited to do the substitution and it simplifies everything. — |
|
As the system exists today, you are right, you need to double-init and there is no way around it. Right now I am working on a new config evaluation utility that runs before the system boots. This will remove the need for the double init and will leverage systemd for variable substitution instead of (incorrectly) attempting to do it. |
|
Hi, Please keep me updated on this! In the meantime, slightly on topic but out of scope, I've written some tools to do cloud-config templates.. First one (called starship), will substitute variables in your cloud-config (inside the double-init phase) with either the system environment, or an environment you specify (by command line or EnvironmentFile in it's unit) which is generated by the second tool. The second tool (really just a script to gather metadata) waits on data sources (eg. waits for you to attach a public IP) before configuring the environment. The second tool Requires= and is After= the first tool, so your template wont be rendered before the data you require is available. Also, in creating the template, I had the idea to use template directories, for example, you could create a directory structure like: And, it would automatically generate a cloud-config based off that structure. Especially with write_files it would make it really easy to add files, especially binaries, as they would be added and generated as an encoded form. Your thoughts? |
|
Honestly? It's too complicated. Here is how I imagine "etcd.service.d/metadata.conf" looks: This way, cloud-init provides "/etc/environment", and "etcd.service" sources it. Let systemd do the substitutions for you. |
|
Ah, I understand what you mean now - and that makes sense, and I suppose the same could be done with fleet metadata instead of using the created helper units. There's not much else in the cloud-config then that would really need this that wouldn't mostly be done in a unit I suppose, maybe hostname: except, that could be done in a unit just to set the hostname after - though - I personally find this not over-complicated and actually helpful - but I'm always open to other / new / hopefully better ways of doing things! The idea of the directory structure still appeals to me though, I don't know if you were referencing that part or not, the rest does seem to be a little redundant / complicated though - though it would really, really depend on how much/what you were stuffing into your cloud-config - personally, I say why not use it to bootstrap everything you can possibly stuff in it - that's what it's meant for. Thus, having directories I can easily drop in files/remove/copy for each cluster seems a lot easier than editing the file every time - technically less risky too. Though - if your cloud-config is only a few lines or pulls a script that does the rest - it really is pointless. Your final thoughts appreciated (and all other input too of course) otherwise - since I was trying to use something in a way it wasn't designed - this isn't really an issue - other than the double init problem which you say you are building a separate tool for, correct? Feel free to close at your discretion. |
|
I always envisioned cloud-init to be used for initial system bootstrapping. Setting up things like networking and mounting disks. The configuration of running services is best left to another tool (e.g. fleet, puppet, ansible). Unfortunately, cloud-init was given too much capability and magic early on and, given our auto-updating nature, we couldn't prune it back. I'm confident that this new tool will make the distinction between machine configuration and application configuration more clear. |
|
Awesome. However, on this issue I just noticed something - I should really look at the source but.. I just accidentally pasted my template instead of my bootstrap into a machine, one of the first lines is That gets replaced by my template script - however - this WAS replaced by something? coreos-cloudinit? It was using the hostname directly from the OpenStack metadata service somehow.. The hostname SHOULD have been literally ${HOSTNAME} in this instance. Why was it replaced!? I thought the {public,private}_ip{4,6} variables were the only ones replaced..? Also, where's (specifically) it getting this metadata, from the metadata service as it's set to --oem=ec2-compat? Most of the metadata I need does come from the metadata service in this instance (though the flexibility is nice) - if all this time, I could have used any variables set in metadata, I could've saved significant time.. Though, some processing and maybe double-init still needed - at least, in this provider? as you can see, the domain is Just noticed this by accident, I thought I even tried that before and it wasn't replaced.. odd?! |
|
When I ran your config, I get: Is the config being processed by a shell at any point?
|
|
I've just tried writing a cloud config and to use variables and hit a wall -- http://pastie.org/10076641 I'd like to hear when your new tool is testable. Since ai was trying etcd2.0 in a container, maybe it's not the use case you see in the future though -- etcd2.0 will be "just there" and putting in a container not needed. |
|
I am also looking forward to this new way of bootstrapping coreos machines. Currently we are doing things like: - name: etcd.service
command: start
drop-ins:
- name: 50-public-ip.conf
content: |
[Service]
EnvironmentFile=/etc/environment
ExecStart=
ExecStart=/usr/bin/env ETCD_ADDR=${COREOS_PRIVATE_IPV4}:4001 \
ETCD_PEER_ADDR=${COREOS_PRIVATE_IPV4}:7001 \
/usr/bin/etcdbut this will break of course, if the default |
|
What about adding mustache support to cloud init? So the variables come from a file or url and are merged with cloud-config.yml using mustache tags. If you need things like ip addresses then you can write helpers. |
|
👍 |
Friday Mar 06, 2015 at 12:14 GMT
Originally opened as https://github.com/coreos/coreos-cloudinit/issues/325
I was trying to do something along the lines of..
With a "double init" where I init a basic system, run scripts to define all those variables, and then re-init with my real cloud-config..
I thought I could simply put them in /etc/environment and include that with my unit?
I'm guessing systemd doesn't do this (Kinda doing what's mentioned in https://github.com/coreos/coreos-cloudinit/issues/205 - the double init)
As a work around for now, I can have my scripts that set those variables push them to the cloud-init..
The text was updated successfully, but these errors were encountered: