WebPlatform.org Salt Stack deployment scripts
SaltStack HTML Shell Python Scheme Vim script Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
_grains
_utils
apache
backup
basesystem
bench
bots
buggenie
builder
code
cron
dhcp
discourse
elasticsearch
environment/files
etcd
etherpad
formula
fxa
gdnsd
gerrit
git
groups
hhvm
hosts
hypothesis
java
logging
logwatch
lvm
mail
mediawiki
memcached
mercurial
mmonit
monitor
mysql
network
nginx
nodejs
ntp
nutcracker
phabricator
php-fpm
php
piwik
postfix
python
reactor/reactions
resolvconf/files
roles
rsync
salt
specific
specs
ssl
subversion
sudo
sysstat
tasks/dreamobjects_upload
users
webat25
webplatform
wordpress
xfs
xinetd
.gitignore
README.md
top.sls

README.md

WebPlatform server environment configuration

Making a new salt-master

Send a copy of init.sh from webplatform/ops in salt-master/ to an empty Ubuntu 14.04 VM and run the script as root.

To deploy web apps, we are currently assuming that the salt master will host all the code repositories along with the dependency management tools installed.

In order to allow us to have a VM to prepare each web app, the following has to be run on the salt master. We will eventually create a VM specifically for that later in this project.

Security groups configuration

All services that requires TCP/UDP connectivity should have, in the state file or configuration file, an annotation about it.

To find all security groups that requires communication, search the files for notes that contains ## SecurityGroup. This string can be as a comment, or at the end of a configuration line.

For example, MediaWiki Settings.php has a line like this:

$wpdUdp2logDest = 'salt:8420'; ## SecurityGroup port: UDP 8420 @salt demux.py

All security groups are expected to allow local network communication exclusively by allowing only incoming traffic from 10.10.10.0/24 or any other local network IP class available.

As for inter network communication, we should maintain explicitly which security groups are allowing outside communication.

Connecting with SSH

Not ALL VMs in an environment has a publicly available IP address. To connect to them, you have to configure your local SSH client to use the salt master as a proxy. SSHing to a VM is done in the following way:

ssh app1.staging.wpdn

The app1.staging.wdpn is visible from the salt master through either a hosts file entry or a local DNS server but your own local machine cannot see it. This is where your local SSH config comes into play.

To access them, add to your ~/.ssh/config:

## Use salt as a Jump box
##
## Reference:
##   - http://serverfault.com/questions/337274
##   - https://wikitech.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option
##   - http://blog.pluralsight.com/linux-ssh-jumpbox
Host staging.wpdn
  Hostname salt.webplatformstaging.org
  ProxyCommand none
Host *.staging.wpdn
  ProxyCommand ssh -e @ -o StrictHostKeyChecking=no -a -W %h:%p staging.wpdn

Config stashes

Bookmarks

Useful related documentation

Maintenance commands

Here are a few commands that can be done with Salt Stack.

Every commands here can be run manually. Ideally there should be always a salt state to enforce anything permanent. But sometimes we have to act quickly and update the states later.

  1. Rewrite a setting in one config file

    salt app* file.replace /etc/php5/apache2/php.ini pattern='expose_php = On' repl='expose_php = Off'

  2. Get to know what are the grants for one user

    salt db* mysql.user_grants accounts '%'

  3. Reload apache2 config (or any service)

    salt app* service.reload apache2

  4. Can we write to the filesystem?

    salt bots cmd.run "touch /root/TestIfWeCabWrite && echo 'OK' || echo 'Writing to filesystem failed'" salt bots cmd.run "rm /root/TestIfWeCabWrite && echo 'OK' || echo 'Writing to filesystem failed'"

  5. Upgrade Operating System packages (If it has a service, it’ll also handle its restart)

    salt app* pkg.upgrade

  6. Delete a user

    salt * user.delete foobar remove=True force=True

  7. New salt minion

    nova boot --image Ubuntu-14.04-Trusty --user-data /srv/opsconfigs/userdata.txt --key_name renoirb-staging --flavor lightspeed --security-groups default,all,frontend app1 salt-key -y -a app1 salt app1 test.ping salt app1 grains.get level salt app1 state.highstate

  8. Assing a floating IP

Assuming our VM app1 has private IP 10.10.10.170 and we want floating IP of 173.236.254.223

  • We need the floating IP id identifier (e.g. foo) in the 4 you’ll get. Technically the second value should be empty, that’s where we should see our VM private IP once its done. Identifier will be a UUID string

    neutron floatingip-list | grep 173.236.254.223
    | foo |      |  173.236.254.223  |    |
    
  • We need to know which ethernet adapter to bind the floating IP to. We need the first value.

    neutron port-list | grep 10.10.10.170
    | bar |      | fa:16:3e:c1:6c:a0 | {"subnet_id": "...", "ip_address": "10.10.10.170"}
    
  • We do have what we need to assign, enter them in this order:

    neutron floatingip-associate --fixed-ip-address 10.10.10.170 foo bar
    
  1. Before updating salt states, ensure you have all IP addresses
  • Get on any VM, locally (notice salt-call) what are the IP addresses it has from its neighboors

    salt-call mine.get \* network.ip_addrs
    
  • Ask every VMs what they have (basically it asks every node to ask themselves to their neighboors what they have)

    salt \* mine.get \* network.ip_addrs
    
  • Force an update of the mine

    salt \* mine.flush
    salt \* mine.update
    
  • If the previous flush didn’t work, in that case lets restart all salt-minions so it’ll listen to what we mean

    salt \* service.restart salt-minion
    salt \* saltutil.sync_all
    salt \* mine.update
    
  1. Get the IP addresses of a VM
  • All adapters

    salt \* grains.get ip4_interfaces
    
  • Only the eth0 adapter

    salt \* grains.get ip4_interfaces:eth0
    
  1. Create a database and privileges on masterdb. If there is replication, the secondary MySQL servers with replication will get the changes by themselves
  • Create a database

    salt -G 'roles:masterdb' mysql.db_create dbname utf8 utf8_general_ci
    
  • Add database user and privileges

    salt -G 'roles:masterdb' mysql.user_create dbuser '%' somepass
    salt -G 'roles:masterdb' mysql.grant_add 'ALL' 'dbname.*' 'dbuser' '%'
    
  1. Remove a file from multiple machines

    salt * file.remove /etc/monit/conf.d/exim4.conf