Move most Salt docs in tree

This also includes Vagrant information,
and should cover all the steps of a normal Salt workflow.

This does not (yet) include information about
setting up new Salt masters/minions,
or how to restart various services cleanly after a deploy.

Moving docs in-tree ensures they stay up to date with code changes,
as opposed to having to remember to update the wiki after long
review cycles.
The corresponding sections will be removed from the wiki
and replaced with pointers to the in-tree docs.

docs/salt.md

@@ -1,6 +1,244 @@
 # Notes on Salt
 
-## Looking up previous runs
+## Salt overview
+
+We use [SaltStack](https://saltstack.com/) (Salt for short)
+to configure our infrastructure machines.
+
+We're currently on the Salt 2016.3 release branch, so make sure to look at the
+[right version of the docs](https://docs.saltstack.com/en/2016.3/contents.html).
+
+Salt configurations are meant to be idempotent and can be applied as many times
+as you like; a single deploy is termed a `highstate`, which
+runs code to make minion(s) match the requested configuration.
+
+Salt has a number of different modes of operation, including masterless and SSH;
+we are using the default mode, which is a single master and many minions.
+Note that the `servo-master1` machine runs both a `salt-master` process
+and a `salt-minion` process, so we can Salt that machine itself.
+Deploys should be run from the master machine, `servo-master1`.
+
+Also, note that we may occasionally run multiple masters in a Salt multi-master
+setup, e.g. this occurred when we migrated from a master on Linode to a master
+on EC2. See [this issue](https://github.com/servo/saltfs/issues/281) for more.
+
+## Salt workflow
+
+This (the servo/saltfs) repository holds all of our Salt code, which describes
+how we want to configure our machines.
+We are using a feature of Salt called gitfs, which means that the Salt master
+will automatically use the latest version of this repo on the master branch
+at all times, which makes it hard to get out of sync and makes deploys easier.
+
+For this reason, the recommended workflow for making changes is to make PRs
+to this repository; however, note that there is a
+[manual override]([Discouraged: Testing in production])
+that can be used if necessary.
+
+### Getting started
+
+The best way to make changes is to try them out locally,
+shepherd them through the review and pull request process,
+then apply them in production.
+
+Clone the [servo/saltfs](https://github.com/servo/saltfs/) repo locally,
+and create a new feature branch:
+
+```console
+$ git clone git@github.com:servo/saltfs.git servo-saltfs
+$ cd servo-saltfs
+$ git checkout -b new-feature
+```
+
+Make changes as needed on your branch.
+
+### Testing with Vagrant
+
+You can test your changes using [Vagrant](https://vagrantup.com);
+a Vagrantfile is included in the repo.
+Vagrant's Salt provisioner tends to make backwards-incompatible changes often,
+so only a few versions of Vagrant will work at any time;
+the Vagrantfile has information about which ones.
+Currently, version 1.8.1 is recommended;
+note that many OSs like Ubuntu may still have older versions like 1.7.x,
+so you may need to
+[download a different version](https://releases.hashicorp.com/vagrant/).
+Note that we are using the [VirtualBox](https://www.virtualbox.org/) backend
+for the actual VMs; the Vagrant installer should automatically install it for
+you on Windows and macOS.
+
+Each node type has a corresponding Vagrant target,
+and Vagrant will run the Salt states when provisioning.
+For example, you might run these commands from within
+your checkout of the saltfs repo (same directory as the Vagrantfile):
+
+* `$ vagrant up`: Start + provision all VMs configured in Vagrantfile,
+  alternately `$ vagrant up servo-master1` to start just one
+* `$ vagrant ssh servo-master1`: Connect to a running VM
+
+Note that your prompt will change once you're inside a VM.
+
+* `vagrant@vagrant-ubuntu-trusty-64$ sudo apt-get -y install cowsay; cowsay 'servo-saltfs ❤  Vagrant'`:
+  Run commands in the VM
+* `vagrant@vagrant-ubuntu-trusty-64$ exit`: Leave the VM
+
+Back outside the VM:
+
+* `$ vagrant provision servo-master1`: (Re-)apply the Salt states
+  on the servo-master1 VM
+* `$ vagrant halt`: Halt all running VMs
+
+Note that Salt states are (should be) idempotent,
+so you can run the provisioning as many times as you want.
+Remember to re-run the `vagrant provision` command to apply your changes
+to your VMs after you edit their config files.
+If you're making changes, please run `vagrant provision` at least twice
+to ensure this invariant of idempotency.
+
+:warning: The Vagrantfile gives each VM 1 GB of memory,
+which is enough to apply the Salt states but not enough to build Servo.
+If you need to build Servo, edit the Vagrantfile locally to allot 4 GB
+(remember not to commit this change!):
+
+```ruby
+vbox.memory = 4096
+```
+
+:dash: [vagrant-cachier](http://fgrehm.viewdocs.io/vagrant-cachier/) is
+recommended to make running `vagrant provision` faster
+by caching downloaded packages.
+The Vagrantfile will automatically take advantage of it:
+run ```vagrant plugin install vagrant-cachier``` to install it.
+
+There's also a test script that currently has a few lints;
+it requires Python 3.5 (currently must be installed manually)
+and can be run as follows (must be inside the VM):
+
+```console
+vagrant@vagrant-ubuntu-trusty-64$ ./test.py
+```
+
+### PR + Review process
+
+Once you're happy, `git commit` your changes,
+`git push` them up to your fork of the saltfs repo on Github,
+and [open a pull request](https://github.com/servo/saltfs/compare).
+
+A Servo reviewer should review your changes,
+roughly along [these lines](https://github.com/servo/servo/wiki/Code-review).
+
+:memo: Take a look at the STYLE_GUIDE.md in the repository root
+for some helpful hints.
+
+### Deploying changes
+
+Once your PR has been approved and merged by Travis,
+it will be available on the master branch and Salt will automatically
+use that code for deploys.
+
+Examples of highstates, which should be run as root from the Salt master:
+
+* `$ salt '*' state.highstate`: Apply all configs to all hosts
+* `$ salt 'servo-linux*' state.highstate test=True`: See what would happen
+  if we applied all configs to all servo-linux hosts
+
+The '*' is a glob which means to apply the Salt states on all of the minions;
+you can specify just one minion ID instead to only run the Salt states there.
+[More complicated targeting is also available.]
+(https://docs.saltstack.com/en/2016.3/ref/cli/index.html#using-the-salt-command)
+
+:warning: Make sure to run with `test=True` mode first to see what effect your
+changes will make before actually deploying! Read through the results carefully
+and investigate anything that looks strange.
+
+Steps to take:
+
+```console
+$ ssh root@servo-master1.servo.org
+root@servo-master1$ sudo salt '*' state.highstate test=True 2>&1 --force-color | less -R
+root@servo-master1$ sudo salt '*' state.highstate 2>&1 --force-color | tee salt_highstate_pr_number.log | less -R
+```
+
+Please make sure to log all Salt actions which perform changes to files,
+e.g. by using `tee` as in the last command line!
+Occasionally things go wrong, and having logs really helps debug,
+particularly when we have to hand off between administrators.
+
+Some deploys will require additionally manual steps, such as restarting
+Buildbot or another service; these should be recorded in a checklist on the PR,
+and checked off when completed to keep track of things and if a PR is deployed.
+
+See [the wiki](https://github.com/servo/servo/wiki/Buildbot-administration)
+for more information on how to restart Buildbot, Homu, etc. cleanly.
+
+### Reverting changes
+
+Since the Salt master will pull from the latest git master,
+the easiest way to revert a change is to `git revert` the change
+and shepard the revert through a PR + review into master.
+
+### Discouraged: Testing in production
+
+To avoid configuration drift, it's highly recommended to always use gitfs.
+However, sometimes it's necessary to test or run deploys directly on the master,
+without going through the saltfs master.
+
+You can clone the saltfs repo to `/tmp/salt-testing-root/saltfs` on the master,
+which acts as an override directory.
+See the `salt/files/master/ADMIN_README` in this repo for more information.
+
+:warning: Make sure to clean out the directly when you are done using it!
+
+
+You can also run Salt locally on a minion.
+It's easier to do this with Vagrant (use `vagrant provision` from outside
+the VM) on a local VM,
+but it's also possible to do on our actual infrastructure if needed.
+
+:memo: Make sure you're logged in as root or use sudo when doing this!
+`salt-call` needs to run as root.
+
+As always, use `test=True` mode first to verify any changes.
+This is especially helpful for debugging,
+as you can see what changes Salt will apply without actually performing them.
+Remove the `test=True` to actually run the highstate.
+
+```console
+$ salt-call state.highstate test=True 2>&1 --force-color | less -R
+$ salt-call state.highstate 2>&1 --force-color | tee salt_call_highstate_pr_number.log | less -R
+```
+
+This has the same effect as running Salt from the master,
+but Salt will additionally print out debug information as it runs each state,
+which can be helpful when debugging.
+
+The above will reach out and pull down the state tree and relevant pillar info
+from the Salt master.
+You can also use a local state and pillar tree, i.e. for testing changes.
+First, create local copies of the state tree and pillar tree.
+The location doesn't matter,
+but let's say you put them in `/tmp/salt/states` and `/tmp/salt/pillars`.
+You can make any changes you'd like to test.
+Then, to run Salt completely locally, run:
+
+```console
+$ salt-call --local --file-root=/tmp/salt/states --pillar-root=/tmp/salt/pillars state.highstate test=True 2>&1 --force-color | less -R
+```
+Change the folder paths as necessary.
+
+:warning: This is great for one-off testing because you have local versions of
+the Salt states and pillars that you can tweak without affecting other machines,
+but it can be easy to get out of sync with other machines.
+If you make any kind of permanent change,
+make sure it finds it way into the saltfs repo to make it repeatable!
+
+
+## Salt administration
+
+See [the wiki](https://github.com/servo/servo/wiki/SaltStack-Administration)
+for more information about setting up new minions and/or masters.
+
+### Looking up previous runs
 
 Salt keeps track of the results of each `job` it performs,
 where a single `job` may be running a single `state.highstate` on one minion,
@@ -38,3 +276,16 @@ root@servo-master1$ salt-run jobs.lookup_jid <jid> --out=highstate
 ```
 
 See [the Salt docs](https://docs.saltstack.com/en/2016.3/ref/runners/all/salt.runners.jobs.html#module-salt.runners.jobs) for futher documentation.
+
+### Upgrading Salt
+
+We're using a manual update process for now since there are so few machines:
+first on the salt master, then on the minions. Things to be aware of:
+
+* It's necessary to restart the salt-master and salt-minion services to apply
+  config changes, but restarting a master or minion service will interrupt an
+  ongoing highstate.
+  Make sure to run the highstate more than once to fully converge on changes.
+* Masters need to be updated before minions, but `salt '*' state.highstate`
+  cannot enforce ordering - make sure to update just the master first with
+  `salt 'servo-master1' state.highstate`.

salt/files/master/ADMIN_README

@@ -5,8 +5,7 @@ This folder takes precedence over gitfs (which reads file from the master
 branch of the saltfs repository), so it is important to follow a few guidelines:
 
 - Avoid using this folder if possible: prefer to test changes locally using the
-steps under "Making changes" at
-https://github.com/servo/servo/wiki/Buildbot-administration
+steps under "Salt workflow" in the `docs/salt.md` file.
 
 - If you do need to use this folder, use it only as long as you need it, and
 make sure to remove the cloned saltfs folder afterwards. Any files that are