Move buildmaster from Linode to AWS (tracking issue) #281

edunham · 2016-03-28T17:40:32Z

Async steps

At 9am PST Wednesday 3/30

sftp /home/servo/homu/main.db from old master to new; restart homu
In the Cloudflare interface, edit build.servo.org record to point at new master. All webhooks use DNS, so they will not need to be modified.
Retry a failed build to verify that everything's working

aneeshusa · 2016-03-28T19:29:29Z

We actually don't have instructions for setting up a Salt master yet, so I'll need to write those up and add them to the wiki (e.g. installing the right version of the right package).

Also, if you're trying to do this as a multimaster transition (which I recommend) then you'll need to first list both masters in the minion configuration file, then after verifying the new master is correctly set up removing the old one. Note: if you're going to give both masters servo-master as an ID, don't have them connect to each other or they will get confused by seeing two minions with the same ID. (This affects the copying the /etc/salt/pki/master step).

metajack · 2016-03-28T19:31:18Z

The private google doc should have some possibly out of date instructions for setting up a salt master. I think it was mostly bootstrappable except for some minor things. Thought these days, we probably have things like minion keys that need to be dealt with. When I wrote the instructions I was doing a totally fresh master :)

aneeshusa · 2016-03-28T19:56:32Z

I'd be interested in taking a peek at that doc purely out of curiosity :)

As for the DNS, the fact that Homu and nginx are running on the same box as the Salt master is just an artifact of our environment, and we shouldn't bake it into our configuration files. We should add a separate DNS entry for salt.servo.org (pointing at the new master, via A record not CNAME). In the case of multimaster, this lets us set the minion configuration to say:

master:
  - build.servo.org
  - salt.servo.org

right after we set up the new master. After restarting the minions and checking connectivity to the new master, we can update it to say simply:

master: salt.servo.org

This lets us decouple the DNS changes for Salt from the DNS changes for Homu, when we switch over the build.servo.org DNS record.

What's the timezone for the 9am switchover? I may or may not be awake that early.

larsbergstrom · 2016-03-28T20:04:15Z

I think it was originally a private etherpad, and those servers have since been moved behind a firewall and then burned down (ether-pocalypse).

It's 9AM US Pacific time :-)


        Auto merge of #282 - aneeshusa:remove-longview, r=edunham

Remove longview setup for servo-master Linode exodus Longview is Linode's proprietary monitoring service. We are moving the servo-master machine from Linode to EC2, so we can and will no longer use Longview, so this commit removes it. Refs #281  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/282)

edunham · 2016-03-28T22:10:52Z

@aneeshusa Thanks for the feedback!

I just checked the secrets doc and it does not currently contain any setup instructions for the build master.

How much of the master setup can we do through Salt itself?

Does my amended checklist for doing a dual-master switchover look sane?

aneeshusa · 2016-03-29T00:19:55Z

Master setup:

For now, I just plan to create a script to install the correct salt-master package. Setting up the configuration file and start the corresponding service can be manual for now, as it is for setting up a new minion (although Salting this is on my todo list).

More complicated setup is IMO not worth automating right now; e.g., keeping the list of accepted minion keys in sync would want some kind of CMDB (even if it's just flat files), and ditto for keeping the pillars synced. (GitFS should keep the file tree synced).

Checklist:

We should explicitly separate setting up a Salt minion on the new machine from setting up a Salt master on the new machine on the checklist (since we haven't Salted the process of setting up a Salt master yet). It looks like the checklist already does this, but I'd like to be more clear for that section:

Create new EC2 instance, assign it an elastic IP
Install a Salt minion using the instructions from https://github.com/servo/servo/wiki/Buildbot-administration#linux and run a local highstate (see my later comments about the ID)
Install the Salt master package with instructions from the wiki (TODO: write these instructions)
[various master configuration steps - pillar data, file tree, PKI data, master config file?]
Start the salt-master service on the new machine
Point minions at both machines, etc.

We also need to add some additional steps to the switchover. Earlier, I was thinking to reuse the servo-master ID on the new machine, but on further reflection I think this will be more trouble than it's worth (as I mentioned in an earlier comment, we'd need to add steps to special-case the minion configs, edit the PKI files, etc.). Instead, we should be using separate IDs for each machine; this will require some more steps as well. I'm writing this up in more detail and will leave another comment.

Also, I have a busy week, so any chance of pushing the switchover date back a few days would be appreciated!

aneeshusa · 2016-03-29T11:15:36Z

Salt minion IDs are meant to be unique, so we should follow the convention instead of trying to reuse the servo-master ID, as it will cause trouble when running with two hot masters. Here's my new thinking:

New steps (we should do these before bringing up the new master):

Amend the top file so that the servo-master section is amended to servo-master\d+ in order to handle multiple masters.
- We should also update the .travis.yml file to use servo-master1
Switch the current master ID from servo-master to servo-master1. Steps to take on the current master:
- Edit the minion ID in the /etc/salt/minion file (removing the /etc/salt/minion_id file if necessary)
- Go into the /etc/salt/pki/master/minions directory and rename the servo-master file to servo-master1
- Restart the Salt minion, and once it comes back up, confirm connectivity with test.ping and targeting with a highstate in test=True mode

At this point, we can proceed with setting up the new machine, and assigning it a Salt minion ID of servo-master2.

Also, I don't know what I was thinking yesterday with the DNS records, but here's my updated recommendations after some sleep:

Salt minion IDs should match the local hostname on each machine (we should Salt this/add to setup instructions on wiki at a later time)
Each machine should also get a corresponding A record, e.g.. A records for servo-master1.servo.org and servo-master2.servo.org pointing at the old and new masters.
Use CNAME records for functional identification, i.e. build.servo.org.

The goal is that for a given machine, the Salt minion ID == the hostname for the machine == the DNS A record for that machine, and that this is an immutable identifier for that machine. Separately, when we point at a given hostname for functional reasons (i.e. the Homu webhooks), we should use a CNAME record to point to the appropriate machine, because the particular machine that is responsible for these settings could change at any time and the application should not know.

Updated minion config settings:

master:
  - servo-master1.servo.org
  - servo-master2.servo.org

After the switchover:

master:
  - servo-master2.servo.org

Since we'd like the minions to connect to all the masters and not just one, we can use the immutable names directly and we don't need a salt.servo.org CNAME record anymore. This also will make it easier to go back to multimaster mode in the future if needed.

We may also want to consider adding master_alive_interval: 30 to our minion configs: https://docs.saltstack.com/en/2015.5/ref/configuration/minion.html#master-alive-interval, although this isn't necessary.


        Auto merge of #284 - aneeshusa:handle-multiple-masters, r=edunham

Handle multiple (redundant) masters Update the Salt states to handle multiple minions that host masters. This will allow us to easily enter redundant multimaster mode to handle switching over our master from Linode to EC2, by using separate IDs for each machine instead of trying to reuse the `servo-master` ID. See #281 (comment) for more details. I haven't updated the `common/map.jinja` file yet; are we still using these hostnames in the `/etc/hosts` file or is everything happening via DNS lookups?  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/284)

edunham · 2016-03-29T20:25:56Z

@aneeshusa The production salt setup is now dual-master. They're servo-master0 (the old linode host) and servo-master1 (the new AWS host). I've sftp'd all the secrets and master configs over, and they both yield identical ping and highstate results except that servo-master1 isn't aware of a few decommissioned hosts like servo-head.

I believe this covers our bases for a smooth transition on the saltfs side of things, and now the buildmaster+homu+DNS move is all that's left to worry about.

I've amended all the minions' /etc/salt/minion files to contain

master:
  - servo-master0.servo.org
  - servo-master1.servo.org

and they appear to be connecting to both buildmasters successfully at the same time.

root@servo-master1:~# salt-run manage.status
down:
up:
    - servo-linux-cross1
    - servo-linux-cross2
    - servo-linux1
    - servo-linux2
    - servo-mac1
    - servo-mac2
    - servo-mac3
    - servo-macpro1
    - servo-master0
    - servo-master1

root@servo-master0# salt-run manage.status
down:
    - servo-head
    - servo-linux-android1
    - servo-master
up:
    - servo-linux-cross1
    - servo-linux-cross2
    - servo-linux1
    - servo-linux2
    - servo-mac1
    - servo-mac2
    - servo-mac3
    - servo-macpro1
    - servo-master0
    - servo-master1

edunham · 2016-03-30T16:32:48Z

Everything (http://servo-master1.servo.org/homu/queue/servo == http://build.servo.org/homu/queue/servo, http://servo-master1.servo.org/buildslaves == http://build.servo.org/buildslaves) appears to be working correctly. servo/servo#10285 shows that webhooks are working.

edunham · 2016-03-30T16:48:18Z

The only surprise that I ran into was that in /srv/pillar/buildbot.sls, master was set to servo-master, which had an IP hardcoded in the hosts file from https://github.com/servo/saltfs/blob/master/common/map.jinja . Changing that setting to master: build.servo.org and re-running the highstate causes the slaves to look up the buildmaster with DNS rather than directly, and solved the problem.

@aneeshusa, I read a bit about managing Salt with Salt and while I think it's a good idea to manage the salt config files, it looks like it introduces a variety of complications. If you'd like to invest the time into making it work, I'd happily help get a PR through, but for now I think that making this type of major changes to the Salt setup is such an infrequent occurrence that our effort is probably better focused on more commonly used parts of the system.

edunham · 2016-03-30T16:55:18Z

We're shutting down servo-master0 (the Linode host) at 5pm PST today. If you need logs from a build that happened overnight before then, substitute servo-master0.servo.org for build.servo.org in the URL to access them. After servo-master0 has been shut down, you'll have to rerun a build to get its results.

aneeshusa · 2016-03-30T17:04:25Z

👏

I did run a grep -r 'servo-master' against the saltfs repo, but didn't think to check the pillars. We should remember to check the pillars for next time.

Does buildbot properly respect DNS TTLs/will it retry and reconnect if we change the build.servo.org DNS entry in the future?

We still list servo-master with the old IP on the wiki - can we update that?

Agreed on the Salt changes for now.

edunham · 2016-03-30T17:09:44Z

I don't know the details of how Buildbot handles TTLs, but kicking the buildslave processes seemed to cause them to repeat the lookup. So Buildbot itself probably isn't caching too hard.

Wiki updated. I'll open an issue to discuss migrating away from hardcoded IPs in common/map.jinja entirely.

aneeshusa · 2016-03-30T17:11:17Z

Kicking buildbot should be good enough in that case.

Already opened an issue for common/map.jinja - see #287.

aneeshusa mentioned this issue Mar 28, 2016

Remove longview setup for servo-master Linode exodus #282

Merged

aneeshusa mentioned this issue Mar 29, 2016

Handle multiple (redundant) masters #284

Merged

edunham mentioned this issue Mar 30, 2016

Use serde in devtools servo/servo#10285

Merged

larsbergstrom mentioned this issue Mar 30, 2016

Include square brackets for IPv6 addresses in Document::domain. servo/servo#10287

Merged

edunham closed this Mar 30, 2016

This was referenced Mar 30, 2016

Remove hardcoded IPs #288

Closed

Tracking Issue: Salt-In-Salt #296

Open

servo / saltfs

Move buildmaster from Linode to AWS (tracking issue) #281

Move buildmaster from Linode to AWS (tracking issue) #281

edunham commented Mar 28, 2016

aneeshusa commented Mar 28, 2016

metajack commented Mar 28, 2016

aneeshusa commented Mar 28, 2016

larsbergstrom commented Mar 28, 2016

edunham commented Mar 28, 2016

aneeshusa commented Mar 29, 2016

aneeshusa commented Mar 29, 2016

edunham commented Mar 29, 2016

edunham commented Mar 30, 2016

edunham commented Mar 30, 2016

edunham commented Mar 30, 2016

aneeshusa commented Mar 30, 2016

edunham commented Mar 30, 2016

aneeshusa commented Mar 30, 2016

servo / saltfs

Join GitHub today

Move buildmaster from Linode to AWS (tracking issue) #281

Move buildmaster from Linode to AWS (tracking issue) #281

Comments

edunham commented Mar 28, 2016

Async steps

At 9am PST Wednesday 3/30

aneeshusa commented Mar 28, 2016

metajack commented Mar 28, 2016

aneeshusa commented Mar 28, 2016

larsbergstrom commented Mar 28, 2016

edunham commented Mar 28, 2016

aneeshusa commented Mar 29, 2016

aneeshusa commented Mar 29, 2016

edunham commented Mar 29, 2016

edunham commented Mar 30, 2016

edunham commented Mar 30, 2016

edunham commented Mar 30, 2016

aneeshusa commented Mar 30, 2016

edunham commented Mar 30, 2016

aneeshusa commented Mar 30, 2016