[WIP] Cleaning up the www setup to match production #54

kenperkins · 2015-02-20T05:36:14Z

This is a work in progress; the plan is to get the anisble playbook up to date with production such that we can re-build an identical box in moments should the need arise.

Figure out how to setup dist and staging users
Figure out how to setup crontab jobs
deploy dist-indexer and npm install in /home/dist/dist-indexer
get promote_nightly.sh into /home/staging/promote/
get update-download-stats.sh into /home/iojs

Current Changes

Detected if dhparam.pem exists, saving a boatload of time
adding a bunch of paths into the nginx config from /home/dist
Added @kenperkins as an authorized_keys member
reworked authorized_keys setup to be a bit cleaner
added dist, staging users with ssh keys
added crontab jobs for promote_nightly.sh and update-download-stats.sh

Open Questions

What is /root/mkdldtxt.sh?
What is /root/countdldtxt.sh
Who is responsible to create /home/dist/public and /home/dist/public-test directories?
What is /home/dist/dist-indexer/ls-types.js?
What is /home/dist/dist-indexer/transform-filename.js?
What are all of the files in /home/staging/promote?
Who creates /home/staging/staging directory?

- Detected if dhparam.pem exists, saving a boatload of time - adding a bunch of paths into the app from /home/dist

- Also sets up authorized_keys for staging, dist - Includes setting up the id_rsa and id_rsa.pub keys in .ssh

setup/www/ansible-playbook.yaml

+      with_items:
+        - "{{ server_user }}"
+        - "{{ staging_user }}"
+        - "{{ dist_user }}"


setup/www/ansible-playbook.yaml

+
+    - name: Configure | setup cron for download-stats
+      cron: name="update_download_stats" minute="30" hour="1" job="/home/iojs/update-download-stats.sh" user="root"
+      tags: setup


kenperkins · 2015-02-20T23:40:31Z

This is getting close, but I'm unable to find promote_nightly.sh in source control. @rvagg are you able to assist on this? should I just take it from the current production webserver?

kenperkins · 2015-02-21T00:46:30Z

Additional open questions:

What is /root/mkdldtxt.sh?
What is /root/countdldtxt.sh
Who is responsible to create /home/dist/public and /home/dist/public-test directories?
What is /home/dist/dist-indexer/ls-types.js?
What is /home/dist/dist-indexer/transform-filename.js?
What are all of the files in /home/staging/promote?
Who creates /home/staging/staging directory?

kenperkins · 2015-03-18T20:53:23Z

Reminder @rvagg that I'm blocked on finishing this without some input.

rvagg · 2015-04-15T05:47:13Z

Really sorry @kenperkins, I'm a terrible bottle-neck for these things, way too overloaded. Trying to get on top of some of these things now.

What is /root/mkdldtxt.sh?

An initial version of what's in /home/iojs/update-download-stats.sh, it's redundant now. It only ever was meant to be a temporary "let's see what these numbers look like" thing but that script is now run via cron and exposed via https://iojs.org/download-stats.json because ppl asked for it.

What is /root/countdldtxt.sh

Part of the above, redundant now too. There were two scripts to do the job that one is now doing.

Who is responsible to create /home/dist/public and /home/dist/public-test directories?

The promote script(s) are responsible for /home/dist/public/. There is a mkdir -p in there but the root directories should be created by the script I think since they are served by nginx. Currently there needs to be next-nightly, nightly, release (aliased ad https://iojs.org/dist/) and there is also a test directory but that's not controlled by any tool, it's been a place to share some test builds, manually, so unimportant.

What is /home/dist/dist-indexer/ls-types.js?

What is /home/dist/dist-indexer/transform-filename.js?

Parts of dist-indexer https://github.com/iojs/build/tree/master/tools/dist/dist-indexer, but also:

There seems to be an incomplete refactor going on there and I'm wondering if a commit or two got lost somewhere, transform-filename.js should be used by dist-indexer.js as well as ls-types.js but it's only used by the latter with some duplicated code in dist-indexer.js.
ls-types.js is used by the scripts in /usr/local/bin, you'll see there are 3 dist-* scripts there now and they are used for the release process so that authorised release people can run (tools/release.sh](https://github.com/iojs/io.js/blob/v1.x/tools/release.sh) to promote and sign a release without needing to log in to the server. Those files need to become part of a scripted setup of the server too.

What are all of the files in /home/staging/promote?

I've put them in a PR including a README, see #76 and note that they are also invoked by scripts in /usr/local/bin which also need to be part of this setup

Who creates /home/staging/staging directory?

It should be created by server setup. Each of the release build slaves has a jenkins config which makes it ssh in to the server as staging and mkdir -p and then scp in the build artifacts, but as with /home/dist/ we should probably set the basics of it up including the next-nightly, nightly and release subdirectories.

kenperkins · 2015-04-15T15:16:48Z

Wow, it's like Christmas morning! Thanks @rvagg I'll try to resurrect this as soon as I can find some engineering time.

jbergstroem · 2016-05-03T05:53:15Z

This unfortunately looks stale. Going to close it but feel free to reopen with a rebased version seeing how a lot of the changes has gone in one way or the other.

kenperkins added 4 commits February 19, 2015 21:32

Cleaning up the www setup to match production

0397be3

- Detected if dhparam.pem exists, saving a boatload of time - adding a bunch of paths into the app from /home/dist

cleaning up name on the ssl dir task

d7755aa

Fixing formatting on iojs.org nginx conf

2db3516

Adding dist, staging users to ansible config

35a2861

- Also sets up authorized_keys for staging, dist - Includes setting up the id_rsa and id_rsa.pub keys in .ssh

kenperkins reviewed Feb 20, 2015
View reviewed changes

setup/www/ansible-playbook.yaml

with_items:

- "{{ server_user }}"

- "{{ staging_user }}"

- "{{ dist_user }}"

This comment was marked as off-topic.

Sign in to view

kenperkins added 2 commits February 20, 2015 14:24

Cleaning up README for www setup

c5d394b

Adding crontab jobs

7297c9f

kenperkins reviewed Feb 20, 2015
View reviewed changes

Adding update-download-stats to the prod webserver

03a089f

Adding support to deploy dist-indexer to the webserver

897eb32

kenperkins added enhancement infra website labels Feb 26, 2015

rvagg mentioned this pull request Apr 15, 2015

State of the build (io.js) April 2015 #77

Closed

mhdawson force-pushed the master branch from 2ae729a to 8fce5b9 Compare July 21, 2015 22:17

jbergstroem closed this May 3, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Cleaning up the www setup to match production #54

[WIP] Cleaning up the www setup to match production #54

kenperkins commented Feb 20, 2015

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

kenperkins commented Feb 20, 2015

kenperkins commented Feb 21, 2015

kenperkins commented Mar 18, 2015

rvagg commented Apr 15, 2015

kenperkins commented Apr 15, 2015

jbergstroem commented May 3, 2016

[WIP] Cleaning up the www setup to match production #54

[WIP] Cleaning up the www setup to match production #54

Conversation

kenperkins commented Feb 20, 2015

Current Changes

Open Questions

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

kenperkins commented Feb 20, 2015

kenperkins commented Feb 21, 2015

kenperkins commented Mar 18, 2015

rvagg commented Apr 15, 2015

What is /root/mkdldtxt.sh?

What is /root/countdldtxt.sh

Who is responsible to create /home/dist/public and /home/dist/public-test directories?

What is /home/dist/dist-indexer/ls-types.js?

What is /home/dist/dist-indexer/transform-filename.js?

What are all of the files in /home/staging/promote?

Who creates /home/staging/staging directory?

kenperkins commented Apr 15, 2015

jbergstroem commented May 3, 2016