Meta: move deploy process to Travis #2941

domenic · 2017-08-17T17:54:54Z

Meta: move deploy process to Travis

This builds and validates the spec entirely on Travis, and then deploys
it to the WHATWG servers. Previously Travis was only running build and
validation steps, but then throwing away the results, letting custom
deploy architecture handle the actual deploy. This replaces the custom
deploy architecture with something in source control and
web-host-agnostic.

This deploy is Docker-based, for two main reasons:

It is not easy to install FreePascal 3.x onto Travis CI's default
Ubuntu configuration, which is fairly old (even with recent Trusty
updates). It is simpler to create a Docker container with a recent OS
and install FreePascal 3.x there.
Docker has a good mechanism for caching previous results and not doing
unnecessary work until the cache is invalidated. This is important as
it allows us to avoid reinstalling prerequisite packages and
recompiling Wattsi on every build. Travis CI has some caching support,
but it is not as full-featured as Docker's. The intermediate
containers created for this caching are stored on Docker Hub; see
https://hub.docker.com/r/whatwg/html-deploy.

While here, we update the Travis configuration to no longer runs Travis
on both pull requests and branches, but instead only on pull requests.

After about a day of working on this I am pretty happy with it. As you can see from e.g. https://travis-ci.org/whatwg/html/builds/265797292 build times are pretty fast (less than 2 minutes) and don't do a lot of unnecessary work.

Although the actual deploy part is untested, I am hoping that works pretty straightforwardly. Maybe I should test it by temporarily disabling the "if on a pull request, bail out" logic.

Code review appreciated, although it's fine if it's not in depth. Also I'm curious if people think deploy.sh and the Dockerfile should belong in html-build; it's not a problem to move them.

sainaen · 2017-08-17T20:55:32Z

Dockerfile

+
+# Get Wattsi and html-build; compile Wattsi and put it in the PATH.
+RUN apt-get update && \
+    apt-get install -y ca-certificates curl git unzip fp-compiler-3.0.0 && \


You probably want fp-compiler (virtual package) or fp-compiler-3.0.2 (concrete version) instead of fp-compiler-3.0.0 (outdated, spark64-only version)

That explains it...

sainaen · 2017-08-17T21:07:11Z

deploy.sh

+./html-build/build.sh
+
+# Conformance-check the result
+/usr/lib/jvm/java-8-oracle/jre/bin/java -jar vnu.jar --skip-non-html ./html-build/output


I'm not sure debian:sid image has this by default. You'd need to install java too.

Any thoughts on the right package name? Apparently we want Oracle Java 8...

sainaen · 2017-08-17T21:18:09Z

Dockerfile

+
+# Get Wattsi and html-build; compile Wattsi and put it in the PATH.
+RUN apt-get update && \
+    apt-get install -y ca-certificates curl git unzip fp-compiler oracle-java8-installer && \


Oh, I'm sorry, I might have misled you! There's no Oracle's Java package in Debian. You'll need to use PPA (or some other way) and, somehow, accept their license.

Well, happy to use not-Debian; any suggestions?

Sorry, don't know any. Oracle has a scary license with a no-distribution thing. So Ubuntu doesn't have it either (only in PPAs.)

With Ubuntu's PPA, though, it's possible:

add-apt-repository ppa:webupd8team/java && apt-get update && apt-get install oracle-java8-installer

but it will ask you to accept the license on first install.

To workaround that (not sure about legal side of this though):

echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections

source

sainaen · 2017-08-17T21:20:59Z

deploy.sh

+ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
+ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
+ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
+ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}


One last thing: I strongly suspect these env variables will not be visible from within the container, you'll need to pass them in.

Yeah, I think the dockerfile does that successfully

This builds and validates the spec entirely on Travis, and then deploys it to the WHATWG servers. Previously Travis was only running build and validation steps, but then throwing away the results, letting custom deploy architecture handle the actual deploy. This replaces the custom deploy architecture with something in source control and web-host-agnostic. This deploy is Docker-based, for two main reasons: * It is not easy to install FreePascal 3.x onto Travis CI's default Ubuntu configuration, which is fairly old (even with recent Trusty updates). It is simpler to create a Docker container with a recent OS and install FreePascal 3.x there. * Docker has a good mechanism for caching previous results and not doing unnecessary work until the cache is invalidated. This is important as it allows us to avoid reinstalling prerequisite packages and recompiling Wattsi on every build. Travis CI has some caching support, but it is not as full-featured as Docker's. The intermediate containers created for this caching are stored on Docker Hub; see https://hub.docker.com/r/whatwg/html-deploy. While here, we update the Travis configuration to no longer runs Travis on both pull requests and branches, but instead only on pull requests.

sideshowbarker · 2017-08-18T02:27:44Z

deploy.sh

+TRAVIS_PULL_REQUEST=${TRAVIS_PULL_REQUEST:-false}
+
+# Work in the parent directory
+cd ..


cd ..

Nit: If cd fails for some reason (though yeah I realize that in this particular case it almost certainly can’t…), it’s always best to instead do:

cd .. || exit

I think the errexit at the top takes care of that

annevk · 2017-08-18T07:08:56Z

It seems slightly cleaner to move the files if it's not a problem. The other thing I was wondering about is that here .travis.yml looks a little different from elsewhere. There's no ENCRYPTION_LABEL and DEPLOY_USER. Also, did you add the public key to the server already?

annevk · 2017-08-18T08:50:17Z

This ends up fixing whatwg/html-build#103 by the way, once it lands.

annevk · 2017-08-18T08:51:35Z

deploy.sh

+
+# scp to the WHATWG server
+echo "$SERVER $SERVER_PUBLIC_KEY" > known_hosts
+scp -r -o UserKnownHostsFile=known_hosts html-build/output "$DEPLOY_USER@$SERVER:$WEB_ROOT"


Should we use whatwg/html-build#99 (comment) instead? Or better as a follow-up?

Hmm, yeah, perhaps I'll try that.

domenic · 2017-08-18T14:19:02Z

There's no ENCRYPTION_LABEL

ENCRYPTION_LABEL got inlined into some of the commands in .travis.yml. There I went for brevity (one line, but with the encryption label duplicated twice in that line) over configurability.

We need to either decrypt in .travis.yml, or have two deploy.sh scripts, one for running on Travis (with access to encrypted environment variables), and one for running inside the Docker container.

Although, doing two deploy scripts is not so bad, one we move them out of the repo. I guess I'll try that.

and DEPLOY_USER.

I've hard-coded it to your username, since this is only for one spec. DEPLOY_USER is used since other specs deploy to various different users all through a single deploy script. But we're not using a shared deploy script here.

Also, did you add the public key to the server already?

Yep.

This adds a new subdirectory, ci-deploy, with scripts and other resources for building, validating, and deploying the spec entirely on Travis. Previously Travis was only running build and validation steps, but then throwing away the results, letting custom deploy architecture handle the actual deploy. This replaces the custom deploy architecture with something in source control and web-host-agnostic. This deploy is Docker-based, for two main reasons: * It is not easy to install FreePascal 3.x onto Travis CI's default Ubuntu configuration, which is fairly old (even with recent Trusty updates). It is simpler to create a Docker container with a recent OS and install FreePascal 3.x there. * Docker has a good mechanism for caching previous results and not doing unnecessary work until the cache is invalidated. This is important as it allows us to avoid reinstalling prerequisite packages and recompiling Wattsi on every build. Travis CI has some caching support, but it is not as full-featured as Docker's. The intermediate containers created for this caching are stored on Docker Hub; see https://hub.docker.com/r/whatwg/html-deploy. The ci-deploy/README.md file contains more instructions on how this is expected to be used, and will be used by whatwg/html's .travis.yml in whatwg/html#2941.

This adds a new subdirectory, ci-deploy, with scripts and other resources for building, validating, and deploying the spec entirely on Travis. Previously Travis was only running build and validation steps, but then throwing away the results, letting custom deploy architecture handle the actual deploy. This replaces the custom deploy architecture with something in source control and web-host-agnostic. This deploy is Docker-based, for two main reasons: * It is not easy to install FreePascal 3.x onto Travis CI's default Ubuntu configuration, which is fairly old (even with recent Trusty updates). It is simpler to create a Docker container with a recent OS and install FreePascal 3.x there. * Docker has a good mechanism for caching previous results and not doing unnecessary work until the cache is invalidated. This is important as it allows us to avoid reinstalling prerequisite packages and recompiling Wattsi on every build. Travis CI has some caching support, but it is not as full-featured as Docker's. The intermediate containers created for this caching are stored on Docker Hub; see https://hub.docker.com/r/whatwg/html-deploy. The ci-deploy/README.md file contains more instructions on how this is expected to be used, and will be used by whatwg/html's .travis.yml in whatwg/html#2941. Fixes #11. Fixes #99. Fixes #103.

domenic · 2017-08-18T21:09:37Z

This is ready for review again. Proposed new commit message:

Meta: move deploy process to Travis

This uses the scripts introduced in
https://github.com/whatwg/html-build/pull/117 to move our build and
deploy process to Travis CI.

While here, we update the Travis configuration to no longer runs Travis
on both pull requests and branches, but instead only on pull requests.

It needs to be merged after the corresponding html-build PR, and until that is merged Travis will fail this branch. Marking "do not merge yet" for that reason.

This adds a new subdirectory, ci-deploy, with scripts and other resources for building, validating, and deploying the spec entirely on Travis. Previously Travis was only running build and validation steps, but then throwing away the results, letting custom deploy architecture handle the actual deploy. This replaces the custom deploy architecture with something in source control and web-host-agnostic. This deploy is Docker-based, for two main reasons: * It is not easy to install FreePascal 3.x onto Travis CI's default Ubuntu configuration, which is fairly old (even with recent Trusty updates). It is simpler to create a Docker container with a recent OS and install FreePascal 3.x there. * Docker has a good mechanism for caching previous results and not doing unnecessary work until the cache is invalidated. This is important as it allows us to avoid reinstalling prerequisite packages and recompiling Wattsi on every build. Travis CI has some caching support, but it is not as full-featured as Docker's. The intermediate containers created for this caching are stored on Docker Hub; see https://hub.docker.com/r/whatwg/html-deploy. The ci-deploy/README.md file contains more instructions on how this is expected to be used, and will be used by whatwg/html's .travis.yml in whatwg/html#2941. Fixes #11. Fixes #99. Fixes #103.

domenic added do not merge yet Pull request must not be merged per rationale in comment spec tooling labels Aug 17, 2017

domenic force-pushed the move-deploy branch from 02ad259 to 88444f5 Compare August 17, 2017 18:05

sainaen reviewed Aug 17, 2017

View reviewed changes

domenic force-pushed the move-deploy branch from ef1d03a to 310643b Compare August 18, 2017 00:42

domenic removed the do not merge yet Pull request must not be merged per rationale in comment label Aug 18, 2017

domenic requested review from sideshowbarker and annevk August 18, 2017 00:42

sideshowbarker approved these changes Aug 18, 2017

View reviewed changes

annevk reviewed Aug 18, 2017

View reviewed changes

domenic mentioned this pull request Aug 18, 2017

Add scripts for deploying via Travis CI whatwg/html-build#117

Merged

domenic force-pushed the move-deploy branch from 2e50162 to 5e79f56 Compare August 18, 2017 18:29

Move most of the stuff to html-build

33d2132

domenic force-pushed the move-deploy branch from 5e79f56 to d0da918 Compare August 18, 2017 19:21

domenic force-pushed the move-deploy branch from d0da918 to 33d2132 Compare August 18, 2017 21:06

domenic added the do not merge yet Pull request must not be merged per rationale in comment label Aug 18, 2017

sideshowbarker approved these changes Aug 18, 2017

View reviewed changes

annevk merged commit a99b51e into master Aug 19, 2017

annevk deleted the move-deploy branch August 19, 2017 06:55

domenic removed the do not merge yet Pull request must not be merged per rationale in comment label Aug 19, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meta: move deploy process to Travis #2941

Meta: move deploy process to Travis #2941

domenic commented Aug 17, 2017 •

edited

Loading

sainaen Aug 17, 2017 •

edited

Loading

domenic Aug 17, 2017

sainaen Aug 17, 2017 •

edited

Loading

domenic Aug 17, 2017

sainaen Aug 17, 2017

domenic Aug 17, 2017

sainaen Aug 17, 2017

sainaen Aug 17, 2017 •

edited

Loading

sainaen Aug 17, 2017

domenic Aug 17, 2017

sainaen Aug 17, 2017

sideshowbarker Aug 18, 2017

domenic Aug 18, 2017

annevk commented Aug 18, 2017

annevk commented Aug 18, 2017

annevk Aug 18, 2017

domenic Aug 18, 2017

domenic commented Aug 18, 2017

domenic commented Aug 18, 2017 •

edited

Loading

Meta: move deploy process to Travis #2941

Meta: move deploy process to Travis #2941

Conversation

domenic commented Aug 17, 2017 • edited Loading

sainaen Aug 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sainaen Aug 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sainaen Aug 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annevk commented Aug 18, 2017

annevk commented Aug 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenic commented Aug 18, 2017

domenic commented Aug 18, 2017 • edited Loading

domenic commented Aug 17, 2017 •

edited

Loading

sainaen Aug 17, 2017 •

edited

Loading

sainaen Aug 17, 2017 •

edited

Loading

sainaen Aug 17, 2017 •

edited

Loading

domenic commented Aug 18, 2017 •

edited

Loading