Add scripts for deploying via Travis CI

This adds a new subdirectory, ci-deploy, with scripts and other
resources for building, validating, and deploying the spec entirely on
Travis. Previously Travis was only running build and validation steps,
but then throwing away the results, letting custom deploy architecture
handle the actual deploy. This replaces the custom deploy architecture
with something in source control and web-host-agnostic.

This deploy is Docker-based, for two main reasons:

* It is not easy to install FreePascal 3.x onto Travis CI's default
  Ubuntu configuration, which is fairly old (even with recent Trusty
  updates). It is simpler to create a Docker container with a recent OS
  and install FreePascal 3.x there.
* Docker has a good mechanism for caching previous results and not doing
  unnecessary work until the cache is invalidated. This is important as
  it allows us to avoid reinstalling prerequisite packages and
  recompiling Wattsi on every build. Travis CI has some caching support,
  but it is not as full-featured as Docker's. The intermediate
  containers created for this caching are stored on Docker Hub; see
  https://hub.docker.com/r/whatwg/html-deploy.

The ci-deploy/README.md file contains more instructions on how this is
expected to be used, and will be used by whatwg/html's .travis.yml in
whatwg/html#2941.

.travis.yml

@@ -8,5 +8,5 @@ addons:
       - shellcheck
 
 script:
-  - shellcheck build.sh
-  - shellcheck lint.sh
+  - shellcheck *.sh
+  - shellcheck ci-deploy/*.sh

ci-deploy/.dockerignore

@@ -0,0 +1,3 @@
+# Git checkout metadata changes every time you check out. If we left these directories there, Docker's caches (based on
+# comparing the contents of the files ADDed) would be invalidated.
+**/.git

ci-deploy/Dockerfile

@@ -0,0 +1,25 @@
+# This Dockerfile is just used to run on Travis CI in an environment that can easily and repeatedly
+# install our build dependencies.
+FROM debian:sid
+
+RUN apt-get update && \
+    apt-get install -y ca-certificates curl git unzip fp-compiler default-jre
+
+ADD wattsi /whatwg/wattsi
+
+RUN cd /whatwg/wattsi && \
+    /whatwg/wattsi/build.sh
+ENV PATH="/whatwg/wattsi/bin:${PATH}"
+
+ADD html-build /whatwg/html-build
+
+# Note: we do not ADD /whatwg/html, but instead mount it using --volume in .travis.yml, since it
+# contains the deploy_key, and thus should not be part of the image. The image is cached, publicly,
+# on Docker Hub.
+ENV HTML_SOURCE /whatwg/html
+
+ARG travis_pull_request
+ENV TRAVIS_PULL_REQUEST=${travis_pull_request}
+
+ENV SKIP_BUILD_UPDATE_CHECK=true
+ENTRYPOINT ["bash", "/whatwg/html-build/ci-deploy/inside-container.sh"]

ci-deploy/README.md

@@ -0,0 +1,10 @@
+# HTML Standard CI Deploy
+
+This directory contains files used specifically for deploying the HTML Standard on Travis CI. They are not generally relevant to local builds.
+
+The setup is assumed to be a directory containing:
+
+- A subdirectory `html-build` containing the contents of this entire [whatwg/html-build](https://github.com/whatwg/html-build) repository
+- A subdirectory `html` containing the contents of the [whatwg/html](https://github.com/whatwg/html) repository
+
+Then, run the `html-build/ci-deploy/outside-container.sh` script. What it does is documented via inline comments; check it out to learn more.

ci-deploy/inside-container.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+set -o errexit
+set -o nounset
+set -o pipefail
+cd "$(dirname "$0")/../.."
+
+WEB_ROOT="html.spec.whatwg.org"
+DEPLOY_USER="annevankesteren"
+
+SERVER="75.119.197.251"
+SERVER_PUBLIC_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP7zWfhJdjre9BHhfOtN52v6kIaDM/1kEJV4HqinvLP2hzworwNBmTtAlIMS2JJzSiE+9WcvSbSqmw7FKmNVGtvCd/CNJJkdAOEzYFBntYLf4cwNozCRmRI0O0awTaekIm03pzLO+iJm0+xmdCjIJNDW1v8B7SwXR9t4ElYNfhYD4HAT+aP+qs6CquBbOPfVdPgQMar6iDocAOQuBFBaUHJxPGMAG0qkVRJSwS4gi8VIXNbFrLCCXnwDC4REN05J7q7w90/8/Xjt0q+im2sBUxoXcHAl38ZkHeFJry/He2CiCc8YPoOAWmM8Vd0Ukc4SYZ99UfW/bxDroLHobLQ9Eh"
+
+cd html
+SHA=$(git rev-parse HEAD)
+cd ..
+
+export HTML_OUTPUT="output"
+
+# Environment variables set from outside
+TRAVIS_PULL_REQUEST=${TRAVIS_PULL_REQUEST:-false}
+
+# Build the spec into the output directory
+./html-build/build.sh
+
+# Conformance-check the result
+echo "Downloading and running conformance checker..."
+curl --remote-name --fail https://sideshowbarker.net/nightlies/jar/vnu.jar
+java -jar vnu.jar --skip-non-html $HTML_OUTPUT
+
+mkdir "$HTML_OUTPUT/commit-snapshots"
+cp "$HTML_OUTPUT/index.html" "$HTML_OUTPUT/commit-snapshots/$SHA"
+
+# Note: $TRAVIS_PULL_REQUEST is either a number or false, not true or false.
+# https://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables
+if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then
+  echo "Skipping deploy for non-master"
+  exit 0
+fi
+
+echo ""
+find "$HTML_OUTPUT" -type f -print
+echo ""
+
+chmod 600 html/deploy-key
+eval "$(ssh-agent -s)"
+ssh-add html/deploy-key
+
+# scp to the WHATWG server
+echo "$SERVER $SERVER_PUBLIC_KEY" > known_hosts
+scp -r -o UserKnownHostsFile=known_hosts "$HTML_OUTPUT" "$DEPLOY_USER@$SERVER:$WEB_ROOT"

ci-deploy/outside-container.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+set -o errexit
+set -o nounset
+set -o pipefail
+
+HERE=$(dirname "$0")
+cd "$HERE/../.."
+
+DOCKER_USERNAME="domenicdenicola"
+DOCKER_HUB_REPO="whatwg/html-deploy"
+# DOCKER_PASSWORD is set from the outside
+# TRAVIS_PULL_REQUEST is set from the outside
+# ENCRYPTION_LABEL is set from the outside
+
+git clone https://github.com/whatwg/wattsi.git wattsi
+
+# Copy the Docker-related stuff into the working (grandparent) directory.
+cp "$HERE"/{.dockerignore,Dockerfile} .
+
+# Build the Docker image, using Docker Hub as a cache. (This will be fast if nothing has changed
+# in wattsi or html-build).
+docker pull "$DOCKER_HUB_REPO:latest"
+docker build --cache-from "$DOCKER_HUB_REPO:latest" \
+             --tag "$DOCKER_HUB_REPO:latest" \
+             --build-arg "travis_pull_request=$TRAVIS_PULL_REQUEST" \
+             .
+
+# Decrypt the deploy key from this script's location into the html/ directory, since that's the
+# directory that will be shared with the container (but not built into the image).
+ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
+ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
+ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
+ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}
+openssl aes-256-cbc -K "$ENCRYPTED_KEY" -iv "$ENCRYPTED_IV" \
+        -in "$HERE/deploy-key.enc" -out html/deploy-key -d
+
+# Run the inside-container.sh script, with the html/ directory mounted inside the container.
+docker run --volume "$(pwd)/html":/whatwg/html "$DOCKER_HUB_REPO:latest"
+
+# If the build succeeded and we got here, upload the Docker image to Docker Hub, so that future runs
+# can use it as a cache.
+docker tag "$DOCKER_HUB_REPO:latest" "$DOCKER_HUB_REPO:$TRAVIS_BUILD_NUMBER" &&
+docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD"
+docker push "$DOCKER_HUB_REPO"

test.txt

@@ -0,0 +1 @@
+93