build.md

Riju build system

Riju's build system is complex and takes some time to explain. Bear with me. (If you just want to add or modify a language, you can read the tutorial instead.)

Depgraph

The high level interface to Riju's build system is a tool called Depgraph, which knows about all the build artifacts and has advanced mechanisms for determining which of them need to rebuild based on content hashes. Normally you can just use Depgraph to build the artifacts you need. However, in some cases you may want to interact with the lower level for more precise operations. This is done via Makefile. (Furthermore, there are a few one-off artifacts such as the admin image which are not part of the main build system, which means that they are managed directly by Makefile.)

Available build artifacts

• image:ubuntu: A fixed revision of the upstream ubuntu:rolling image that is used as a base for all Riju images.
• image:packaging: Provides an environment to build Debian packages. Depends on image:ubuntu.
• image:runtime: Provides an environment to run the Riju server and the test suite. Depends on image:ubuntu.
• image:base: Provides a base image upon which per-language images can be derived. Depends on image:ubuntu.
• deb:lang-xyz: For each language xyz, the Debian package that installs that language into the base image. Depends on image:packaging and the build script for the language (generated from the install clause of langs/xyz.yaml).
• deb:shared-pqr: Same but for shared dependencies, which are also archived as Debian packages.
• image:lang-xyz: For each language xyz, the per-language image used for user sessions in that language. Depends on image:base, deb:lang-xyz, and possibly one or more deb:shared-pqr.
• test:lang-xyz: An artifact certifying that the xyz language tests passed. Depends on image:runtime, image:lang-xyz, the test suite and API protocol code, and the xyz language configuration.
• image:app: Built on top of image:runtime but including the Riju server code, so that it can be run standalone. Depends on image:runtime and the application code.
• deploy:ready: Deployment configuration, ready to upload. Depends on image:app and all test:lang-xyz.
• deploy:live: Pseudo-artifact corresponding to actually running the deployment, which is a blue/green cutover in which all languages and the application server are updated at once.

Depgraph abstractions

Each artifact has:

• A list of zero or more dependencies on other artifacts.
• A recipe to build it locally assuming that its dependencies are also available locally.
• A recipe to upload the local build artifact to a remote registry.
• A recipe to download the artifact from a remote registry to overwrite the local version.
• A way to compute a content-based hash of the artifact's dependencies and non-artifact inputs (data files and code). Crucially this does not require the dependencies to actually be built (the hash for each artifact is based only on its dependencies hashes), so it's possible to compute hashes for the entire dependency tree before doing anything.
• A way to check the hash currently attached to a local artifact, which can be compared to the desired hash to see if it needs to be rebuilt.
• A way to check the hash currently attached to an artifact in a remote registry, which can also be compared to the desired hash.

There are special types of artifacts:

• Manual artifacts do not have a hash until they are generated. Therefore, they must be built manually before the rest of the dependency calculations can proceed. image:ubuntu is a manual artifact since its hash depends on what we download from ubuntu:rolling.
• Publish artifacts do not have a hash after they are generated. Therefore, nothing can declare a dependency on them. deploy:live is a publish artifact. (Actually deploy:ready is a publish artifact too, but that is an implementation detail because I was lazy about my abstractions.)

Usage of Depgraph

$ dep --help
Usage: dep <target>...

Options:
  --list         list available artifacts; ignore other arguments
  --manual       operate explicitly on manual artifacts
  --hold-manual  prefer local versions of manual artifacts
  --all          do not skip unneeded intermediate artifacts
  --registry     interface with remote registry for caching
  --publish      publish artifacts to remote registries
  --yes          execute plan without confirmation
  -h, --help     display help for command

You can run dep --list to list all the available artifacts. Then dep name-of-artifact [names-of-more-artifacts...] will generate those artifacts. Depgraph is like Terraform in that it will compute a plan and then ask you to confirm before proceeding.

By default Depgraph will generate artifacts locally only, although it can be instructed to also interact with a remote registry if you've set up the infrastructure appropriately. Pass --registry to enable this mode, in which case Depgraph will download remote artifacts when appropriate versions exist in the registry. Pass --publish along with --registry to also cache generated artifacts in the remote registries. Of course --publish is required to build deploy:live.

For dealing with image:ubuntu specifically, you probably just want to fetch Riju's version (available in a public ECR repository) using make sync-ubuntu to keep in sync. However if you do want to update to the latest ubuntu:rolling, it's dep image:ubuntu --manual.

The other options (aside from --yes) are mostly not too useful. Depgraph is very sophisticated and should always compute the minimum necessary build plan based on any changes you have made. So, you don't need to worry about the details! (Except when the hashing isn't working properly. Then you cry.)

Makefile

To get a "quick" overview, run make help.

Preliminary targets

There are a couple of targets that are independent of Depgraph and need to be run just to make sure various bits of state and generated files are up to date. Depgraph and/or ci-run.bash take care of this.

• make ecr: Authenticate to ECR, needed to push and pull. The authentication only lasts for 12 hours unfortunately, although it does survive an admin shell restart.
• make all-scripts: Generate packaging scripts (build.bash and install.bash in build/{lang,shared}) from YAML configuration.
• make system: Compile setuid binary used for spinning up and tearing down user containers. This is needed early because we use real containers in the test suite.

Building Depgraph artifacts

First let's go through each of the Depgraph-enabled artifacts above. For each one, there's:

• a way to build it locally
• a way to publish the local version to a remote registry
• a way to download the remote version locally

Docker images

Generally you build a Docker image named image:foobar using make image I=foobar, you publish it with make push I=foobar, and you download it with make pull I=foobar. Pass NC=1 to make image to disable Docker cache (although this is fairly rarely useful, and in general for this to work with Depgraph we need a more sophisticated mechanism).

There are one or two exceptions to this, unfortunately:

• For language images (image:lang-foobar), it's make image I=lang L=foobar.
• For image:ubuntu, you likely don't want to "build" it yourself (meaning take the latest ubuntu:rolling from Docker Hub). You can synchronize with upstream Riju using make sync-ubuntu.

For any Docker image image:foobar, you can jump into a shell using make shell I=foobar. This has some optional arguments:

• E=1: Expose Riju ports outside the container. Most likely used as make shell I=runtime E=1 inside the admin shell.
• EE=1: Same as E=1, but expose ports on 0.0.0.0 outside the container. This is helpful if you're running on the dev server and want to be able to access the development version of Riju in your browser.
• CMD="make something": Instead of launching an interactive Bash shell inside the container, run the specified shell command (using Bash) and exit.

Riju source code and build directories are typically cross-mounted at /src inside all non-user containers, so there is generally no need to rebuild and/or restart containers when making changes.

Note that all of this section applies also to the admin and ci images, which are not otherwise involved with Depgraph (and are based directly on upstream ubuntu:rolling).

Note also that admin uses --network=host and maps a number of directories such as ~/.ssh and ~/.aws, plus the Docker socket, inside the container, so you can treat an admin shell more or less the same as your external development environment.

Debian packages

Build a language package using make pkg T=lang L=xyz (where there exists langs/xyz.yaml). Build a shared dependency package using make pkg T=shared L=pqr (where there exists shared/pqr.yaml).

This has to be done in the packaging image, and will abort otherwise. So, for short, make shell I=packaging CMD="make pkg T=lang L=xyz".

To debug package installation, you can jump into a persistent packaging shell (make shell I=packaging) and break down the process into three steps:

• make pkg-clean T=lang L=xyz: Delete and recreate packaging directories for xyz.
• make pkg-build T=lang L=xyz: Run the packaging script. Substitute build for debug to instead start a shell in the environment of the packaging script, where you can operate manually.
• make pkg-deb T=lang L=xyz: Compress the results of the packaging script into a file build/lang/xyz/riju-lang-xyz.deb. You can pass Z=(gzip|xz) to enable compression, which is disabled by default to save on time during development. Otherwise, packages are automatically recompressed before registry upload time.

Uploading a package to the registry is make upload T=lang L=xyz, and download is make download T=lang L=xyz.

Tests

You can run tests for a specific language (inside the runtime image only, otherwise it will abort) using make test L=xyz. L can also be a comma-separated list of languages. You can additionally (or instead) filter by test type, e.g. make test L=python T=lsp. Uploading and downloading test hashes is only implemented at the Depgraph layer.

Final deployment

• make deploy-config: Build the deployment configuration JSON that will be pushed to S3.
• make deploy-latest: Push it to S3.
• make deploy: Combination of the above.

Application build

We have three compiled parts of Riju:

• Frontend assets (compiled with Webpack)
• Setuid binary used for privilege deescalation (compiled with LLVM)
• Supervisor binary used on deployed images (compiled with Go tooling)

For development:

• make frontend-dev (compile frontend assets, auto recompile on change)
• make system-dev (compile setuid binary, auto recompile on change)
• make supervisor-dev (compile supervisor binary, auto recompile on change)
• make server-dev (run server, auto restart on change)
• make dev (all four of the above)

For production:

• make frontend (compile frontend assets)
• make system (compile setuid binary)
• make supervisor (compile supervisor binary)
• make build (all three of the above)
• make server (run server)

Miscellaneous utilities

• make sandbox: Bash shell emulating a user session at the command line, with many useful functions in scope for executing various commands from the language configuration YAML.
• make lsp: LSP REPL. This is not working currently as it needs to be updated for the new build system that uses per-language images.
• make dockerignore: Update .dockerignore from .gitignore.
• make tmux: Start a tmux session conveniently with variables from .env.
• make env: Load in .env environment variables in a subshell. You can also do this for your current session (though it won't affect new tmux panes) with set -a; . .env; set +a.

Infrastructure

There are wrappers for packer and terraform in the repository bin that deal with some niceties automatically. Run make packer to do an AMI build, and use terraform commands from any directory.