ContinuousIntegration

Continuous Integration

Keep testing. Keep shipping. Catch bugs early.

Problem

Game assets tend to have lots of dependencies across various disciplines. Projects can easily break when people change things without realizing which other things depend on them. The longer it takes to notice the problem, the longer it takes to identify and fix the change that caused it.

Furthermore, at any given time there's a slightly different work-in-progress version of the project on each developer's machine, making it difficult to reproduce issues seen on another machine. Some issues might even manifest only on the target platform.

¯\_(ツ)_/¯
It works on
my machine.

Solution

We want to reduce the time between making a change and seeing it on the target platform as much as possible. We want to have checks that tell us when something broke. We want to run those checks after every change. That means we have to automate the entire process.

We achieve continuous integration and delivery by

1. maintaining a single repository,
2. automating the build process,
3. automating the testing process, and
4. automating the shipping process.

Maintaining a single repository

If you're already using version control you can skip ahead. Keep using what works for you. If you're not using version control yet, currently your best options are GitHub Enterprise and Plastic SCM. Both cost money, but not too much. They are well worth it. Stay away from Unity Collaborate. It doesn't work properly and its feature set is severly limited. It might be useful for game jams, but that's about it.

Automating the build process

If you're using GitHub Enterprise, you might want to give GitHub Actions a try. Don't use Unity Cloud Build, it is too basic for anything serious. We'll be using Jenkins here, which is free, open source, and runs locally.

Jenkins

We connect Jenkins to the repository and tell it to build the project every time a change gets committed. That's a lot of builds, so it's best to install Jenkins on an unused machine, which we will also use as our build server.

After installing Jenkins and logging in to http://localhost:8080 with the initial admin password, Jenkins will ask which plugins to install. We're going to "Install suggested plugins". We won't need most of them, but it's too tedious to pick plugins manually. Speaking of plugins, everything in Jenkins is a plugin, most of them contributed by volunteers. Most of them are not production ready, which is why there often are several plugins for the same task, but none of them does what we need.

Build job

To make Jenkins build the game, we need to wrap the build process in a Jenkins job. We're going to bypass all the plugins and keep things as simple as possible by using the "Pipeline" template.

The job configuration in Jenkins is fairly minimal. All we need Jenkins to do is to download the pipeline script from our repository and then execute it. We'll put all the details in the script.

In our pipeline script, we

1. update to the latest revision on the repository,
2. start Unity and let it import assets,
3. run the unit tests,
4. make a build, and finally
5. upload the build.

  environment {
    GIT = '"C:\\Program Files\\Git\\bin\\git.exe"'
    UNITY = '"C:\\Program Files\\Unity\\Hub\\Editor\\2018.4.14f1\\Editor\\Unity.exe"'
    PROJECT = 'ContinuousIntegration'
    PLATFORM = 'Win64'
    OUTPUT = 'Build/ContinuousIntegration.exe'
    STEAMCMD = '"W:\\Jenkins\\steamworks\\tools\\ContentBuilder\\builder\\steamcmd.exe"'
    STEAMUSERNAME = 'steam_username_here'
    STEAMPASSWORD = 'steam_password_here'
    STEAMSCRIPT = '"ContinuousIntegration\\BuildScripts\\Steam\\app_build.vdf"'
  }
  stage('Import Assets') {
    steps {
      bat "$UNITY -batchmode -logFile - -projectPath $PROJECT -buildTarget $PLATFORM -quit -accept-apiupdate"
    }
  }
  stage('Run Unit Tests') {
    steps {
      bat "$UNITY -batchmode -logFile - -projectPath $PROJECT -buildTarget $PLATFORM -runEditorTests"
    }
  }
  stage('Build') {
    steps {
      bat "$UNITY -batchmode -logFile - -projectPath $PROJECT -buildTarget $PLATFORM -quit -buildWindows64Player $OUTPUT"
    }
  }

Automatic the testing process

Now that we're running the unit tests before each build, it is a good idea to cover as many potential problems as possible using unit tests. We start with a basic set of consistency checks.

        [Test]
        public void ShaderHasErrors()
        {
            var infos = ShaderUtil.GetAllShaderInfo();
            foreach (var info in infos)
            {
                Assert.IsFalse(info.hasErrors, "Shader '{0}' has errors.", info.name);
            }
        }

It takes time to develop a comprehensive test suite. As a guideline, it's a good idea to write a test each time a new issue appears in the build. The goal is to write the new test in such a way that it prevents that issue from ever happening again.

Automating the shipping process

We now have a build that passed some basic checks, but that still doesn't mean much. What we need is real people testing it.

It does not exist until you ship it. --Jonas Bötel

The good news is that uploading builds to Steam is reasonably straightforward and well documented.

    stage('Upload Build') {
      steps {
        echo "$STEAMCMD +login $STEAMUSERNAME $STEAMPASSWORD +run_app_build $STEAMSCRIPT"
      }
    }

Steam keeps all uploaded builds around forever, which is super useful when something goes wrong and we have to revert to an earlier build. On the other hand, we don't really want to risk breaking the game every day. So let's make sure to automatically upload our builds to a beta branch on Steam, by configuring the setlive field in the app_build.vdf. That way our players can opt in to play the latest and greatest builds if they feel brave. Every few weeks, when the build is stable enough, we can release it to the default branch using Steam's backend.

Further reading

• Continuous Integration by Martin Fowler
• Continuous integration and automated testing by Michele Krüger
• Unite 2015 - Continuous Integration with Unity by Jonathan Peppers
• Setting Up a Build Server for Unity with Jenkins by William Chyr

Translations

• 台灣繁體中文 (zh-TW)

If you find this workshop useful and speak another language, I'd very much appreciate any help translating the chapters. Clone the repository, add a localized copy of the README.md, for example README-pt-BR.md, and send me a pull request.