Jenkins CI Automation for Unreal Engine 4 Projects

This tutorial and repository provides documentation and resources on how to configure a Jenkins CI installation to build and compile an Unreal Engine 4 project.

This fork is modified mainly in these aspects:

• Step1_DeleteAutoGenerated.bat: This script will remove auto generated files from a UE4 Project. It's recommended to run this script to prevent potential errors, specially if your project is heavily made in C++.
• Step3_CompileScripts_Win64.bat: Now it uses the Unreal Batch Builder instead of MSBuild.exe from Visual Studio.
• Step4_RunTests.bat: Script for running functional tests using the Unreal Automation System. Go to "considerations" section for extra arguments (i.e. server/client arguments).
• RunJenkinsAsDesktopMode.bat: Script that needs to be placed in %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup Runs Jenkins in desktop mode (default mode is as a service) so it can interact with UE4Editor.exe. Needed for running the tests.
• /LogParsers/: Include 3 LogParsers files (compilation log parser, test log parser and build log parser). Used with the Log Parser Plugin in Jenkins for highlighting/parsing errors, warnings and relevant info. These logs are optional except for the "test log parser" which is needed for determining if a tests failed. The configuration of these files will determine when a tests fails, succeeds or is marked as unstable.

Created by Zack Devine for Skymap Games.

Modified by Kevin Yabar for Cronofear Softworks.

Considerations

• You don't need to do Step2 and Step3 if your project is Blueprints only.
• How automated tests should look https://www.youtube.com/watch?v=ivLFP2eApto
• How to make functional automated tests in UE4: https://www.youtube.com/watch?v=f4LpDXjFgVQ you can also use C++, but UE4 tests in C++ is awful (no hotreload support).
• How to make better automated tests (high level scripting) in UE4: http://forum.skookumscript.com/t/about-automated-tests-using-sk/1510 This uses a plugin for using the SkookumScript programming language in UE4. This language is really good for gameplay programming too. Check why this tool is awesome here: http://error454.com/2017/03/09/death/tick
• How to connect the builds from Jenkins to the releases manager in Visual Studio Team Services (upload .zip files and i think, log results, to VSTS): https://www.youtube.com/watch?v=rCZ5ICkwarQ
• How to generate custom poll scripts for Jenkins: https://crontab.guru/
• Sometimes for some reason, after Jenkins build your projects, if you try to generate files in any UE4 project. (Right click, Generate VS File) there will be an error saying that you don't have enough permissions. Also, if you open a .sln project and try to build a UE4 project, VS wil shot the same error. It's annoying and i didn't found a perfect solution. But you can do the following to solve this problem:
  • Always run .sln as administrator: https://stackoverflow.com/questions/22672072/how-do-i-add-a-default-run-as-administrator-when-i-open-a-sln-file-from-the-comm check the answer from "Prasanth Louis"
  • Execute "Step1_StartBuild.bat" as admin in any UE4 project. This will "unlock" the permissions and you'll be able to generate VS files in any UE4 Project.
• Extra UE4 command arguments for automated testing: https://docs.unrealengine.com/en-us/Programming/Basics/CommandLineArguments
• I didn't find a way of running the tests on packaged games (.exe projects). It seems that the only way of running functional tests is trought the editor. However, i think that SkookumScript can be used to easily run tests (using exclusively SK and not relying on the UE4 automation tool) even when the game is running on a different platform. Like a PS4.

What's the goal of this

If you're like me and didn't hear about devOps in your entire life then you might be wondering why you need this.

The point of using Jenking to automate your builds (and your tests) is to give you confidence that your project is going well. If someone in your team makes a mistake and send a commit to the master repo that breaks the game (test fail or build fail), then Jenkins can send an alert so the issue can be solved the moment it happens and not two weeks before the release of the game. Bugs are waaaaaaaaay easier to solve in a code/commit you've made some hours ago than several weeks ago.

Also, if you're developing a project that needs continuous releases (MMORPG, FreeToPlay, etc) then this system is perfect for you.

The nice thing about all this is that once you've set everything up. The builds and tests will run automatically, potentially saving you and your team lots of hours!

---

Prerequisites

Before you begin: This documentation is solely meant for Jenkins running on Windows servers and desktops. This may also work on GNU/Linux build servers, however the included build scripts have only been tested for Windows installations.

• To get started, please first download and install Jenkins on the computer you wish to use as a build server. (Jenkins Installation Docs)

• Clone or download this repository to your CI server to access the required build files for the final steps of the tutorial.

• You will also need to download cURL (only if posting notifications to Slack) and Unreal Engine 4.

• 7-zip https://www.7-zip.org/ for compressing the builds.

• Java is also needed https://www.java.com/es/download/ - Jenkins can use Java to run as a desktop app so it can open an instance of the UE4Editor.exe.

Before starting, this is a setup example i'll be using to explain things. You don't have to use this setup for your project. It's also reccomended to have Jenkins running in a different computer (or cloud server).

https://imgur.com/a/QlPotv8

• The root folder (name of my project) contains 3 folders:
  • /Project/ this is my local UE4 Project git repo, not related directly to Jenkins.
  • /Assets/ this is my local game assets git repo, not related directly to Jenkins.
  • /Jenkins/Builds/ This is where the Step5/Step6 and Step7 scripts from this repo will create my builds.
  • /Jenkins/Projects/ This is where Jenkins is downloading the UE4 Project from my master repo (i'm using VSTS).
  • /Jenkins/@Project/ Autogenerated by Jenkins.
  • /Jenkins/scripts/ This is where i'm placing the scripts from this repo. The only exception is RunJenkinsAsDesktopMode.bat

---

Before starting, about the scripts from this repo

• PostToSlack.bat: Allow you to post messages to slack, go to the original repo to see more examples.

• RunJenkinsAsDesktopMode.bat: Allows to run Jenkins as desktop mode using Java. Needs to be placed on %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup so it can be executed automatically when windows starts.

• Step1_DeleteAutoGenerated.bat: Will delete autogenerated folders from a UE4 project so the build is less prone to errors. Modify it so it points to your Jenkins Workspace (where Jenkins downloads the master repo).

• Step2_StartBuild.bat: This will generate VS Project Files. Is similar to manually right clicking a .uproject file>>Generate Visual Studio Project Files. Modify it so it points to your Jenkins Workspace (where Jenkins downloads the master repo).

• Step3_CompileScripts_Win64.bat: This will build/compile your source files in Win64. Modify it so it points to your Jenkins Workspace (where Jenkins downloads the master repo). This can be changed to Win32 but i don't think it's supported in a development environment. You should change this to Mac/Linux in case you're using a different platform to make your game. Keep in mind that these scripts have only been tested on Windows.

• Step4_RunTests.bat: This script will run run all the tests that are labeled as "Functional Tests". All the tests created using https://www.youtube.com/watch?v=f4LpDXjFgVQ are labeled like that by default. Relevant parameters:

  • ExecCmds="Automation RunTests Functional Tests": Run tests labeled as "Functional Tests"
  • -log=autolog.txt: Create a file with the tests log in the "Jenkins Worspace\Saved\Logs". Example: C:\ProjectName\_JenkinsCICD\Project\Saved\Logs\autolog.txt.
  • type "C:\ProjectName_JenkinsCICD\Project\Saved\Logs\autolog.txt": Prints the log from "autolog.txt" into Jenkins.

• Step5_BuildFiles_Win64.bat: This will create a Win64 executable of your project in "\Builds".

• Step6_Optional_CookProject_Win64.bat: Similar to Step5 but creates an executable with less dependencies. I think this is useful in case you want to run the executable in a computer that doesn't have UE4 installed.

• Step7_Archive.bat: Compress Step5/Step6 results into a .zip file.

• LogParsers/1-CompilationLogParse.txt: Optional. For determining when a compilation will fail or be marked as unstable.

• LogParsers/2-TestLogParse.txt: Mandatory. Will mark test as failed if "Test Failed" string is found in the log.

• LogParsers/3-PackageLogParse.txt: Optional. For determining when a packaging will fail or be marked as unstable.

Compilation and Package seems to work fine without the need of a log parser. Log parsing is recommended tho as it highlights errors, warnings and relevant info.

Let's BeginPlay()

Installing Jenkins

I'm not going to explain everything here, only the parts that are specific for this setup.

• Install Java
• Install Jenkins, in case Jenkins create a service "Jenkins" by default, go to services and disable it.
• Modify RunJenkinsAsDesktopMode.bat - Not much needs to be modified if you installed everything in their default path. You may want to change the port Jenkins will use tho.
• Open RunJenkinsAsDesktopMode.bat as admin, wait for Jenkins to install and go to your address:port location using your web browser.
• Follow the steps to configure Jenkins (default installation should work.)
• Install the Log Parser Plugin in Jenkins.
• Go to Manage Jenkins>Configure System>Console Output Parsing and create 3 parsing rules (compilation log, test log and package log). Each log parser should point to the files from the /LogParser/ folder from this repo. i.e. https://imgur.com/a/M3z4M3e - Keep in mind that only the Test log parser is mandatory.

Note: The Log Parser Files are not 100% tested. You may experience "false positive" errors. You can edit the log parsers to get rid of any false positives tho.

• Return to Jenkins Home and create a New Item of type "Folder". This folder will contain our 3 Freestyle projects (compilation, test and package).

Creating the Compilation Job item in Jenkins

This job should fail if the compilation of your source fail.

• Create a new item inside the created Folder, this item will be a Freestyle Project. I named this item "Step1-Compilation".
• General* Click on "Advanced" and check Use custom workspace. This is the directory of your workspace (where Jenkins will download the master repo). In my example, i'm using C:\ProjectName\Jenkins\Project.
• Source Code Management: Set up your source control repository.
• Build Triggers: I'm using "POLL SCM" with the following parameters 0 0 * * * that will trigger the job each day at midnight. https://crontab.guru/ is useful for creating custom triggers.
• Build: Use windows batch to call the first 3 steps. Similar to this example:

call C:\ProjectName\_JenkinsCICD\scripts\Step1_DeleteAutoGenerated.bat
call C:\ProjectName\_JenkinsCICD\scripts\Step2_StartBuild.bat
call C:\ProjectName\_JenkinsCICD\scripts\Step3_CompileScripts_Win64.bat
rem You can call PostToSlack.bat here too

• Post-Build: Use "Console Output Parsing" and use the "Compilation Log Parser" here. Mark "Unstable on Warning" and "Failed On Error". Unstable on warning is optional but since this is the compilation of your source files, you probably don't want to have any warnings.

Creating the Test Job Item in Jenkins

This job should fail if at least one test fails.

• Create a new Freestyle Project inside your folder. I named this item "Step2-RunTests".
• General*: The same step as above.
• Source Code Management: Nothing here.
• Build Triggers: We want to trigger this job after "Step1-Compilation" finishes. So we choose "Build after other projects are built" and write "Step1-Compilation" as parameter.
• Build: Use windows batch to call step 4. Similar to this example:

call C:\ProjectName\_JenkinsCICD\scripts\Step4_RunTests.bat
rem You can call PostToSlack.bat here too

• Post-Build: "Console Output Parsing" is mandatory in this step. Use the "TestLogParse" here. You probably don't want to mark "Unstable on Warning" since we don't care about UE4 related warnings here. We only care if the tests fail or not.

Creating the Package Job Item in Jenkins

This job should fail if the packaging fails.

• Create a new Freestyle Project inside your folder. I named this item "Step3-PackageProject".
• General*: The same step as above.
• Source Code Management: Nothing here.
• Build Triggers: We want to trigger this job after "Step2-RunTests" finishes. So we choose "Build after other projects are built" and write "Step2-RunTests" as parameter.
• Build: Use windows batch to call step 5, 6(optional) and 7. Similar to this example:

call C:\ProjectName\_JenkinsCICD\scripts\Step5_BuildFiles_Win64.bat
call C:\ProjectName\_JenkinsCICD\scripts\Step7_Archive.bat
rem You can call PostToSlack.bat here too

• Post-Build: Use "PackageLogParse". Mark "Failed On Error". Unstable on Warning is optional since UE4 will throw warning for Project-Unrelated stuff. You can modify "3-PackageLogParse.txt" so it get rids of every posible "false positive" so you can use "Unstable on Warning".

Last Step

Don't forget to copy "RunJenkinsAsDesktopMode.bat" to %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup.

Final considerations

• I think Skookumscript could be used to write the tests in a simpler way. Also, SK could be used to run the tests on packaged projects.
• Step 3,5,6 and 7 can be modified/duplicated so it works on different platforms too. You could package your game on Android, Linux, etc. using Jenkins.
• In case of deploying for Android. The packaging will generate a "Install_ProjectName_Shipping-armv7-es2.bat" that can be used to automatically install the project into your phone. This batch file could be called from Jenkins too. However, the batch requieres a "DEVICE" value. I'm not sure what that value is (i tried the code that appears if i write "adb devices" in cmd).
• "RunJenkinsAsDesktopMode.bat" will open a batch windows each time you turn on your pc. The window can be closed but in case you want it to run on the background, then you could use something like this http://www.winhelponline.com/blog/run-bat-files-invisibly-without-displaying-command-prompt/ (This was nos tested by me)