Proposal: Building Godot as part of a larger project, changes to Build System

[sdfgeoff]

A game project made with Godot is made from two parts: Godot and the game.
Most of the time, people download Godot from the website, and export the game manually. Ideally, no-one needs to compile Godot on their own.

However, for some projects where Godot has local changes, a custom version of Godot is necessary. In this case, it is desired that Godot is build as part of the project.

For example, if I have the directory structure:

Project_name
 ├ lib
 │    ├ godot
 │    │    └ SConstruct
 │    └ gd_native_module
 ├ my_awesome_game
 │    ├ project.godot
 │    └ main.tscn
 └ SConstruct

In this case, there are two features that would be nice:

1. The ability to build Godot as part of the main projects SConstruct file (ie use Godot's SConstruct as an SConscript)
2. The ability to use variant_dirs. From a user this would be scons p=x11 variant_dir=path/to/x11. inside a larger projects SConstruct this may look like SConscript('lib/godot/SConstruct', env={'p':'server'}, variant_dir='bin/godot-server'). This would allow building server, editor and release builds without overwriting each others .o files (Currently, if you do scons p=x11 scons p=server then another scons p=x11 will rebuild from scratch. This plays havoc with trying to cache builds done on CI [thankfully SCONS_CACHE does help with this]).

There are currently two things preventing these features:

1. Many functions currently use "custom" path wrangling. For example the detection of modules is currently done by files = glob.glob("modules/*"), and some modules use things like: monoreg = imp.load_source('mono_reg_utils', 'modules/mono/mono_reg_utils.py') If nothing else, these should be done by Scons internal functions such as: Dir("modules").glob("*").
2. The modules that do use the Dir() class often use it in association with Dir("#") which references the root of the SConstruct hierarchy. Normally this is the godot repository, but in the structure proposed above, it would be the project_name directory.

Today I spent some time investigating the feasibility of these changes, and these changes gets approved, I'm happy to do the work and make a PR. My current approach would be to:

1. Internally, route the godot base path through all the modules/python files. Probably using the construction environment that is already shared to many places (all the SCsub's have it).
2. Fix all the custom path wrangling to use godot's base path, using Scons' internal functionality where it makes sense.
3. Whatever else is necessary to get variant_dir working (which shouldn't be much at this stage)

So. Does anyone have any thoughts about these changes? Are there any ways of achieving what I've listed that don't require changing the current build system?

[capnm]

In most projects I generally use the out-of-source builds ( https://stackoverflow.com/questions/1762044/how-to-do-an-out-of-source-build-with-scons ) Also, e.g. when debugging the source code directories are not full of object code files and so better readable.

Yes, it would be a good idea if Godot supported that!

[Xrayez]

I've tried to workaround this with many hacks:

#!/usr/bin/env python
# SConstruct (game project)

import os
import sys

env = Environment()
# SConscriptChdir(0)
godot_path = os.path.join(os.getcwd(), "godot")
env.fs.pathTop = godot_path
env.fs.Top = Dir(env.fs.pathTop)
env.fs.SConstruct_dir = Dir(godot_path)
env.fs.chdir(Dir(godot_path))
sys.path.insert(0, godot_path)
env.SConscript("godot/SConstruct")

While some of the above methods did help to compile Godot from within game project's root Sconstruct, none combination of them actually make it possible to do so.

So currently I just use Python's subprocess to run another instance of scons, something like this:

#!/usr/bin/env python
# SConstruct (game project)

import subprocess

# Project specific options goes here ...

engine_args = ["scons", "target=release_debug", "tools=yes", "extra_suffix="mygame"]
subprocess.run(engine_args, check=True, shell=True, cwd=engine_abs_path)

# Export project here ...

That way, you can rest assure that no matter what happens during Godot development, you can have somewhat reliable builds using almost the same mechanism.

You might also be interested in this: #36922.

[sdfgeoff]

Yes it works, but it has the scent of Recursive Make Considered Harmful. In this case: recursive scons considered harmful.

Your example of using subprocess would probably be better written as a scons command builder rather than subprocess. This would:

• Mean that the build runs at Scons build time rather than at Scons parse-sconscript-file-time
• Allow artifacts to be cached by scons
• Allow other tasks to properly depend on godot being built using Scons to allow parallelization

Running godot builds as a command execution rather than as a Scons SubScript has several disadvantages.

• Variant (eg linux, windows, server build) builds overwrite artifacts in the source directory, so requires fetching/restoring files from cache multiple times in the same build. This results in scons re-performing the linking step resulting in longer build times and the cache growing really fast.
• The outer Scons has to run the inner scons each time because it doesn't know if any files have changed. This means that anything that depends on the Godot binary also has to run, regardless of if there is any change. The result is that if you have a pipeline: build_assets -> build_scons -> export_game -> package_game_as_zip_file -> run_tests -> upload_to_developent_build_cache then the pipeline will re-run the second half of the pipeline every single time, even if no files have changed.

Yes, it works, and it was what I was doing. It resulted in no-op builds taking about 5 minutes (a no-op build of Godot is about 30-40 seconds, need 2 builds of Godot, then assets are re-imported and re-exported, the game is packaged etc (about 3 minutes)). If Godot has had a Sconscript file, a no-op build would be only fractionally longer than a single Godot no-op build (31-41 seconds).

---

</begin_rant>
I haven't looked at it for a while, but if I remember correctly Godot's use of Scons is poorly done. Many things are done at script execution time rather than at build time. Although Scons is nice and flexible because it's written in python, this tends to result in build scripts being a mess of Scons and general python (of which your example: using subprocess.run rather than env.Command is an example, and my earlier mention of all the custom path wrangling). A simpler build system or a more restrictive one alleviates this problem.

Fortunately I haven't used Godot for ages due to other problems with the engine, so, meh.

Also, this recursive-scons-problem is why I dislike the fact that every modern language has it's own build system. Rust has Cargo, javascript has webpack etc. etc. Everything using it's own build system means that any multi-language orchestration is an exercise in futility.
</end_rant>

[Xrayez]

Your example of using subprocess would probably be better written as a scons command builder rather than subprocess.

Not sure, the Command builder is meant to accept source/target and not something which can act as a wrapper for the command line, I'm likely wrong on this. I also use some utility functions already such as git pull to optionally synchronize the engine version against tracked branch:

def sync(self):
    args = ['git', 'pull']
    subprocess.run(args, check=True, shell=True, cwd=self.engine_abs_path)

There's Execute function which seems to be more suitable, but again I wouldn't rely on using SCons-specific methods with Python hacks already in place to build Godot with existing limitations.

The outer Scons has to run the inner scons each time because it doesn't know if any files have changed.

In some cases I specify skip_build option explicitly or implicitly depending on the context.

if not env["skip_build"]:
    # Workaround some of the engine's limitations.
    engine.apply_patches() 
    engine.build() # runs SCons.
    # The tree is dirty after applying patches, clean up.
    engine.reset()

But yeah it doesn't not handle all possible nuances.

One example of such optimization is when you need to generate help text depending on whether you have --help option passed, docs:

If you know that your configuration does not define any additional help text in subsidiary SConscript files, you can speed up the command-line help available to users by using the GetOption function to load the subsidiary SConscript files only if the the user has not specified the -h or --help option, like so:

if not GetOption('help'):
    SConscript('src/SConscript', export='env')

And AFAIK that's inherent disadvantage of SCons being slow at parsing configuration files in general.

I think SCons is a bit too powerful indeed, so that only the chosen can work with it. 😛

Note that I fully agree with your concerns but I present an alternative for those who can accept these limitations being not critical for their projects. Indeed it would be better if Godot is built within the same scons process but seeing existing development philosophy, it's not something which is going to happen or accepted, so it's not really worth to fix it on the engine side, unless you're willing to maintain your own fork of Godot, IMO.

At the very least I find SCons to be as a nice replacement for plain python scripts. Namely not having to setup your own command line parsing etc. With the additional advantage of being able to build the engine itself along the way, so that's nice (despite it being not elegant).

[mhilbrunner]

May be superseded by godotengine/godot-proposals#606

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!