Allow putting modules that compile into the engine in a project's folder

[Sixmorphugus]

Godot version:
3.1

Issue description:
For my own project, I've started making a few engine modules. These modules include a custom image format loader, an ingame visual scripting system, and a voxel model importer/resource system. I have looked into NativeScript already and find it inappropriate for these use cases for many reasons (including but not limited to: fiddly setup, difficulty generating Visual Studio solution files, missing support for many native engine types, access to external libraries I need that I would otherwise have to include 2 versions of, missing engine functions and utility classes, no direct ability to register new node types or engine classes without auto-attached scripts, slightly-too-different API, and missing engine memory management functions), so please don't bring up NativeScript. If I could do this sort of thing with scripts, I would have.

My game is called Marla Died Too, so I am currently happy enough creating my modules individually with the prefix "md_" (Marla Died) and putting them into my local engine repository. I have a computer built for recompiling Unreal Engine several times a week, so scons iteration time is not an issue (scons, being a fairly good build system, also only compiles files I changed most of the time anyway). However, it would be very useful if I could put these modules somewhere in my project directory, rather than the engine directory, and still have scons find them. Ideally, I would like to be able to run builds in my own project folder and generate a Visual Studio view that visually separates my own project's module files from the engine. Then, rather than having to work inside the engine folder and with a copy of the engine repository that I need to update myself, I can work in my own project's directory and not have to worry about it.

The build process would not change. scons would still recompile the engine around my modules, depending on the SCstub files they contained, exactly as it compiles modules now. All this would add is a much greater ability for those who work with projects that add engine code to organize their work. NativeScript users would not be affected.

If there's interest in this, I could potentially start developing a pull request that does this myself. If not, I will probably make something anyway for my own use. It seems like it would be an incredibly useful thing to support.

[Xrayez]

Supporting this, perhaps there should be implemented an easy way to specify a custom search path for modules within the scons build system.

What I currently do is simply symlink all my custom C++ modules into the engine/modules with a python build script, which also acts as a wrapper over scons.

Also see for #28343 for detailed discussion and pitfalls. I ended up moving it all away from project folder, so the structure looks like this now:

project being an actual Godot project, engine added as a submodule (using master branch myself so have to keep track of exact version used), and modules hold my git modules which are symlinked into engine/modules, then I just recompile it with build.py.

You'd also want to exclude symlinked modules from checked out engine submodule by adding excluding engine/modules folder locally, just add this to .git/modules/godot/info/exclude (or similar path):

modules/

This just avoids the git displaying the repository as "dirty".

I have looked into NativeScript already and find it inappropriate for these use cases for many reasons

I'm also kinda reluctant because of these reasons tbh, I'd just rather go straight to the source. 😅

[Sixmorphugus]

@Xrayez this is pretty cool, I think I'm going to try and do the same thing with my setup for the time being.

[aaronfranke]

Are you saying that a prebuilt version of Godot should keep scons inside it and then build your module somehow integrating it, or that you want to keep Godot as a "submodule" in your project and load engine modules from outside the engine folder then rebuild it?

[Sixmorphugus]

Are you saying that a prebuilt version of Godot should keep scons inside it and then build your module somehow integrating it, or that you want to keep Godot as a "submodule" in your project and load engine modules from outside the engine folder then rebuild it?

Neither. This would be a feature just for people who have the engine source code on their machine as you (as far as I am aware) cannot build modules without it, but I was thinking it should not require godot and the project to be any specific places relative to each other (including godot as a submodule of your repository, I guess).

Just the ability to pass an arg to the normal scons files to add additional modules folders would work well. Ideally, however, godot (if you built it from source) would optionally generate an scons file inside your project's folder and put a second modules folder in there as well. Running scons from inside your project's folder would build the engine normally, but additionally consider any modules it finds in the project's folder. And, of course, the generate visual studio solution function would generate a solution in your project's directory with two projects, one that contains all the godot files and runs the original godot scons build script, and one that contains your own project and runs the extended scons build script that builds the engine with your modules added in.

[Xrayez]

Stumbled upon another use case.

[fire]

My proposal is for an argument to scons that allows the developer to put a path to search for modules.

scons p=x11 -j8 search_modules=~/godot-modules/ 

Edited:

Or

scons p=x11 -j8 module_path=~/godot-modules/ 

[Xrayez]

@fire, I do make up a command list of disabled built-in modules which I don't use in a project anyway:

# Disable unused modules
for name in modules_config['modules_disabled']:
    engine_args.append('module_' + name + '_enabled=no')

so passing an additional argument for the search path would be alright.

I'm using what seems to be a undocumented feature right now which could also be possibly used to specify additional modules path to look for (sorta like bash-profile):

godot/SConstruct

Lines 95 to 104 in 1361fa7

	# Build options
	customs = ['custom.py']
	profile = ARGUMENTS.get("profile", False)
	if profile:
	if os.path.isfile(profile):
	customs.append(profile)
	elif os.path.isfile(profile + ".py"):
	customs.append(profile + ".py")

In my wrapper script, I specify the profile script by:

# Fetch additional custom config
config_custom_path = os.path.join(root_dir, config.config_custom)
if os.path.isfile(config_custom_path):
    engine_args.append('profile=' + config_custom_path)

In the profile script, you can set any command line arguments basically.

[fire]

The design of the API doesn't require any modification of the code nor a need to make a custom script.

For example:

git clone https://github.com/godot-extended-libraries/godot-sqlite sqlite
git clone https://github.com/godotengine/godot
cd godot
scons p=x11 -j8 module_path=../

This arrangement would be a valid compile of the engine.

It would be the same as:

git clone https://github.com/godotengine/godot
cd godot/modules
git clone https://github.com/godot-extended-libraries/godot-sqlite sqlite
cd ..
scons p=x11 -j8

[willnationsdev]

So, there are two things I'm aware of that might be relevant here:

1. When running scons, you can pass an argument with a flag (a -C flag, iirc) that lets you specify a directory in which to run scons rather than using the current directory as the default. This would be useful if, for example, you wanted to have an EditorPlugin run scons against Godot (or some other terminal script).

2. The thing that should be tested/verified is whether an SCsub file, which specifies the additional includes/sources/libraries to be included in a given build target, can exist in an ancestor/sibling/cousin directory and still work. There's a chance that scons would fail to work properly with an SCsub that isn't contained in a descendant directory from the main selected directory.

The way I would see something like this going though would be to let someone go to Project > Tools (or something) and create a tool option that generates/initializes local module code for the user. They could pick a module name, which would build...

/modules
/modules/<module_name>
/modules/<module_name>/SCsub
/modules/<module_name>/config.py
/modules/<module_name>/register_types.h
/modules/<module_name>/register_types.cpp

You would then need some data-driven way of ensuring that the SCSub would get picked up by Godot's main SConstruct file. Maybe have a per-project environment variable that references the project's modules directory and then users could pass a project_modules_dir=<path> argument which the SConstruct script could use to find the modules folder for that project and start including each of those in the compilation process.

[Xrayez]

There's a chance that scons would fail to work properly with an SCsub that isn't contained in a descendant directory from the main selected directory.

I've seen somewhere that SCons doesn't work with relative paths outside the build root, for the reasons of simplified and reliable multi-threaded builds AFAIK.

The way I would see something like this going though would be to let someone go to Project > Tools (or something) and create a tool option that generates/initializes local module code for the user.

It would be great if Godot could support C++ module generation out of the box. Actually I've written a python package for this. The complexity arises by having to conditionally generate some parts of a module generation which are specific to each Godot version...

[Xrayez]

There's a working WIP implementation #36883 which could help solve this. The only caveat is that the implementation doesn't assume any kind of module to be passed to game_module path, but rather as a general-purpose scons C++ package (the way I see it):

scons platform=windows tools=yes target=release_debug game_module=D:/packages/my_package -j8

which could then be made to build a collection of modules instead.

To be more concrete, you'll have to either explicitly list the modules in package's SCsub, or actually replicate the way current root Sconstruct collects modules:

# my_package/SCsub

Import('env')

module_env = env.Clone()
module_env.add_source_files(env.modules_sources, "*.cpp")

for module in your_module_list:
    env.modules_sources = []
    SConscript(module + "/SCsub") # compile each module individually
    lib = env_modules.add_library("module_%s" % module, env.modules_sources)
    env.Prepend(LIBS=[lib])

So in theory you could even write your own engine/framework with this as you can build your own hierarchical chain by calling SConscript, so yeah I think this should solve most uses cases.

[Xrayez]

I've implemented #36922 to specifically resolve this, as well, whatever works! 🙂

[Sixmorphugus]

This is very nice. Thank you for your hard work!