Dependency Manager - for Plugins

[nonunknown]

Describe the project you are working on:
GraphNode Based State Machine
Describe the problem or limitation you are having in your project:
Sometimes my plugins depends on another plugin to work at maximum power.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
To solve this I was thinking about a plugin dependency manager.
Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Ok, so for the plugin I'm working on I need the plugin ScreenDebugger to show variables for debug puposes, and show the resulting JSON to the end user at runtime. For facilitating the life of end-user a dependency manager would work fine, so how it was going to work?

File Structure

Addon/
L PluginName/
L Scripts/
L Icons/
L Dependencies/
L dependency.cfg

Dependency.cfg

Qodot="446" #asset ID
Sound Manager="361"

[Version]

Qodot="2.1" # get 2.1 version
Sound Manager="default" #gets the last version

At the Plugin.gd that the user is working on:

func _on_enter_tree():
    var err = Dependency.download()
    if err == OK: print("dependencies downloaded successfully")

When the Engine reads this code

If everything is working well the engine will show this window for each plugin:

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Maybe it can, But the objective here is to facilitate the end-user's life
Is there a reason why this should be core and not an add-on in the asset library?:
You got me! maybe this can be done as plugin!

[YuriSizov]

Dependency management is a complicated subject, but it should be tackled in asset library at some point. Couple of notes, though.

1. I think that we should not introduce a new config file, though. plugin.cfg is already there. It can have a section for dependencies.
2. This information should be provided in asset library as well, so the UI for submissions needs to be enhanced. Ideally, most of the information should be imported via plugin.cfg, and not inputted manually, as it is now.
3. There should be two types of dependencies: required and non-critical. Required dependencies need to be installed with your plugin. User must be asked to confirm their installation, though. Non-critical dependencies can be omitted, but you must be able to check in plugin code if those are installed and enabled. Some additions to EditorPlugin class are required for that.
4. An API to fetch and install asset library data from GDScript/GDNative must be introduced. This would allow plugin developers to provide "install on demand" type of controls.

[blockspacer]

NOTE: conan, mentioned in another proposal can be used to manage dependencies #606

[YuriSizov]

@blockspacer This has nothing to do with external tools. This proposal is entirely for inner plugin system.

[nonunknown]

@blockspacer and @pycbouh btw being external can be cool too, why? if there is a main "Dependency" folder can be used to build "modules/plugins" or for example a empty addons folder just with a dependency file can download all plugins mentioned in the file!

[YuriSizov]

@nonunknown Well, this conan thing is for C/C++ modules. Godot plugins are not that.

Also, I would argue against any "dependency" folders and other kinds of separations for plugins, that are used directly and plugins that are required by other plugins. A flat list is what we should have in the end. And if we force a user to install a plugin, then that plugin must be available to that user.

[blockspacer]

@pycbouh BTW, conan is universal and can package and manage even archives and pre-built binaries, it it not C++ only

see Conan: A Python package manager https://docs.conan.io/en/latest/howtos/other_languages_package_manager/python.html
and https://docs.conan.io/en/latest/howtos/other_languages_package_manager/go.html

With this flexibility, Conan is able to do very different tasks: package Visual Studio modules, package Go code, build packages from sources or from binaries retrieved from elsewhere, etc.

Python code can be reused and packaged with Conan to share functionalities or tools among conanfile.py files. Here we can see a full example of Conan as a Python package manager.

[willnationsdev]

This topic was also mentioned in #142, although, it doesn't present a formal proposal for how to implement a dependency manager.

[blockspacer]

mentioned at
godotengine/webrtc-native#14 (comment)

conan allows to manage deps not only for C++. For example, GDScript plugin may be packed into archive during conan create command. In game-project-dir conanfile stores deps of created package and can unpack created plugin during conan install command with auto-resolving deps (no recursive deps problem).

Again, idea is simple: game project lists required plugins in custom conanfile. during conan install all required plugin files must be unpacked at some destination folder. Engine is able to detect plugin folders (for example by file watcher or re-import button, as done with assets) and auto-install plugins.

[Faless]

Again, idea is simple: game project lists required plugins in custom conanfile. during conan install all required plugin files must be unpacked at some destination folder. Engine is able to detect plugin folders (for example by file watcher or re-import button, as done with assets) and auto-install plugins.

@blockspacer
Wouldn't we need to ship conan with godot? Would it work on all platforms (e.g. Android/iOS/HTML5)?

[blockspacer]

@Faless

Wouldn't we need to ship conan with godot?

From developers machine perspective:

If you want to install plugins from editor's GUI (deep integration with editor), then it is possible to place conan in some folder near engine and run it without system-wide installation. Note that conan requires python 3 (python3 can also be installed not system-wide, but in specific folder, near to game engine).

For other, simplest use case (per-project plugins without deep integration with editor), all plugin files can be unpacked manually into plugins folder. conan just allows to automate that unpack step and find dependencies, resolve recursive dependencies, e.t.c. For simplest use-case (when developer just needs to download and unpack some files) plugin creator can use not only conan, but maven, NPM (whatever tool he wants).

More details about desired type of integration with editor will be helpful. For example, do we need in-engine GUI to install plugins from internet (requires to run conan install command from editor), not only from local filesystem (developer than can use any preferred tool to download and unpack files to plugins folder, not only conan)?

From end user machine perspective (target platform):

No need to install conan for end users, cause it is tool for developers.

Would it work on all platforms (e.g. Android/iOS/HTML5)?

conan can run anywhere python3 can run. It's main task is to find and download files based on some metadata (plugin version, deps, e.t.c.). So target platform may be any (Android/iOS/HTML5/whatever you want) cause target platform don't require installation of development tools.

Anyway, as said above

Dependency management is a complicated subject

so it may good idea to not create custom solution from scratch but integrate with existing one (conan, maven, NPM, e.t.c.)

[Faless]

(Android/iOS/HTML5/whatever you want) cause target platform don't require installation of development tools.

We are working towards running the editor on HTML5 and Android (and possibly iOS too), which will be available in 4.0, so if the GDScript plugins has to be handled like that during game development, whatever solution we choose, it must support those platforms too.

More details about desired type of integration with editor will be helpful. For example, do we need in-engine GUI to install plugins from internet (requires to run conan install command from editor),

Did you even try Godot? We already have a GUI for installing handling plugins in the editor, and an asset library to download them from the internet (using the editor).

[blockspacer]

@Faless thanks for detailed response.

running the editor on HTML5 and Android

It may be not not good idea to run python3 on HTML5 and Godot may require custom dependency management solution for that platform. Note that if you want to allow manipulation of files on host system (if HTML5 editor can communicate with host OS and file-system, similar to VueJs CLI what can run terminal commands from browser), than it does not differ from host OS (and can run any tool from host OS).

For mobile it must be possible to run python3, but i'm not sure if it may be good idea to try.

We already have a GUI for installing handling plugins

Sorry for my ignorance. I haven't tried to install any plugins in Godot editor (used unmodified editor), now i understand that issue better

[Faless]

@blockspacer Then I guess we'll have to find a different solution that can support all our editor platforms.

[nonunknown]

@blockspacer Then I guess we'll have to find a different solution that can support all our editor platforms.

C++ doesnt have any framework for dealing with dependency managing? Maybe if there is one this could be implemented as a module

[Calinou]

Closing in favor of #142, which has more information about the feature proposed here.