Improve Maintnance of Optional Plugins

[AdamsLair]

Problem

• Right now, optional plugins like Steering and DynamicLighting are prone to errors, since they are almost never tested.
• The issue will likely become larger and larger, since more optional plugins may be added to the repository over time.

Solution

• Modify Nightlybuild to always build all plugins, but copy only the standard ones into the package. If a build error occurs, stop and report.
• Add some basic Unit Tests for plugins and add them to the Unit Testing queue, either by including them in the default NUnit project, or by creating more NUnit projects that will be invoked by NightlyBuilder.
• To keep compile and loading times down, think about splitting the monolithic Duality VS solution into multiple solutions. Maybe one containing everything, one only for the standard version without optional plugins?
  • Make sure to always favor adding projects over referencing prebuilt Assemblies, within the Duality repository itself.

[Andrea]

Alternatively you could check out build solutions (you could use myget they have a build server that oss projects can use for free)
Also, spining a window on each test is probably going to be a problem.
BTW your tests are actually integration tests, unit test test a unit (generally that means a class) your tests tend to inlcude many classes

[lukaspirkl]

I like the idea about separation. I guess these optional plugins are technically same as any other third party plugin or plugin user is creating for game. So currently I am missing some "official" way how to share plugin code. Plugin needs reference to Duality's core so how would one create repository which can be easily downloaded, built and attached to game?

For example look at these two projects:

• https://github.com/BraveSirAndrew/DualityScripting
• https://github.com/SirePi/FrozenCoreDuality

They are both done differently and I am not able to simply open and run any of them. FrozenCore needs to be placed into exact folder to find Duality and DualityScripting requires NuGet package which is not publicly available (or at least it was not found automatically).

So I would really love to have some comfortable and ideally unified way how to share and use third party plugins. Maybe it could be possible to integrate NuGet package manager into editor and download plugins as packages without source code compilation. Actually this option could be pretty cool. Core of Duality could be also published by NuGet so we (plugin developers) will be able to easily share source code with all dependencies and publish binaries to NuGet repository for everyone.

I think that plugins and whole extensibility is main selling point of Duality (at least for me) so something like that could increase the number of plugin developers => more interesting features => more users => more games. :-)

[BraveSirAndrew]

Hi existent

Apologies for not getting around to sorting this out sooner. We've just been so flat out on our game, I haven't had time! You can use our versions of all of the main Duality repos by adding this link - https://www.myget.org/F/6416d9912a7c4d46bc983870fb440d25/ - to your NuGet Package Manager's Package Sources dialogue in Visual Studio. NuGet will then look at our feed (which is totally public on MyGet) to pull down the dependencies.

I've actually been thinking about ways to integrate NuGet into Duality recently, as it absolutely seems to be the way to go when dealing with this sort of thing. We already have problems internally with versioning some of our own plugins, and a dependency manager would totally just make those go away. I might try and find some time this weekend and look into it. Sounds like fun:)

[AdamsLair]

@Existent has a very good summary on the topic. I agree that a unified way of getting plugins into Duality would be great - and that NuGet is a good candidate for providing that. Didn't get around to look into it thoroughly, but I'd appreciate a Pull Request if @BraveSirAndrew finds a good way of integrating it. :)

That said, the Duality repository should definitely work out of the box, so I'd favor a non-destructive approach that doesn't require anyone to install NuGet before being able to build Duality.

Edit: Some questions / issues that we might need to talk about:

1. If there is a central Duality package repository where Core, Dependencies and all of the Plugins go, where to host it? Is there something like GitHub for NuGet packages?
2. In order to keep the Duality repository working out-of-the-box, @Andrea mentioned a lightweight command line version of NuGet that could be included in the repo - would that be a viable option, or did I get something wrong?
3. Would it be a good idea to integrate some kind of NuGet "Duality Plugin Browser" into Duality itself, like @Existent proposed? It is a pretty cool thought indeed - but how much work would that mean? Are there existing solutions or APIs to do something like that?
4. Would there be a need for additional server infrastructure aside from the one provided by point 1?

[lukaspirkl]

1. There is http://www.nuget.org/ which is public repository for NuGet packages. It is also connected with http://www.symbolsource.org/ so you should be able to host the debugging symbols and sources to incredibly easy debugging. Basically this allows you to step into the code from NuGet package.
2. The NuGet.exe is the client. Point is that you don't have any third party dlls in the repository but only the identifiers (unique name of the package and version number). In the csproj file, there is some hook which calls the NuGet.exe when build starts to resolve these dependencies and download what is missing from the repositories. So for plugins, you would need to publish the Duality (the dlls which are needed in plugins) as a NuGet package. Then the plugin will use this mechanism to get these dependencies. This will be better than my explanation: http://docs.nuget.org/ :-)
3. There are source codes of NuGet.exe and other tools on CodePlex: http://nuget.codeplex.com/ I think some inspiration can be found there. Also some API should be described somewhere for sure.
4. I guess that you can rely on public nuget repository mentioned at point 1. The manager in the editor could just filter publicly accessible packages. For example the version of duality is 1.2.3 find all packages which contains duality 1.2.3 in the list of their references.

Quick googling brings for example these results:

• http://www.aaron-powell.com/posts/2011-02-20-creating-a-nuget-plugin-engine.html
• http://matthamilton.net/nuget-for-plug-ins
• http://exsertus.wordpress.com/2013/12/20/nuget-as-a-plugin-deployment-platform/

[AdamsLair]

I've now begun looking into NuGet. My first goal is to replace Dualitys submodule-driven dependency management with NuGet package driven dependency management. Once that is done, I'll think about how to handle unified plugin management with NuGet.

Thoughts so far:

1. Duality, DualityEditor and DualityLauncher should probably not be NuGet packages.
   • Everyone who develops Duality plugins already has Duality, so there should be no need ever to pull it dynamically via NuGet. Also, there is no real benefit from doing so, at the cost of bombarding the NuGet repository with a new library version on each release.
   • Releases should probably still be a simple binary download package.
   • The Duality core repository (engine, editor, launcher) shouldn't be split. Due to development reasons, it's probably a good idea to also keep standard plugins in there. Optional plugins can be moved distinct repositories.
2. Duality Plugins could be distinct NuGet packages.
   • Also the standard ones. No first-class or second-class plugins!
3. There could be a Plugin Manager integrated into DualityEditor that allows browsing available plugin packages.
   • It keeps track of what plugins are currently incorporated into the current Duality project.
   • It allows browsing and "un / installing" plugins into the current Duality project.
   • Could also allow updating them to the latest version.

[BraveSirAndrew]

I didn't get as much time over the weekend to work on this as I was hoping for, but I've managed to do a quick spike of how we might include a NuGet-powered plugin manager inside Duality. I've committed my spike here - https://github.com/BraveSirAndrew/duality/tree/plugin_manager. It's really bare bones right now, has no tests, and is generally super hacky. It uses a command line interface which supports only two commands, -list and -install. List will show all of the packages in the online repository (which I've temporarily set up at https://www.myget.org/F/dualityplugins/), and -install [pluginName] will install the plugin called pluginName.

For version compatibility, the code checks the major version of the Duality assembly against the first dependency found in the nuget package. I'm cheating here by telling NuGet not to install dependencies itself and using the first dependency as a minimum Duality version check instead, as I couldn't find another way to do that. It then pulls in any dependencies manually, although this code is incomplete as my sample plugin doesn't have any dependencies:)

The version thing brings up the question of how version numbers should be treated going forwards, if the plugin manager is to become a real thing. I think some form of semantic versioning (http://semver.org/) is needed, so plugins can be guaranteed to run against a major version.

[lukaspirkl]

Also we should keep in mind the situation when one plugin is depending on another one. For example steering plugin can be used in some AI plugin.

[AdamsLair]

• Duality Versioning. Yes. Will either write a command line tool to increase version numbers or keep track of that myself. To get proper semantic versioning, I'll probably need to do it personally though.
• Version Checks in the plugin manager should probably be opt-out.
• If Duality is newer than a plugin, allow to install it but display a warning ("yellow text severity") that it might be out of date and cause problems.
• If Duality is older than a plugin, don't display it by default, except if opted-out of version checks above.
• I think, @BraveSirAndrew 's approach already checks a plugins dependencies. As @Existent stated, this is of course a requirement, let alone because of editor plugins requiring their core equivalent - in case some authors decide to keep it separate.

[AdamsLair]

Allright, I've given this topic some more thought. Here are my results so far:

• The "Package Manager" should be a fixed (non-plugin) UI module of the Duality Editor. It could be arranged functionally similar to the "Manage NuGet Packages" dialog in the VS plugin.
• It can double as an integrated Auto-Updater, if Core, Editor and Launcher are handled as packages as well. Therefore, they probably should be.
• The set of currently used packages can be stored in a distinct XML file in the projects root directory, similar to the way it is done with VS projects and NuGet.
  • It will be automatically maintained by the Package Manager, though.
• Automatic Package Restore: On startup, the editor checks whether all the packages from its registry file are fully available. The missing ones will be downloaded from the NuGet repo.
• Since the editor can't update itself or currently used binaries, a secondary helper program will be needed for completing updates.
  1. The Package Manager will download all the packages to a temporary "Update" subfolder.
  2. Once finished, it will configure a separate update tool via XML file, close itself and the editor, and run the helper tool.
  3. The tool waits for the editor to shutdown and then copies all the downloaded files to their appropriate locations, overwriting old versions and deleting ones that are scheduled for removal.
  4. After copying, it will rebuild the projects local plugin.
     • If it completes successfully, it will start the editor and shut itself down.
     • If the rebuild failed, display a message to the user that asks him to fix his build, open the plugin project and shut itself down.
  5. On startup, the editor checks for an "Update" subfolder. If existing, it will be deleted.
• Duality Plugin NuGet packages can be identified by defining certain required tags, and checking for a dependency to "AdamsLair.Duality". The repository should be queryable by tag, dependency checking is just making sure it's no accidental match.
  • Release Notes could be evaluated and / or displayed before updating or installing.
• The above mentioned "package file" of the project could also allow to specify additional, custom NuGet repository URLs to query for plugins as well.

Still quite sketchy, but it's a start. Feedback appreciated.

[lukaspirkl]

I really like the direction this is going in.

Are you also planning to support usual NuGet packages? I mean what if plugin will have usual NuGet package as an dependency? Should this be also handled and resolved by build-in package/plugin manager? It could lead to potential version mismatch when two plugins will be using different versions of same package. But on the other hand, it would be quite unexpected that these dependencies won't be resolved.

[BraveSirAndrew]

This is sounding very cool.

@Existent I think so long as the package.xml lists all of the plugins that are installed (which will include all dependencies), NuGet can be made to correctly handle shared dependencies, using the usual semantic versioning rules.

@AdamsLair Using tags to mark packages as Duality plugins might not be necessary if we can find a good place to host just Duality plugins. Additionally though, would this be a problem for dependencies, which might exist outside the Duality repo and wouldn't be marked with the required tag? For example, the scripting plugin now has some dependencies on FSharp assemblies, hosted on NuGet. The plugin manager would have to pull those assemblies down as well, even though they're not tagged.

Where I see this ultimately ending up is in something like Unity's (hope that's not a dirty word around here :) ) package management, where assets can be bundled with a plugin. NuGet lets you run powershell scripts after installing that could be used to move files around. In the case of assets, that would mean moving them to the data folder.

[AdamsLair]

@BraveSirAndrew

Using tags to mark packages as Duality plugins might not be necessary if we can find a good place to host just Duality plugins.

Yep, that's true. But what's the advantage of finding a different place, when the default one is already pretty good? As long as there is no rule against using the NuGet servers to host plugins for a very specific software framework, that is.

The tag solution should also work with plugin dependencies, as those tags would only be the first and easiest indicator of what NuGet packages to show within the Package Manager. As you pointed out, when installing a package, of course all of its dependencies need to be installed and managed, too! One solution: They could be silently installed in the same way, internally handled in the same way, just not being displayed to the user.

like Unity's (hope that's not a dirty word around here :) )

Nah, it's fine. :)

---

I will be unavailable for ~ a week, but I'll try to look into this topic a little more when I return.

[lukaspirkl]

I am not sure about current infrastructure or process how new version is built, but today I found this link http://www.hanselman.com/blog/AppVeyorAGoodContinuousIntegrationSystemIsAJoyToBehold.aspx it could be useful for automatic builds and test runs so I am sharing it.

[BraveSirAndrew]

That looks super cool! It might be worth noting that the current Duality tests most likely won't run on there though, as no VM under the sun (that I could find) virtualises the graphics card, and most Duality tests spin up a dummy OpenGL context.

[AdamsLair]

That looks super cool! It might be worth noting that the current Duality tests most likely won't run on there though, as no VM under the sun (that I could find) virtualises the graphics card, and most Duality tests spin up a dummy OpenGL context.

True. It's partly due to an unfortunate design of the Duality framework that doesn't (and currently can't) separate framework initialization from graphics initialization, and partly due to the fact that some tests actually draw stuff and compare it to reference images.

That said, AppVeyor looks very cool. Will keep that in the back of my mind.

[AdamsLair]

Some more notes:

• In order to support packages that contain Data, Plugins or even root files, there needs to be some kind of information, where certain package files need to go after unpacking.
  • It appears to be hard to maintain directory structures within a NuGet package, since some parts of the files path will be interpreted according to framework versions, etc. and Duality shouldn't interfere with NuGet on this level and shouldn't break NuGet conventions as well.
  • NuGet offers certain directory conventions that could be used.
  • In order to stick with these conventions, all binaries should go directly into the lib folder, content will go into the content folder and copied to Data after unpacking.
    • The Duality Core, Editor and Launcher require special behavior, as their lib folder needs to be copied directly to the root folder, instead of the Plugins folder.
    • In order to deal with this, the target folder for lib contents can be determined depending on whether or not certain tags are present in the packages tag list.
      • If both "Duality" and "Plugin" are present, it goes to the Plugins directory.
      • If only "Duality" is present, it goes to the root directory.
      • If none of them is present, it goes to the Plugins directory, since this is also a valid place for external dependencies anyway, and also the most error-proof location to use by default.
• Using the Nuget.Core API, it is easy to automatically resolve dependency packages, independent from whether they are Duality plugins or arbitrary NuGet packages.
  • Since they will be copied using the same logic as above, there will be no need for additional code handling them.
  • However, certain dependencies are to be ignored, since most plugins probably depend on Duality, OpenTK or similar, and they should not be allowed to pull each of their favorite version into the mix. Version handling will be part of the editor module anyway.
    • There could simply be a blacklist of files not to update that can be generated by asking which files belong to Duality Core, Editor or Launcher. Those are the ones not to change by anyone else, and also not to copy to the Plugins subfolder. Don't allow OpenTK clones in there!
• NuGet provides certain features that Duality doesn't need for package management.
  • Ignore all framework-related settings. Duality packages should be framework-agnostic. This is just unnecessary complexity, as Duality will only move forward and .Net is backwards compatible anyway. Will also be likely to cause confusion and problems when introducing Mono / Multiplatform frameworks.
    • When the same file is provided for a specific framework and without one, favor the one without specific framework version.
    • If there is no unspecific version, use the highest one below Dualitys.

Again, feedback and suggestions are appreciated.

[AdamsLair]

• A good point to start would be defining the package config file of Duality, and the automatic package restore on startup. UI, Updates, Installing, etc. can follow later.
• It will be important to keep track of which packages are currently referenced by the project (including their version), as well as which files belong to each package.
  • Which packages are referenced is easy to maintain as persistent information within the config file.
  • Which files belong to each package shouldn't be stored within the config file, but retrieved from the NuGet repository when needed. Storing it locally would be redundant.
    • If a certain package cannot be found, any file could potentially belong to that package. Uninstalling the package would be impossible.
    • Maybe, there should be a local cache of files belonging to a package that is automatically generated when installing or updating a package. If there is no cache entry, ask online.
• Automatic package restore should only add new files, not delete unknown ones. Not everything is a package, some plugins are just locally compiled, not published, etc.
  • The only occasion where it is okay to delete files is when uninstalling a plugin, but only if each file isn't used by another plugin which is still installed.

By now, after all these concepts, I really want this implemented... just need to find some time to actually do it. If it all works out, this might turn out to be awesome :)

[AdamsLair]

• It's probably a good idea to introduce an abstraction layer between Duality and NuGet.
  • The only access point for NuGet should be a single class in DualityEditor.
  • All other classes, plugins and users will only work with Duality-specific package management classes.
  • The actual implementation remains hidden, direct user interaction with NuGet should never be necessary.
• Only expose a heavily reduced interface to make internal changes in the future easier.
  • Un/Install (Duality) packages
  • Query installed and available (Duality) packages

[AdamsLair]

Status Update: There is a limited prototype in the repository right now. All of this is deactivated by some commented out lines.

• Maintaining a package config file, including repository Url and referenced packages.
• Set up basic Duality Package Management API and implemented parts of it (experimental).
  • Query available packages from the server.
  • Query referenced / locally installed packages.
  • Install and Uninstall a package.
• Set up DualityUpdater tool and ApplyUpdate.xml config file, as well as basic restart-and-update logic

Immediate ToDo:

• Actually get Install and Uninstall right and test it thoroughly.
  • Fix target paths for non-Duality, non-Plugin binaries.
  • Don't just blindly remove files from uninstalled packages, check their usage first.
  • Don't just blindly copy files from installed packages, check their version first. Only overwrite, if newer.
  • Also, make file usage by packages queryable via PackageManager class.
• Build packages for all Duality plugins, editor, launcher and core, put them in a local repository and see what happens.
  • Package Install, Package Uninstall, Updates, Self-Updating, ...
• Upgrade NightlyBuilder to install all packages prior to packing. The binary package should work without internet and contain a base selection of packages in order to run out of the box.

[AdamsLair]

Status Update: Fixed a lot of the above issues and created package specifications of all currently existing Duality modules for testing. Incorporated package building into the NightlyBuilder process.

Immediate ToDo:

• Remove lazy-verify technique entirely, since it has the potential to cause a lot of problems. Re-introduce a verify-all on startup.
  • To make it usable, display a BigTaskDialog during verification and let it happen before DualityEditorApp.Init
  • To keep binary download size to a minimum, no longer incclude any plugins nor launcher pre-installed in the .zip package. Let them download via package manager on startup via PackageConfig.xml, because they will be re-downloaded anyway even if they were already there, to build the local NuGet repo.
  • The Duality binary .zip will only contain the Editor and the least possible amount of dependencies. Only the ones needed to perform the first-start verification. Maybe even the engine Core could be left out, depends on when .Net actually requires its dependencies.
• Start building the PackageManager GUI, both for the user and for debugging.
  • Due to the newly introduced lazy-loading, deep package information won't be available until the packages in question are verified by doing a NuGet install to build up the local repository.
• Let the NightlyBuilder update PackageConfig.xml upon build. It should reference the specific versions of all the latest packages, i.e. the ones that are currently built locally.

[AdamsLair]

Status Update: Package Management is operational in an experimental state. Successful tests for self-updates have been performed with a minimalist binary .zip package that downloads all of the required packages itself on first startup.

Immediate ToDo:

• Start building the PackageManager GUI, both for the user and for debugging.

[Andrea]

Hi there
Great to see the progress on this.
I have a little suggestion, have you ever heard of branches? It is generally a good workflow for long running tasks like this one. :)

[AdamsLair]

Yup, know about branching. Will probably use next time :)

[AdamsLair]

Status Update: Began implementing Package Management user interface in a separate editor plugin. Since the package management API is implemented in the editor itself, others could easily provide their own package manager module while still using the same backend.

Immediate ToDo:

• Rewrite the whole menu handling code in MainForm class, since the current version isn't able to let the user specify where to place a certain element within the menu.
  • Somewhat a distinct task, but since it came up during first tests and will definitely be needed, might as well do it now.
  • Implement some kind of abstract "Menu Model", which is automatically applied to the actual menu hierarchy. Plugins would modify the model, and the MainForm will update itself based on that.
  • Build a distinct class for that, so it's reusable.
• Continue building the PackageManager GUI, both for the user and for debugging.

[AdamsLair]

Status Update: Menu Handling has been rewritten with a new and shiny model / view system. Work is being done on the PackageManager GUI, attempting to implement it in an awesome, asynchronous way right from the beginning. Don't freeze while retrieving online data, be fully usable no matter how much has been retrieved yet.

Immediate ToDo:

• Continue working on the PackageManager GUI

Concept Stuff:

• There definitely should be a "safe update" mode where only updates are applied that are close-to-guaranteed to play well together with updated binaries and existing packages.
  • Try to not ever break stuff by updating!
  • In order to determine compatibility, specifying "version ranges" in NuGet is impractical. A package author cannot possibly know up to which future version his plugin will be compatible. No way.
  • Instead, introduce a semantic version convention for Duality and hope everyone else adapts, once documentation is provided.
    • Versions are officially defined like this: major.minor.build.revision. Assume the following:
      • Revision updates contain only tweaks and fixes that do not change existing behavior in a significant way. No API changes.
      • Build updates may contain changes that modify existing behavior in a way that brings it closer to existing specification. No removal of API, additions are allowed.
      • Minor updates may introduce any kind of change, including removal of certain API, as long as the overall API structure remains the same.
      • Major updates may do just about anything.
    • Edit: There actually is an existing specification. Maybe incorporate that? Although, since major versions are the only ones allowing incompatible updates, that would be quite a lot of major updates during regular development.. not good. Let's try this very-close definition:
      • Major: Major incompatibility possible.
      • Minor: Minor incompatibility possible.
      • Patch: Perfectly compatible.
      • Revision / Build: Insignificant, discourage use.
    • It will be assumed that Revision updates are always safe, Build updates are usually safe, Minor updates are potentially unsafe and Major updates are definitely unsafe.
    • An update is always safe if the release dates of all depending packages are newer than the one of the dependency, despite what the version number rule above suggests. This is especially useful when doing an "Update All" operation.
  • Let the editor check updates using that convention in order to determine whether they are safe / how safe they are.

[AdamsLair]

Status Update: Now correctly handling cases when Duality packages are installed implicitly / as dependencies of other packages. Also doing a lot of user interface stuff for the Package Manager UI. It is actually usable now when it comes to installing and uninstalling. No fancy special cases though, and no updates, let alone safe updates.

Immediate ToDo:

• Check which packages can actually be uninstalled prior to offering that option and failing with an exception.
• Get Update functionality running.
• Display which packages are installed when viewing online packages.
• Display summaries when browsing packages.
• Implement a "Safe Update" functionality for both the whole project and single packages.

[AdamsLair]

Status Update: Updating is done in first iteration, Uninstall checks are done as well. Also, several bugfixes and tweaks have been introduced.

Immediate ToDo:

• Display which packages are installed when viewing online packages.
• Display summaries when browsing packages.
• Implement a "Safe Update" functionality for both the whole project and single packages.
• Once the first iteration of package management is up and running, publish new NuGet packages for all of Duality, minding the new versioning guidelines above and also getting on par with the current master branch.
• Build a fresh binary package and test thoroughly.
• Write some Unit Tests for package management using a local repository and dummy packages.

[AdamsLair]

Status Update: Polished user interface a little bit, updated all packages and entered Beta mode. Package Management is now available for beta testing, both in binary form and on the master branch. "Safe Update" functionality wasn't implemented, as it seemed more and more like a nice2have feature with questionable relevance.

Immediate ToDo:

• Write a detailed documentation on how to publish packages.
• Await beta testing results and potentially fix a lot of bugs.
• Write some Unit Tests for package management using a local repository and dummy packages.

[AdamsLair]

Status Update: Wrote a little bit of documentation on package Management. Still rough, but it's a start.

Immediate ToDo:

• Introduce an "Update All" button to the user interface.
• For outdated Packages, display both the current version and the newest version.
• For outdated Packages, display the release notes of the newest version below the package description.
• Allow to update a package to a specific version, including downgrades.
• Await beta testing results and potentially fix a lot of bugs.
• Write some Unit Tests for package management using a local repository and dummy packages.

[AdamsLair]

Status Update: Tweaked and fixed a lot of Package Management stuff, including most of the above ToDo entries.

Immediate ToDo:

• Make sure the package list always has focus, so scrolling happens there and not in the filter box.
• Introduce an "Update All" button to the user interface.
• Await beta testing results and potentially fix a lot of bugs.
• Write some Unit Tests for package management using a local repository and dummy packages.

[AdamsLair]

I've tweaked the user interface a little bit more and am now closing the issue, as the main part of the system has been done. Bug reports and feature requests on the system now belong to individual issues as usual.