ULTIMA: Split Ultima 8 support into a separate engine

[ccawley2011]

There are two main advantages of having a separate engine:

• The plugin for the separated Ultima8 engine should be much smaller than the combined plugin, which should help platforms with limited RAM.
• Lua is no longer required to build with Ultima 8 support, and the list of dependencies can be further adapted without affecting the other games.

I'll probably need advice about how to handle existing config files, which is why this is currently a draft.

[PaulGilbert]

I have to say that I'm not sure that I see the two points as particularly beneficial, and that there wouldn't be that many users of platforms with limited RAM clamoring to play Ultima 8. Better to keep all the Ultimas together in the current single engine than undoing the efforts and start splitting them up. Particularly as more of the Ultimas are added.. I'm still hopeful of getting to merge in Exult one of these days :)

[digitall]

@PaulGilbert : I understand your point, though https://bugs.scummvm.org/ticket/14390 means that the engine plugin size is causing issues for some of the more RAM limited ports with just the detection.

[ccawley2011]

Is there a major advantage to keeping them all in a single engine? Not only do the four subengines share very little code with each other (for Ultima 8, it was relatively trivial to remove the notable dependencies in #4582 and #5213, which just leaves the STL replacements and some detection/MetaEngine boilerplate code), but xu4, Nuvie and Pentagram seem to be designed differently enough to each other that sharing code between them might not be practical.

[OMGPizzaGuy]

I'm torn on this one. We have some engines that split: sword1, sword2, sword25 - Others keep together: mm, sccumm, agos
I assume ones kept together do tend to share more. I guessing we could follow suit with more add_engine lines in configure.engine to split it out a bit without moving files, but does the lead to a bunch of #ifdef ENABLE_ULTIMA8 sections throughout the code?

[dreammaster]

I just don't want to see a situation if/when the other games of the series get supported and we end up with eight or more (depending on if you count Escape from Mt Drash and/or Akalabeth) seperate game folders for the games. Keeping them within a single folder/namespace just seems cleaner to my mind. Sword is kind of a special case, since they were some of the original earliest engines created, so Sword25 later was easiest as it's own folder as well.

[sev-]

I am also torn on this. Maybe we could consider adding ultima8 and perhaps others as subengines?

In those examples, sword1 and sword2 are very different engines with little in common, sword25 is a fan-made project on a completely different technology, so no way to merge them. SCUMM shares 90% of the code with the previous versions on each increment.

Coming back to a subengine. As I understand, the current plugin is too big to fit in Wii's memory, so turning Ultima8 as a subengine could potentially help mitigate the issue.

[dreammaster]

Good idea. I'll take care of setting them up as sub-engines now

[dreammaster]

Ok, I've just committed in separating the games into sub-engines. This only counts the main overall games, so Martin Dreams & Savage Empire come under Ultima 6, and likewise Crusader/Remorse come under ultima8

[ccawley2011]

Unfortunately, having multiple subengines only partially solves the problem since the engine plugin will still be very big if all of the dependencies are met. They're better suited to cases like the Crusader games, which both share significant code with the base engine and add a notable amount of new code, but has higher requirements than the base game.

I wouldn't imagine that we'd end up with 8+ separate engines - the Ultima I engine seems to have been designed to allow the first 5 games (and possibly Escape from Mt Drash and Akalabeth as well) to be supported using a common base. Ultima VI, VII, and VIII (and IX) appear to be different enough that there isn't as much to be gained from a unified engine, and would be closer to the Broken Sword examples.

[sev-]

@ccawley2011 could you please elaborate a bit? E.g. if we enable only, say ultima1 and ultima4, will it sill be big? What problem specifically are we trying to solve here? I had an impression that it is ultima8 which is (a) bulky (b) will not run on small systems, as a result, it prevents running any ultima engine game on those smaller systems.

[ccawley2011]

@ccawley2011 could you please elaborate a bit? E.g. if we enable only, say ultima1 and ultima4, will it sill be big? What problem specifically are we trying to solve here? I had an impression that it is ultima8 which is (a) bulky (b) will not run on small systems, as a result, it prevents running any ultima engine game on those smaller systems.

The issue is that if a platform supports highres and 16bit, it would be reasonable for all four subengines to be enabled, and the plugin would be very large as a result. It should also be possible to remove those dependencies at a later date, which means that more platforms would be affected.

There's no special reason for starting with ultima8 other than because it was the easiest to decouple from the rest of the engine. I'll tackle nuvie at a later date.

[dreammaster]

Hmm.. I don't suppose in such cases there's a way for an engine to be split up across several plugin files? Or if not currently, is it something that could be done in the future? Such as maybe with an option in configure.engines to specify a sub-engine should be in it's own plugin file if plugins are turned on? Such a solution would be better long term, since other engines are going to be just as affected for small systems. Things like tSage, Mads/M4 (eventually) will have hardcoded logic for quite a few games. Or the MM engine for that matter, as/when later games get added. I just don't want to end up in a situation where we have a cluttered approach in engines where some games of a series are in one folder, but others arbitrarily aren't just based on size.

[ccawley2011]

Hmm.. I don't suppose in such cases there's a way for an engine to be split up across several plugin files? Or if not currently, is it something that could be done in the future?

That would be nice, but I don't know how practical it would be given the limitations of the builtin ELF loader. I don't think that's beneficial in this case though, since Ultima 8 doesn't share any code with the other games.

[PaulGilbert]

I don't think the percentage of code share is the issue, it's about consolidation and keeping series organised, both in their location within engines/, as well as the namespaces. As I said, over time more games in series will be added (hopefully), such as Exult for Ultima VII, and in other series like Might and Magic 6/7/8, and even though the codebases will be distinct, I'd still like to see them remain categorised under a single overall engine.

I'd see the ability to split an engine up into several plugins would be a better long term solution. I'm not familiar with the current code that handles building as plugins, but presumably the create_project (or equivalent makefile code) could detect a new flag in configure.engines that would indicate a sub-engine should be built as it's own project/plugin file (when building with plugins). So we'd get ultima.dll as well as ultima_ultima8.dll. Then the detection code that figures out the correct plugin to use would need minor changes to ensure it can handle multiple dlls having the same engine Id.

This, in my opinion, if doable, would be a better long term solution than fragmenting the engines.

[ccawley2011]

I don't think the percentage of code share is the issue, it's about consolidation and keeping series organised, both in their location within engines/, as well as the namespaces. As I said, over time more games in series will be added (hopefully), such as Exult for Ultima VII, and in other series like Might and Magic 6/7/8, and even though the codebases will be distinct, I'd still like to see them remain categorised under a single overall engine.

I still feel that having them separated is better for maintainability than having a combined engines. Introducing subengine plugins would be a much more complicated solution than just splitting it up, and would make it harder to reduce the memory footprint on devices with limited RAM.

For reference, the only changes needed besides changing the includes and namespaces were providing separate detection and MetaEngine subclasses and copying the C++ standard library replacements.

[PaulGilbert]

I respect that. And my own feeling is for better maintainability of the codebase is to keep series grouped together, and not pollute the engines/ just to satisfy the existing plugin system of one particular system. In any case, we seem to be at loggerheads on the matter. So I'll leave it up to the leaders to decide one way or the other. Of course, it need not be a permanent thing. If, for example, it's decided to split up the engine, it could always be merged back in later on if the plugin system gets improved.

[OMGPizzaGuy]

Considering the conversation I think I am on board with the split again. However, I'll state what I hope is obvious: this should be held off until after 2.8 release.