Binary model format

[Luomu]

Adds the ability to save and load models in a binary format. This is good because:

• binary models load faster
• allows creating a version of pioneer without Assimp (it's a big dependency)
• more processing options could be added to the assimp loader, while the binary loader can stay fast & straightforward.

This is not really plugged into the game yet, we need to first decide how it should work. For testing purposes, modelviewer dumps a binary version (.sgm) of every model it loads into userfiles/ModelBinTest/. Pressing B attempts to load a binary model from data/models/whatever/. A .sgm must be located where the .model file is, because textures and patterns are not part of the .sgm file.

Doing binary conversions automatically is probably not a good idea, the model cache was a bit of a headache in the LMR days. Binarizing models should be some kind of a built-time process (or don't enable it at all, it's still a nice tool to have).

A FileSystem question: How to create a Serializer::Reader with FS? I had to use straight fopen, but that won't work if the .sgm file is in a zip or something.

[johnbartholomew]

A FileSystem question: How to create a Serializer::Reader with FS? I had to use straight fopen, but that won't work if the .sgm file is in a zip or something.

The real question is: given that Serializer::Reader already works from a big block of data, why can't you give it a ByteRange object that you can get straight from FileSystem? And the answer is because no one bothered to write that code yet :-)

[fluffyfreak]

This is a great idea. Would it be possible to put some other data into the binary blob as well?
I'm thinking of the BVH tree for collisions. We spend a LOT of time building and rebuilding them during loading and a few other places which it would be good to avoid. So if you're converting everything to binary anyway it implies that we could use it as a pre-processing step and avoid regenerating that data at runtime for binary files.

That would make loading a lot faster.

[johnbartholomew]

Should be quite easy to load from FileSystem without going through a FILE* now. Code should be something like:

RefCountedPtr<FileSystem::FileData> bindata = info.Read();
if (bindata) {
    Serializer::Reader rd(bindata->AsByteRange());
    Model *model = CreateModel(rd);
    bindata.Reset();
}

[robn]

I'm starting to think a standalone tool that incorporates all the dev stuff is probably the way to go, so this, ModelViewer, maybe ObjectViewerView, a distance field font generator (partly written on a branch somewhere) and so on.

@johnbartholomew rightly pointed out to me that this could be bit of a pain for modellers during development, as it adds an extra step. It also occurs to me that it needs to be possible to extend the format without breaking old models, otherwise we break mods that are distributing binary models (I've not read the code in full yet, so maybe you already do this).

@nozmajner, opinion? If you had to run a program after saving your model to convert it to something Pioneer can use, would that be a pain?

[fluffyfreak]

There are already a number of extra steps to getting things like ships ingame, perhaps this could even help alleviate that.

Open your model in ModelViewer, we can parse the nodes looking for tags to decide if it's a ship, station, etc or the user can choose to create an ingame object.

An Export Ship button if you like, that then writes the Lua/JSON/whatever file. Bonus points if we let the user do things like configure rotation rate, thrust and other stuff you'd normally have to type by hand.

If anything that extra step then makes things easier.

[fluffyfreak]

I should add that I'm not suggesting we do that for this PR :) Just another way of looking at it.

[nozmajner]

I don't mind the extra step if that means better performance.
What if it's integrated in the model viewer? Maybe in a special build or something, if the aim is to ditch the external model parser (or what it's called), so we can output the model after checking it.

[fluffyfreak]

@Luomu I've brought your sg-binarize-pr up to master and fixed an issue with the new shields and reloading from binary with them. It's all on my fluffyfreak:sg-binarize-pr branch if you want it.

[Luomu]

Oh hey, missed some discussion.

I'm starting to think a standalone tool that incorporates all the dev stuff is probably the way to go

Agreed, but things are hard to separate from the game so it's a long way away. If you can add a new command line option to the game (pioneer --dump-model foobar) it's easier.

It also occurs to me that it needs to be possible to extend the format without breaking old models

Would be nice, but I need more specs for that. If someone implements a custom node then that's already supported, they have to implement the save function and register a loader function.

It's all on my fluffyfreak:sg-binarize-pr branch if you want it.

will check!

Going to merge this soonish to keep bitrot away. But I don't think this is going to be used in pioneer binary distributions quite yet.

[Luomu]

Going to merge this soonish to keep bitrot away.

Done, but obviously need to keep working on this for the same reason