Allow load custom IMG containers

[TheNormalnij]

This feature will save client RAM memory and model replacing time.

API:

-- load IMG file
img = engineLoadIMG( "file.img" )
img = EngineIMG( "file.img" ) -- OOP

-- Get files count in IMG
filesCount = engineImageGetFilesCount( img )
filesCount = img:getFilesCount() -- OOP
filesCount = img.filesCount -- OOP

-- Get file list
fileList = engineImageGetFiles( img )
fileList  = img:getFiles() -- OOP
fileList  = img.files -- OOP

-- fileList  = {
--     [fileID] = "fileName";
--     [nextFileID] = "fileName2";
-- }

for fileID = 1, filesCount  do
   -- actions
end

-- Get any file from IMG
fileContent = engineImageGetFile( img, "model.dff" ) -- You can use fileName
fileContent = engineImageGetFile( img, 1 )                  -- ... or file ID
fileContent = img:getFile( "model.txd" ) -- OOP
fileContent = img:getFile( 2 ) -- OOP

-- Add image to GTA:SA stream
engineAddImage( img )
img:add() -- OOP

-- Remove image from GTA:SA stream
engineRemoveImage( img )
img:remove() -- OOP

-- Link model to image file
-- engineImageLinkDFF( element img, string/number fileName/fileID, number modelID )
-- engineImageLinkTXD( element img, string/number fileName/fileID, number modelID )
engineImageLinkDFF( img, "file.dff", 400 ) -- You can use fileID (faster) or file name, 
engineImageLinkTXD( img, "file.txd", 123 )
img:linkDFF( "file.dff", 400 ) -- OOP
img:linkTXD( "file.txd", 123) -- OOP

-- Restore model default image
engineRestoreDFFImage( 400 )
Engine.restoreDFFImage( 400 ) -- OOP

engineRestoreTXDImage( 123 )
Engine.restoreTXDImage( 123 ) -- OOP

-- Utility function will return model texture ID
textureID = engineGetModelTXDID( 400 )

How to replace model with this feature:

local img = IMG( "infernus.img" )

if not img then
    return false
end

if not img:add() then
    return false
end

local infernusModelID = 411
local infernusModelTxdID = engineGetModelTXDID( infernusModelID )

img:linkDFF( "infernus.dff", infernusModelID )
img:linkTXD( "infernus.txd", infernusModelTxdID )

-- Apply changes
engineRestreamWorld()

Test:

1. Start test resource
2. Spawn infernus cv 411

Limits:
You can load 2 IMG's. It's GTA limitation, but I hope limit can be changed.
IMG size can't bigger that 1GB (MTA limitation)

[TheNormalnij]

local img = engineLoadIMG( "infernus.img" )
local infernusModelID = 411
local infernusTxdID = 20000 + engineGetModelTXDID( infernusModelID )
engineSetModelIMG( infernusModelID, img, 1, 551 )
engineSetModelIMG( infernusTxdID, img, 552, 571 )

A new way to replace model. This way uses separate thread to read model file.

Edit: it was debug code

[TheNormalnij]

I thought i can allocate new object models only after img functions. I will add another PR only with this feature.

[Pirulax]

The thing with engineLoad* functions is that they check the checksum of the files twice(or at least one) on the main thread(so they can return immediately).

[Pirulax]

Theres an issue, called CClientModelCacheManager. It keeps lots of vehicle/ped/weapon/upgrade models in memory
The issue is that it doesnt account for the size of the models now (as it wasn't designed to handle custom engine images). For example: By default a vehicle model is ~100 kbytes, but now, with custom models its, more like 4-5 MB / model, and you get... 250-300 MB that wont be freed, and since thats > 256 MB streaming limit the game starts glitching out (tries to unload models to free up memory, but it cant because vehicles are referenced by the cache manager).

Heres the stream memory => cache memory mapping. It's totally wrong, as vehicle models can be a few megabytes in size now.

mtasa-blue/Client/core/CModelCacheManager.cpp

Lines 222 to 228 in 3f93481

	// Adjust cache numbers depending on amount of streaming memory allocated
	//
	// 64MB streaming = 2+1 MB for peds & vehicles 9 peds + 7 veh
	// 96MB streaming = 4+4 MB for peds & vehicles 18 peds + 28 veh
	// 128MB streaming = 8+8 MB for peds & vehicles 36 peds + 56 veh
	// 256MB streaming = 16+8 MB for peds & vehicles 72 peds + 56 veh
	//

The solution is to either:

• Remove it. (As I dont see the point of keeping it, we should let the streamer handle model loading)
• Refactor it to account for model size. (Currently its designed to use ~100 megs max)

I did a test (removed all the code of it), and everything seems to work just fine.

@Dutchman101 there is some crash fixing code (related to vehicle upgrades), do you know what it fixes?

@ccw808 Any idea what it was built for, what issues does it fix? Edit - Used git blame in VS, and found commit the commit in which it was added (fefd088). I really want to just remove it, but it seems like there's a good reason for its existence. Sadly it uses way too much RAM with this PR. (And also glitches out the game, as these models basically can't be freed by GTA)

[Dutchman101]

@Dutchman101 there is some crash fixing code (related to vehicle upgrades), do you know what it fixes?

I don't really know, but because that crash fix uses report ID 8544 i can tell (from metrics) that approximately 70 users have had it trigger during the past month. Maybe it's very old but after all it's still firing for those users, so i guess if it was cut out they would've crashed there.

But as if it appears to be the report ID is only used in a certain condition (not the entire crashfix - just "if LowestUnsafeUpgrade has fallen"), true averted crash count will probably be higher. So unless you understand the crash reason and how to fix it in a better way I'd advise not to touch it.

[Pirulax]

Yeah, but the issue is that, people now may replace these upgrades with the img, and if the upgrades are big (in file size), then this will use a lot of memory (as from what I understand all upgrades are pre-streamed).
Anyways, I'll leave that crash fix in, and if someone has issues with it, then it could be investigated further.

[Disinterpreter]

Is it ready? I think it is a useful feature.

[Xenius97]

Is it ready? I think it is a useful feature.

No, but on the wiki is an usefun function that works as well.
https://wiki.multitheftauto.com/wiki/EngineLoadIMGContainer

[TheNormalnij]

Is it ready? I think it is a useful feature.

No, but on the wiki is an usefun function that works as well. https://wiki.multitheftauto.com/wiki/EngineLoadIMGContainer

This feature is ready now

[patrikjuvonen]

Merge conflicts must be resolved.

[TheNormalnij]

Fixed

Stale

[Xenius97]

I've tested it, everything looks great. Memory usage is only 700mb when 668mb car pack img container is loaded.
This will solve most of out of memory crashes by models. Good job.

[Xenius97]

@TheNormalnij For objects seems not working properly.
When you replace collision you need full reconnect instead of engineRestreamWorld, but when you load collision too, model will not have textures.

Without collisions:

Collisions loaded, used only restream

After reconnect:

Test resource:
buildings.zip

[TheNormalnij]

Can be avoided via engineReplaceCOL is setTimer

[Xenius97]

Can be avoided via engineReplaceCOL is setTimer

Yes, it's working fine with timer, tested. Maybe put a warning/infobox into wiki when this will be merged.

But i've found another thing, when you load vehicle model to incomatible model (eg. car mod to bike) game will instant crash.
Don't know this happens via regular model replacement too, or not.

[TheNormalnij]

Yes, it's working fine with timer, tested. Maybe put a warning/infobox into wiki when this will be merged.

#2866 makes timer unnecessary

But i've found another thing, when you load vehicle model to incomatible model (eg. car mod to bike) game will instant crash.
Don't know this happens via regular model replacement too, or not.

I think, we can't fix all cases. Any additional checks will slow down performance, when you link IMG files to models.

[Xenius97]

I've noticed something, all engine* functions are supporting raw data except this. Possible to add raw data support to engineLoadIMG? Cause if you wan't to protect your models, you need to compile it with some method like tea, aes128 and impossible to load if function doesn't supporting it.

[Pirulax]

What do you mean by "raw data"?
The point of all of this is to stream data from the disk, rather than load it into memory.
If we were to keep it in memory, then it would be just a worse engineLoad* replacement.
I suggest that instead of trying to encrypt your models, just do DRM.
While, it would be possible to somehow integrate encryption into this function I don't think it would solve the core problem.