OutOfMemoryException when trying to rip small amount of assets from a big game

[SunSerega]

I have one huge game with ~30GB of assets, and it seems even my 64GB of RAM is not enough.
I'm getting a bunch of

[Warning] System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. 
[Warning]    at AssetStudio.BundleFile.CreateBlocksStream(String path) in L:\GIT\AssetStudioMod\AssetStudio\BundleFile.cs:line 174 
[Warning]    at AssetStudio.BundleFile..ctor(FileReader reader, String specUnityVer) in L:\GIT\AssetStudioMod\AssetStudio\BundleFile.cs:line 127 
[Warning]    at AssetStudio.AssetsManager.LoadBundleFile(FileReader reader, String originalPath) in L:\GIT\AssetStudioMod\AssetStudio\AssetsManager.cs:line 256 

But, in my case, I'm trying to execute the -t audio, and I doubt there will be even 1GB of audio in the assets. What can be done to immediately discard everything that will be filtered out, right after loading a .bundle file?

[aelurum]

Have you tried not to load the whole game, but select and load a certain number of files?

[SunSerega]

Almost all of those 30GB are in StreamingAssets\aa\StandaloneWindows64, in 3k separate files. It will probably work if I feed it, not the folders, and instead separate files...
But that would be thousands of instances of the .exe file, which is slow. IDK if it's maybe too hard to implement this within AssetStudio, but it would be much better if it could write and discard things it doesn't need and things it has already processed.

[HallowSL]

There's definitely a memory leak happening. Trying to open this https://mega.nz/file/9kQjlLyZ#-UYE2fcGA1po-b12h6d7AtnbkoEo0YmW2wdfiESjR0A causes memory usage to spike. It opens fine in Perfare's AssetStudio with the memory usage one would expect from a file this small.

Haven't tested in the other net builds, but I'm using the net8 GUI version.

[aelurum]

https://mega.nz/file/9kQjlLyZ#-UYE2fcGA1po-b12h6d7AtnbkoEo0YmW2wdfiESjR0A

This is likely a different case.
It has a modified AssetBundle asset. (some Tencent game, I guess?)
Since AssetStudioMod tries to read more fields from an AssetBundle asset than the original Studio, this issue occurs.
AssetStudioMod expects to read an array size, but instead it gets some garbage value.
And while net472 just throws an error when trying to allocate an array of e.g. 15gb, .net5+ already supports such large arrays, so it doesn't see any problems with allocating such amount of memory)

But yeah, it would be nice to fix that someday.

[aelurum]

Oh, rechecked the AssetBundle v5.4.0 structure.
And your asset wasn't modified after all.
I just really missed one field for v5.4, lol
I'll fix it soon.

[aelurum]

btw, @SunSerega
You can also try to use AssetStudio by Razmoth.
There you can create an asset map and a cab map, which allows you to filter assets before import.
And it's also possible to allow loading of only selected asset types in the settings.
You can read the tutorial here: https://gist.github.com/Modder4869/0f5371f8879607eb95b8e63badca227e

[SunSerega]

I looked into how this fork does things after writing my comment, and I'm pretty sure it at least also filters assets before loading them. What I'm lacking is it ability to extract and unload assets right after loading, instead of loading all assets at once, before starting to extract them.

[SunSerega]

Actually, I was wrong, Razmoth's fork doesn't use >200 MB at any moment - unloading things immediately as they are extracted. And it does so without any maps (I'm still not sure what they are for, that gist didn't explain that to me).
Tho, there are other problems that weren't the case with this fork... Well, it's probably easier to fix that one for my particular case, thanks for pointing me in the right direction.

Should I close the issue? Or does it still have value to you?

[aelurum]

Glad this helped you.
Yeah, I'll close the issue for now.
Maybe I'll return to it sometime later.