Loading patch PCK on Android with big project is slow

[mihe]

Tested versions

Reproducible in: 4.4.beta [93d2706]

System information

OnePlus 8T, Android 14

Issue description

When loading patch PCKs on Android currently, it seems to take a long time to load the first patch when you have many files in the project, which upon investigation seems to stem from the changes made in #90425.

When profiling the MRP (consisting of 5000 barebones scene files) through Android Studio it would seem that it spends about 30 seconds (on my OnePlus 8T) in PackedSourceDirectory::try_open_pack, which is what gets run (here) when you first load a non-main pack.

More specifically, it seems to spend a lot of time dealing with android::ZipAssetsProvider when doing things like DirAccessJAndroid::dir_exists as part of DirAccess::open and DirAccessJAndroid::current_is_dir as part of DirAccess::get_directories.

Here's what the flame graph of the MRP's startup looks like:

Here's a bottom-up view:

Here's a zoomed in view of a few DirAccess::get_directories calls:

Here's a zoomed in view of a few DirAccess::open calls:

Steps to reproduce

I've included a patch PCK in the project, under patches/patch.pck. All it does is change the label in main.tscn to say "Patched" instead of "Not patched".

I've also included the script (generate.py) used to generate the scene files under generated_scenes.

1. Open the MRP
2. Deploy it on an Android device
3. Note the time it takes before you're shown main.tscn

Minimal reproduction project (MRP)

android-patch-performance.zip

[mihe]

CC @tracefree @bruvzg.

[mihe]

Note that there's some issue with loading the patch from the editor currently (which I need to report separately) where it'll happily load the PCK but not actually result in the patched scene file being what's shown, even when doing a deep ignore on the resource cache.

I don't think that particular issue is a regression from #90425, but I figured I should mention it, in case it causes any confusion.

EDIT: Reported the above mentioned issue as #103046.

[tracefree]

Ah yes, sorry about that, I'm afraid this is a consequence of my PR being a bit of a hack! I think that either resource packs and filesystems need a larger overhaul so that the bug #90425 fixes can be resolved in a better way that doesn't require scanning the whole project, or maybe the process can be optimized and sped up somehow on Android?

[mihe]

or maybe the process can be optimized and sped up somehow on Android

Yeah, this seems to be particularly bad on Android. I don't know much about the Android side of things, but judging by the various system APIs in those call stacks it would seem that it's messing around with ZIP files a bunch, which seems somewhat inefficient.

Maybe there's some low-hanging fruit in terms of caching that could help improve things?

[bruvzg]

APK is a ZIP file, and Android do not extract it when installing, so some overhead is expected. But seems Android FS handler doing something wrong, current_is_dir should not iterate over all files.

[bruvzg]

		val fileName = ad.files[idx]
		// List the contents of $fileName. If it's a file, it will be empty, otherwise it'll be a
		// directory
		val filePath = if (ad.path == "") fileName else "${ad.path}/${fileName}"
		val fileContents = assetManager.list(filePath)
		return (fileContents?.size?: 0) > 0

Taking a full directory listing just to check if it's a directly doesn't look like a good idea.