Android: Implement sparse bundle PCK support.

[bruvzg]

Allow to save pack content as individual file with a PCK metadata as a separate file.

• Should fix Loading patch PCK on Android with big project is slow #102998
• Allow using encryption on Android without platform code changes (tested with APK and AAB exports).

Partially superseded #76161

Fixes godotengine/godot-proposals#6675

[bruvzg]

Tested with #102998 MRP, load time with sparse PCK enabled (option in the export settings) decreased from 8570 to 10 msec.

Not sure if there are any benefits on non-Android, but I have enabled the option on all platforms, since implementation is not platform depended.

For the reference, all it is doing is saving PCK header and directory as {name}.sparsepck file and all other files as {name}_{sha256-hash-of-original-name} files in the same directory. Therefore, it avoids slow directory enumeration on Android, since file metadata and list is read {name}.sparsepck (which is fully loaded to memory on load).

[mihe]

It does seem to fix the MRP in #102998, which is great.

I am a bit worried about the ergonomics of this though. It seems like an easy thing to miss that you need to enable this setting if you find yourself suffering from long load times for additional PCKs on Android, which I'm assuming any sizeable project using things like patch PCKs will do eventually.

Seeing as how Android builds are (as far as I can tell) already bundling the individual files in the APK, as opposed to a single PCK file, what exactly is it that this allows you to do differently that couldn't have been done with the "non-sparse" format? Is it about having every file essentially be its own small PCK so they can be encrypted individually, or something along those lines?

Would there be any downsides to using this new format on Android? If not, maybe just having this new format be the default is preferable?

[bruvzg]

what exactly is it that this allows you to do differently that couldn't have been done with the "non-sparse" format? Is it about having every file essentially be its own small PCK so they can be encrypted individually, or something along those lines?

If I understand correctly, seeking the one big PCK file from the APK can also be slow (not sure how relevant it is for the current Android versions). Otherwise, bundling a single non-sparse PCK into an APK should also work.

[bruvzg]

Would there be any downsides to using this new format on Android?

I do not see any.

[mihe]

seeking the one big PCK file from the APK can also be slow

Maybe I'm missing some obvious export setting, but when opening an exported APK, without this new setting enabled, I'm seeing all the assets laid out plainly in the assets folder (or rather their *.remap files along with a .godot folder).

[bruvzg]

Maybe I'm missing some obvious export setting, but when opening an exported APK, without this new setting enabled, I'm seeing all the assets laid out plainly in the assets folder (or rather their *.remap files along with a .godot folder).

Indeed, and it is like this and not a single PCK because it was causing performance issues before.

The only difference with the new option from a current APK layout, it's not enumerating file (and getting file info) from APK, but from a PCK header.

[bruvzg]

So the storage options are:

• Full PCK cons was slow to seek / read when it is in the APK (no supported).
• Files stored in APK as is, file reads are OK, but enumeration/metadata is slow (default).
• Files stored in APK as is, but file list and metadata is stored as PCK header (this PR, a middle ground that should avoid both issues).

[mihe]

Gotcha.

I realize the encryption stuff got intertwined with this, so maybe the format proposed here is simply what has to be done if we want all of that, but would this PR be made any simpler if it had only the PCK header part (i.e. *.sparsepck) that instead somehow referenced the files as laid out in the current APK format? Would it even work?

[bruvzg]

I realize the encryption stuff got intertwined with this

Encryption is basically a free byproduct of the format, and entirely relies on existing PCK encryption code. I have moved some related export code to exposed functions, but without it, it won't be much simpler.

[akien-mga]

Code / architecture changes look great to me.
This is now more self-contained with the sparse PCK path being an implementation detail of the Android export pipeline.

I'd still like @m4gr3d or others in @godotengine/android team to review/test and confirm that this approach is suitable for Android exports. CC @godotengine/xr too as it would affect Quest and Pico.

[mihe]

If it helps, I can confirm that this resolves the issue seen in the MRP of #102998, taking the loading of the patch down from about 12 seconds to a couple of milliseconds.

I can't spot any obvious mistakes or issues in the code, but the encryption part of it is pretty foreign to me.

[m4gr3d]

The flattening of the directory structure and project files rename break projects like https://github.com/GodotVR/godot-openxr-android-surface-javaclasswrapper-example

[m4gr3d]

The code looks good and the feature works as expected from my testing, so this PR looks good to go!

Thanks for this great update!

[akien-mga]

Thanks!