Android: Access to files from APK is boosted #1562

Merged
merged 3 commits into from Nov 12, 2012

Conversation

Projects
None yet
2 participants
Contributor

mingulov commented Nov 8, 2012

Open APK file just once and cache a file list information (name and position inside archive). Also a general ZipFile helper class is created to read multiple files from Zip archives - might be used for another platforms if needed.

Previously it was so for each separate file access operation:

  • Open zip archive, initially parse its structure (unzOpen)
  • Locate the particular file there by a linear search through full archive (unzLocateFile).
    So if file does not exist - still an archive has to be handled fully, even slower than a file exists.
  • Read file.
  • Close zip archive

After this commit:

  • Once - open zip/apk file and collect file list information
    (average consumed time - the same like a search for 2 different files by unzLocateFile)
  • When needed - directly retrieve file position inside the archive, setting it up to the zip reader and reading file

ZipFile class is located in support/zip_support/ZipUtils.h and .cpp to prevent creation of a new files and adding them to multiple project files.

std::map is used to store file list information, if supported by a platform - might be changed to std::unordered_map locally (defined at ZipUtils.cpp).

ZipFIle class declaration uses private implementation to prevent overloading header with a lot of other headers (unzip.h etc).

Speedup is visible even on just a simple TestCpp project on TextureCacheTest. On more complex projects with a lot assets / multiple loading / reloading operations - speedup is much more significant.

@mingulov mingulov Android: Access to files from APK is boosted
Open APK file just once and cache a file list information (name and
position inside archive). Also a general ZipFile helper class is
created to read multiple files from Zip archives - might be used
for another platforms if needed.

Previously it was so for each separate file access operation:
- Open zip archive, initially parse its structure (unzOpen)
- Locate the particular file there by a linear search through
  full archive (unzLocateFile).
  So if file does not exist - still an archive has to be handled
  fully, even slower than a file exists.
- Read file.
- Close zip archive

After this commit:
- Once - open zip/apk file and collect file list information
  (average consumed time - the same like a search for 2 different
  files by unzLocateFile)
- When needed - directly retrieve file position inside the archive,
  setting it up to the zip reader and reading file

ZipFile class is located in support/zip_support/ZipUtils.h and .cpp
to prevent creation of a new files and adding them to multiple
project files.
52ffc9c

@minggo minggo commented on an outdated diff Nov 9, 2012

cocos2dx/support/zip_support/ZipUtils.cpp
+ do
+ {
+ CC_BREAK_IF(!m_data->zipFile);
+ CC_BREAK_IF(fileName.empty());
+
+ ZipFilePrivate::FileListContainer::const_iterator it = m_data->fileList.find(fileName);
+ CC_BREAK_IF(it == m_data->fileList.end());
+
+ unz_file_pos filePos = it->second;
+
+ int nRet = unzGoToFilePos(m_data->zipFile, &filePos);
+ CC_BREAK_IF(UNZ_OK != nRet);
+
+ char szFilePathA[UNZ_MAXFILENAMEINZIP + 1];
+ unz_file_info FileInfo;
+ nRet = unzGetCurrentFileInfo(m_data->zipFile, &FileInfo,
@minggo

minggo Nov 9, 2012

Contributor

It seems that szFilePathA is not needed, can we pass NULL to unzGetCurrentFileInfo()?

@minggo minggo and 1 other commented on an outdated diff Nov 9, 2012

cocos2dx/support/zip_support/ZipUtils.cpp
+
+ unz_file_pos filePos = it->second;
+
+ int nRet = unzGoToFilePos(m_data->zipFile, &filePos);
+ CC_BREAK_IF(UNZ_OK != nRet);
+
+ char szFilePathA[UNZ_MAXFILENAMEINZIP + 1];
+ unz_file_info FileInfo;
+ nRet = unzGetCurrentFileInfo(m_data->zipFile, &FileInfo,
+ szFilePathA, sizeof(szFilePathA) - 1, NULL, 0, NULL, 0);
+ CC_BREAK_IF(UNZ_OK != nRet);
+
+ nRet = unzOpenCurrentFile(m_data->zipFile);
+ CC_BREAK_IF(UNZ_OK != nRet);
+
+ pBuffer = new unsigned char[FileInfo.uncompressed_size];
@minggo

minggo Nov 9, 2012

Contributor

Can cache uncompressed_size too?

@mingulov

mingulov Nov 9, 2012

Contributor

Thank you, a call to unzGetCurrentFileInfo at ZipFile::getFileData is removed, uncompressed_size is cached at ZipFile::setFilter together with file pos info.

@minggo minggo and 1 other commented on an outdated diff Nov 9, 2012

cocos2dx/platform/android/CCFileUtils.cpp
@@ -43,6 +45,7 @@ CCFileUtils* CCFileUtils::sharedFileUtils()
{
s_pFileUtils = new CCFileUtils();
s_strResourcePath = getApkPath();
+ s_pZipFile = new ZipFile(s_strResourcePath, "assets/");
@minggo

minggo Nov 9, 2012

Contributor

s_pZipFile is not deleted?

@mingulov

mingulov Nov 9, 2012

Contributor

Fixed, will be deleted at CCFileUtils::purgeFileUtils together with s_pFileUtils.

Contributor

minggo commented Nov 9, 2012

It is an awesome work.
Thank you.

Owner

mingulov commented on efb61a8 Nov 9, 2012

Thank you for the review, an updating commit is pushed.

Also 'static string s_strResourcePath' is removed from cocos2dx/platform/android/CCFileUtils.cpp - to prevent misunderstanding. If somebody wants to use s_strResourcePath as a separate ZIP resource file and add himself 'setResourcePath()' method, he should re-create s_pZipFile.

Thank you.
I will test it ASAP.

@mingulov mingulov ZipFile - speedup initialization for zip archive by 30-50%
unzip.h/.cpp - unzGoToFirstFile/unzGoToNextFile retrieves information
about the current file, so it is possible to remove redundant calls
to unzGetCurrentFileInfo and speed up general execution.

unzip.h/.cpp are modified accordingly by creation unzGoToFirstFile64
and unzGoToNextFile64 with file info (unz_file_info64) and name as
possible output parameters.

ZipFile::setFilter (it is used to create a file list at ZipFile
constructor) - redundant calls to unzGetCurrentFileInfo64 are removed,
so briefly the time required to generate a zip file list is something
like 1 average old unzLocateFile call.
f73568d

@minggo minggo added a commit that referenced this pull request Nov 12, 2012

@minggo minggo Merge pull request #1562 from mingulov/apk
Android: Access to files from APK is boosted
68fc888

@minggo minggo merged commit 68fc888 into cocos2d:gles20 Nov 12, 2012

@minggo minggo added a commit to minggo/cocos2d-x that referenced this pull request Nov 19, 2012

@minggo minggo fixed #1562:fix a memory leak in CCLabelBMFont baa0a45

@minggo minggo added a commit to minggo/cocos2d-x that referenced this pull request Nov 19, 2012

@minggo minggo issue #1562:update TileMapTest 74e9b93

@rodrigogolive rodrigogolive added a commit to rodrigogolive/cocos2d-x that referenced this pull request Nov 21, 2012

@minggo @rodrigogolive minggo + rodrigogolive fixed #1562:fix a memory leak in CCLabelBMFont f5ac0fe

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 28, 2014

@minggo minggo Merge pull request #1562 from mingulov/apk
Android: Access to files from APK is boosted
6be9326

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 28, 2014

@minggo minggo fixed #1562:fix a memory leak in CCLabelBMFont c235c72

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 28, 2014

@minggo minggo issue #1562:update TileMapTest b9cf1ac

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 29, 2014

@minggo minggo Merge pull request #1562 from mingulov/apk
Android: Access to files from APK is boosted
eec939a

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 29, 2014

@minggo minggo fixed #1562:fix a memory leak in CCLabelBMFont 90b23f0

@angeltown angeltown pushed a commit to angeltown/cocos2d-x that referenced this pull request Apr 29, 2014

@minggo minggo issue #1562:update TileMapTest 8bed96b

@minggo minggo added a commit that referenced this pull request May 4, 2014

@minggo minggo Merge pull request #1562 from mingulov/apk
Android: Access to files from APK is boosted
d4aeb80

@minggo minggo added a commit that referenced this pull request May 4, 2014

@minggo minggo fixed #1562:fix a memory leak in CCLabelBMFont 9dc5302

@minggo minggo added a commit that referenced this pull request May 4, 2014

@minggo minggo issue #1562:update TileMapTest 38232e4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment