Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Archive Reader implementation #16129

Merged
merged 14 commits into from Jun 13, 2023

Conversation

lemonade-dm
Copy link
Contributor

@lemonade-dm lemonade-dm commented Jun 7, 2023

Archive Gem changes

Adding complete implementation of the new Archive ArchiveReader API

This pairs with the implementation of the ArchiveWriter API and
therefore can be used to write and O3AR archive files.

The Archive Reader supports the following operations

  1. ListFileInArchive - This function can query a file within an
    archive by file path or file token which was queried previously from
    a ListFileInArchive call
  2. ExtractFileFromArchive - This function can extract file from the
    archive using either the relative file path of the contained file or
    the file token queried from a call to ListFileInArchive
    The file is extracted to a user provided allocated buffer via an
    AZStd::span<AZStd::byte> paramter
  3. EnumerateFilesInArchive - This functions enumerates all non-deleted files within the archive and invokes a user provided callback with the ArchiveListFileResult structure for each file. The ArchiveListFileResult structure is the same type returned by ListFileInArchive

Archive Factory addition

Added an Archive Reader and Writer Factory implementation

The Archive Reader Factory allows IArchiveReader instances to be created
by external Client gem modules by depending on the Archive.Clients.API target

The Archive Writer Factory allows IArchiveWriter instances to be created
by external Tool gem modules by depending on the Archive.Tools.API
target.

This other Gems such as the AssetValidation or some other library such as the AssetBundler to create Archive class instances needed to interact with the API

Compression Gem changes

Adding a new interface function of GetCompressionAlgorithmName to the ICompressionInterface/IDecompressionInterface interfaces.
Overriding this function can used to provide a human readable name associated with the Compression algorithm

The plan for this change is to make it easier for a future CLI to allow uses to specify the compression algorithm to use when adding files to an archive.

How was this PR tested?

Added UnitTest for the ArchiveReader functionality to the Archive.Editor.Tests module
Validated the existing ArchiveWriter test still passes.

…terfaces

Implementors of `ICompressionInteraface` or `IDecompressionInterface` can now implement the `GetCompressionAlgorithmName` function to provide a human readable name associated with the registered compression/decompression interface.

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
This pairs with the implementation of the ArchiveWriter API and
therefore can be used to write and O3AR archive files.

The Archive Reader supports the following operations

1. `ListFileInArchive` - This function can query a file within an
   archive by file path or file token which was queried previously from
   a ListFileInArchive call
1. `ExtractFileFromArchive` - This function can extract file from the
   archive using either the relative file path of the contained file or
   the file token queried from a call to `ListFileInArchive`
   The file is extracted to a user provided allocated buffer via an
   `AZStd::span<AZStd::byte>` paramter
1. `EnumerateFilesInArchive` - This functions enumerates all non-deleted files within the archive and invokes a user provided callback with the ArchiveListFileResult structure for each file. The `ArchiveListFileResult` structure is the same type returned by `ListFileInArchive`

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
The issue was an `<Item>` element was trying to be created to show the
success value, but there was data associated with it.

The proper way to visualize such an element is to use teh `<Synthetic>`
element.

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Comment on lines 389 to 390
return blockCount <= MaxBlocksNoJumpEntry
? MaxBlocksNoJumpEntry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blockCount <= MaxBlocksNoJumpEntry is checked before using this function in the lines below, so is this check necessary here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No it is not.
I'll remove in this PR if there is a need for a non-trivial update.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also make sure which logic is right. The one below returns zero, this one returns MaxBlocksNoJumpEntry if blockCount <= MaxBlocksNoJumpEntry

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well the inner lambda condition is always returning the "else" part of the Ternary since the condition is checked before calling it.

The entire purpose of the two function is to return the minimum block index for the final set of block lines that don't have a jump entry (confusing and hard to explain, I know).

But pretty much every compressed file in the Archive that is >16 MiB starts with its first block line (index 0) with a jump entry.
And then every three block lines afterwards(index 3, 6, 9, ...) would have a Jump entry if there more than 18 MiB (9 blocks) remaining in the file from the start of that block line.

It is explained better in the Streaming Archive document at here, but the implementation is complicated.

Fixed cmake configure when the attempting to the add the Archive Game
and Editor API targets as dependencies of the Asset Bundler

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
longer mention that it takes ownership of the raw TOC buffer.

Added a `GetBlockLineSpanForFile` doxygen @param parameter for the
Archive TOC View.

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Copy link
Contributor

@spham-amzn spham-amzn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly notes on comments/docs. nice work

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
@lemonade-dm lemonade-dm merged commit e3ba1e1 into o3de:development Jun 13, 2023
3 checks passed
@lemonade-dm lemonade-dm deleted the archive-reader-impl branch June 13, 2023 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants