Adding a LZ4 Compressor/Decompressor to the Compression Gem

[lemonade-dm]

A lz4 compressor implementation has been added that registers with the
the CompressionRegistrar which is available in tools code.
In client code a lz4 decompressor implementation has been added that
registers with the DecompressionRegistrar.

Added UnitTest to the Compression Gem to validate the
CompressionRegistrar and DecompressionRegistrar works properly.

Updated the Compression Interface with a CompressBound function which
provides an upper bound on potential compressed data size given the
uncompressed data size.

Also added more detailed error result structure which can provide an
error string on errors that occurs during compression/decompression

How was this PR tested?

Ran newly added Googletest for the Compression Gem that validates the (De)CompressorRegistrar system and the (De)CompressorLZ4Impl classes

[nick-l-o3de]

While I can see this was a bunch of work, have we considered just adding this component to the existing compression gem instead of making a new one? Each additional gem adds gigabytes of overhead (due to static lib sizes and pdbs) and many seconds of link time. if its just a part of the engine, does it really have to be its own gem?

It feels like it could be just a handful of files in another gem (liek the compression gem itself) which just handles all the small and low-dependency compression formats that the engine itself offers, in one gem... and thus only one link cost and static compile cost?

[lemonade-dm]

While I can see this was a bunch of work, have we considered just adding this component to the existing compression gem instead of making a new one? Each additional gem adds gigabytes of overhead (due to static lib sizes and pdbs) and many seconds of link time. if its just a part of the engine, does it really have to be its own gem?

It feels like it could be just a handful of files in another gem (liek the compression gem itself) which just handles all the small and low-dependency compression formats that the engine itself offers, in one gem... and thus only one link cost and static compile cost?

Yes, this could be added to the existing Compression Gem as part of a builtin list of supported gems.
It would currently only be LZ4 at this time with potential support for Zstd and Zlib coming in the future.

However the Compression Interface gem is meant to be used by other gems to extend the archive system with additional compression algorithms and not necessarily require that compression libraries such as lz4, zstd or zlib be a dependency of the engine.

That being said I think the cost of using those libraries and loading in the few symbols that the compression gem would use, would have negligible cost.

I'll move over the Compressor LZ4 implementation into the Compression gem

[nick-l-o3de]

While I can see this was a bunch of work, have we considered just adding this component to the existing compression gem instead of making a new one? Each additional gem adds gigabytes of overhead (due to static lib sizes and pdbs) and many seconds of link time. if its just a part of the engine, does it really have to be its own gem?
It feels like it could be just a handful of files in another gem (liek the compression gem itself) which just handles all the small and low-dependency compression formats that the engine itself offers, in one gem... and thus only one link cost and static compile cost?

Yes, this could be added to the existing Compression Gem as part of a builtin list of supported gems. It would currently only be LZ4 at this time with potential support for Zstd and Zlib coming in the future.

However the Compression Interface gem is meant to be used by other gems to extend the archive system with additional compression algorithms and not necessarily require that compression libraries such as lz4, zstd or zlib be a dependency of the engine.

That being said I think the cost of using those libraries and loading in the few symbols that the compression gem would use, would have negligible cost.

I'll move over the Compressor LZ4 implementation into the Compression gem

Yes, I'm trying to save compile time here practically. It'd be a different calculus if the gem system added very light weight but each gem currently has a huge minimum memory cost and compile cost. I wish it weren't so... if they were tiny and cheap I'd be all for making lots of them instead of few...

[lemonade-dm]

While I can see this was a bunch of work, have we considered just adding this component to the existing compression gem instead of making a new one? Each additional gem adds gigabytes of overhead (due to static lib sizes and pdbs) and many seconds of link time. if its just a part of the engine, does it really have to be its own gem?
It feels like it could be just a handful of files in another gem (liek the compression gem itself) which just handles all the small and low-dependency compression formats that the engine itself offers, in one gem... and thus only one link cost and static compile cost?

Yes, this could be added to the existing Compression Gem as part of a builtin list of supported gems. It would currently only be LZ4 at this time with potential support for Zstd and Zlib coming in the future.
However the Compression Interface gem is meant to be used by other gems to extend the archive system with additional compression algorithms and not necessarily require that compression libraries such as lz4, zstd or zlib be a dependency of the engine.
That being said I think the cost of using those libraries and loading in the few symbols that the compression gem would use, would have negligible cost.
I'll move over the Compressor LZ4 implementation into the Compression gem

Yes, I'm trying to save compile time here practically. It'd be a different calculus if the gem system added very light weight but each gem currently has a huge minimum memory cost and compile cost. I wish it weren't so... if they were tiny and cheap I'd be all for making lots of them instead of few...

The LZ4 Compressor logic and Unit Test have been moved into the Compression Gem and the CompressionLZ4 gem has been removed.