Integer fields in gzip_index are serialized independently to endianness

[vkuzniet]

Signed-off-by: Viktor Kuznietsov vkuzniet@amazon.com

Issue #, if available:

Description of changes:
gzip_index's have, size and span_size as well as gzip_index_point's in, out fields are converted to Big Endian representations for serialization and are converted back to the hosts architecture upon consumption. This ensures that IndexByteData byte sequence is consistent across multiple architectures.

Testing performed:

• make test && make check && make integration pass
• deployed soci and soci-snapshotter-grpc to EC2 instance and ran container workload for rabbitmq in lazy loading mode and got successful execution.

All the layers on rabbitmq were lazily loaded.

soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/2/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/4/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/5/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/6/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/7/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/8/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/9/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

The workload completed successfully.

2022-09-27 18:23:09.431972+00:00 [info] <0.779.0> started TCP listener on [::]:5672
 completed with 3 plugins.
2022-09-27 18:23:09.647003+00:00 [info] <0.708.0> Server startup complete; 3 plugins started.
2022-09-27 18:23:09.647003+00:00 [info] <0.708.0>  * rabbitmq_prometheus
2022-09-27 18:23:09.647003+00:00 [info] <0.708.0>  * rabbitmq_web_dispatch
2022-09-27 18:23:09.647003+00:00 [info] <0.708.0>  * rabbitmq_management_agent

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[wmesard]

I hate to say this, but the endian wars are over. The bad guys won. Quoting the FlatBuffers documentation:

Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.

We should canonicalize to little-endian, not big-endian. That way these will all be nops on (Linux, Windows) x (x86, ARM).

But speaking of FlatBuffers, this problem would go away if we were modeling the gzip_index data in the fbs file instead of treating it as a binary blob. Should we do that?

[vkuzniet]

I hate to say this, but the endian wars are over. The bad guys won. Quoting the FlatBuffers documentation:

Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.

We should canonicalize to little-endian, not big-endian. That way these will all be nops on (Linux, Windows) x (x86, ARM).

But speaking of FlatBuffers, this problem would go away if we were modeling the gzip_index data in the fbs file instead of treating it as a binary blob. Should we do that?

I think, long term this is the right approach. We would anyway needed to do that at the time we re-write the indexer to Go, so this work seems to be inevitable.

[vkuzniet]

I hate to say this, but the endian wars are over. The bad guys won. Quoting the FlatBuffers documentation:

Each scalar is also always represented in little-endian format, as this corresponds to all commonly used CPUs today. FlatBuffers will also work on big-endian machines, but will be slightly slower because of additional byte-swap intrinsics.

We should canonicalize to little-endian, not big-endian. That way these will all be nops on (Linux, Windows) x (x86, ARM).
But speaking of FlatBuffers, this problem would go away if we were modeling the gzip_index data in the fbs file instead of treating it as a binary blob. Should we do that?

I think, long term this is the right approach. We would anyway needed to do that at the time we re-write the indexer to Go, so this work seems to be inevitable.

Created an issue #81 to address that. Once it's implemented, encode/decode methods from this PR will be obsolete.