[Question] Future-proofing serialized data

[drse]

Question

I decided to use Fury as the serialization mechanism for both data in motion and data at rest across the entire application, and I'm wondering what would be the safest approach to future proof the data and prevent deserialization issues. So far, my approach is to add a 56-bit header for every serialized payload, consisting of the following:

• byte: fury major version
• byte: fury minor version
• byte: fury patch version

followed by an int32 with feature bit flags that correspond to the builder params as follows:

• 0: lang = xlang
• 1: lang = java
• 2: lang = python
• 3: lang = cpp
• 4: lang = go
• 5: lang = javascript
• 6: lang = rust
• 7: is row format
• 8: timeRefIgnored
• 9: compressInt
• 10: compressLong
• 11: compressString
• 12: compatible mode = COMPATIBLE
• 13: compatible mode = CONSISTENT
• 14: checkClassVersion
• 15: checkJdkClassSerializable
• 16: registerGuavaTypes
• 17: requireClassRegistration
• 18: suppressClassRegistrationWarnings
• 19: metaShareEnabled
• 20: scopedMetaShareEnabled
• 21: deserializeNonexistentClass
• 22: codeGenEnabled
• 23: asyncCompilationEnabled
• 24: scalaOptimizationEnabled
• 25: copyRef
• 26: serializeEnumByName
• 27: api = serialize/deserialize
• 28: api = serializeJavaObject/deserializeJavaObject
• 29: api = serializeJavaObjectAndClass/deserializeJavaObjectAndClass
• 30: reserved
• 31: reserved

My rationale is that in cases of future errors due to fury api or application-level changes, the builder can be constructed exactly as it was during the time of original serialization. I'm expecting to end up with multiple languages and configurations (speed and size optimized) across multiple fury versions.

Questions:

• Is this overkill and unnecessary (some of this data is already baked into the fury serialization protocol)?
• Is there any other metadata that should be included or can be safely excluded?
• Do I need more reserved flags for future planned configuration options?
• Is there a more compact way to represent the essential metadata?

Thank you.

[chaokunyang]

Even those flags are exactly the same, the data serialized by fury of one version, may not be deserialized by another version of fury.

Fury don't provide binary compatibility across minor versions. We plan to provide such compatibility from 1.0.0. But it may still take some time

[drse]

That is why I'm including the serializing fury version in the first 3 bytes of the header. The version data would be used at deserialization to instantiate the appropriate shaded fury version, and the flags would then be applied to the builder of that specific version. Is that reasonable, or am I still on the wrong track?

[chaokunyang]

This should work

[pandalee99]

It makes sense.

[jeacott1]

I went with a slightly different approach.
A simple headerFieldCount as the first header allows me to adjust the number of headers over time.
no need for reserved fields.

   try (ForyInputStream furyIn = new ForyInputStream(in)) {
            // Read header count
            int headerFieldCount = readVarInt32(furyIn);

            // Read known headers
            int schemaVersion = readVarInt32(furyIn);
            int foryVersion = readVarInt32(furyIn);
            ...

            int remainingHeaders = headerFieldCount - 2 /*known header count*/;
            for (int i = 0; i < remainingHeaders; i++) {
                int unknownField = readVarInt32(furyIn);
                log.trace("Skipping unknown header field: {}", unknownField);
            }

            ThreadSafeFory versionedFory = foryForVersion(foryVersion);
            versionedFory.deserializeJavaObject(furyIn, clazz);
            ...
   }