Why can Zserio serialize varuint32 arrays but not deserialize them?

[Yesyoor]

Hello,
I am facing a weird issue. I am using Zserio to work with the NDS Standard.
At one point there is an int[ ] array which is part of a class.

The class is called "PoiLayer" and embedded in a SmartLayerTile

I serialize the SmartLayerTile anbd the PoiLayer to Json and to byte arrays.

// SmartTile incl the PoiLayer
DebugStringUtil.toJsonFile(smartTile,jsonSmartTilePath );

//PoiLayer only
DebugStringUtil.toJsonFile(poiLayer,poiLayerFile.getPath() );

Inspecting the results shows a JSON for the SmartLayerTile, where the PoiLayer is embedded as an byte array.

Looking at the result of the JSON for the PoiLayer shows correct results.

The problems begin when deserializing the bytes from the first JSON. the reader will always reach an out of bounds error or tell me that the value is too big for varsize.

So I tried serializing it to an byte array and reading it back in, same errors.
Comparing the byte array of the PoiLayer and the bytes embedded in the SmartTile JSON, shows that all bytes except one are equal, the one that differs is -48 on one side and 208 on the other side (different interpretation unsigned/ signed)

I tried deserializing from bytes, from file, and by calling the constructor, always similar errors.
I ensured the runtime and zserio jar are compatible. What could be the error?

        byte[] fileContent = Files.readAllBytes(poiLayerFile.toPath());

        byte[] bytesPoiLayer = SerializeUtil.serializeToBytes(poiLayer );

        PoiLayer poiLayerFromFile = SerializeUtil.deserializeFromFile(PoiLayer.class, poiLayerFile);
        PoiLayer poiLayerB = SerializeUtil.deserializeFromBytes(PoiLayer.class, bytesPoiLayer);

        PoiLayer poiLayer1 = new PoiLayer( new ByteArrayBitStreamReader(  bytesPoiLayer ) );

This is the specification of a Poi object (dev acc is free):

https://developer.nds.live/schema/poi/2024.03/poi

The error occurs for this variable:

/**Id of a Poi category.*/
subtype varuint32 PoiCategoryId;

which is embedded in this array:

/** List of categories to which the POI belongs. */
PoiCategoryId categoryIdList[];

I tried setting it with this special varsizetype
zserio.runtime.array.Array a = new zserio.runtime.array.Array( new zserio.runtime.array.RawArray.IntRawArray(), new zserio.runtime.array.ArrayTraits.VarUInt32ArrayTraits(), zserio.runtime.array.ArrayType.AUTO); try { BitStreamWriter w = new ByteArrayBitStreamWriter(); w.writeVarSize(0); a.writePacked(w); } catch (IOException e) { throw new RuntimeException(e); } poi.setCategoryIdList(a.getRawArray());

but it gives the same result as
poi.setCategoryIdList( new int[ ] { 0 } );

This makes no sense. it can serialize it but not deserialize it. Sounds like a clear bug to me. Also bytes written to file are different than bytes written to RAM.

[Mi-La]

Hello Yesyoor,

To be honest, I don't understand what exactly is the problem.

Inspecting the results shows a JSON for the SmartLayerTile, where the PoiLayer is embedded as an byte array.
Looking at the result of the JSON for the PoiLayer shows correct results.

This sounds to be ok, since in the PoiLayer within the SmartLayerTile is stored using the extern keyword. Thus only serialized bytes are stored there.

Then I got totally lost in what you are trying further.

I tried setting it with this special varsizetype
zserio.runtime.array.Array a = new zserio.runtime.array.Array( new zserio.runtime.array.RawArray.IntRawArray(), new zserio.runtime.array.ArrayTraits.VarUInt32ArrayTraits(), zserio.runtime.array.ArrayType.AUTO); try { BitStreamWriter w = new ByteArrayBitStreamWriter(); w.writeVarSize(0); a.writePacked(w); } catch (IOException e) { throw new RuntimeException(e); } poi.setCategoryIdList(a.getRawArray());

Here you are just creating an Array wrapper which will contain an empty raw array, then you are writing it to the ByteArrayBitStreamWriter's internal buffer:

1. w.writeVarSize(0) has no sense, it will become just an extra byte with value 0 unrelated with the array (AUTO array will serizalize its size automatically).
2. a.writePacked(w) serializes the empty array (moreover using an internal writePacked method) .. since it's empty, it will result in a single byte with value 0.
3. a.getRawArray() is still unchanged and thus you are setting the empty raw array (i.e. new int[] {}) to the categoryIdList.

Please try to make a minimal non-working example, you can try our interactive zserio compiler (unfortunately supports only python runtime, but you can provide the Java code as an attachment within this discussion).

Serialization and deserialization of varuint32 array in a simple schema seems to be working:

struct Data
{
    varuint32 array[];
};

Data data = new Data();
data.setArray(new int[] {1, 200, 1000});

final byte[] bytes = SerializeUtil.serializeToBytes(data);

Data readData = SerializeUtil.deserializeFromBytes(Data.class, bytes);
System.out.println(data.getArray().length);
System.out.println(readData.getArray().length);

Please also specify version of Zserio you are using.

[Yesyoor]

Hello @Mi-La ,
thank you for the reply and also for the array example and explanations.

my first attempt was to set the array for the poi category ids like so:

private static Poi[] generatePoiMessageList(){
    Poi[] pois = new Poi[2];
    byte coordShift = 5;
    Position2D pos = new Position2D(coordShift, 90, 90 );
    for(int i = 0; i <pois.length; i++){
        Poi poi = new Poi();
        poi.setPoiId(i);
        poi.setPosition( pos );
        poi.setCategoryIdList(new int[]{1});

        pois[i] = poi;
    }

    return pois;
}

It can't be deserialized because the BitStreamReader will reach out for bytes outside of the byte arrays length and tell me that
13232203844' is out of range for varsize type!

It always happens when it tried to decode the category id list. Maybe smth got offset ealier already though.

I'll add the methods I use to generat the PoiLayer, The PoiMetadataLayer, the ExtData structures and the SmartLayerTile here.

I am using NDS Schema Version 2024.03 and the Zserio.jar and zserio-runtime.jar which have been provided with the package builder. It must be Version 2.12 or 2.13.
Do you know which zserio.jar and zserio-runtime.jar should be used for the NDS 2024.03 schemata?
I found some infos in here https://zserio.org/doc/ZserioCompatibilityGuide.html but I didn't get the dependencies and version numbers or where to find them.

private static void generateTestData() {

    smartTile = new SmartLayerTile();
    smartTile.setTileId(123344);

    PoiLayer poiLayer = generatePoiLayer();
    DataLayer l = getPoiDataLayer(poiLayer);

    PoiLayerMetadata meta = generatePoiLayerMetadata();
    DataLayer m  = getPoiDataLayerMetadata(meta);

    smartTile.setLayers( new DataLayer[]{l,m } );

    DataLayerLifetime dataLayerLifetime = new DataLayerLifetime(DataLayerLifetimeType.LIVE, new DataLayerLifetimeInfo(DataLayerLifetimeType.LIVE));
    PackagingDetails packagingDetails = new PackagingDetails((short) 0,(short) 0,(short)0,0, new DeltaEncodingDetails((short) 0, new BigInteger("0")));

    smartTile.setHeader(new SmartLayerHeader((short) 2, new short[]{0,1}, new DataLayerLifetime[]{dataLayerLifetime,dataLayerLifetime}, new PackagingDetails[]{packagingDetails,packagingDetails}, new String("")));

    SerializeUtil.serializeToFile(smartTile, binarySmartTilePath);
    try {
        File poiLayerFile = new File(jsonLayersPath + "_poi.txt");

        DebugStringUtil.toJsonFile(smartTile,jsonSmartTilePath );
        DebugStringUtil.toJsonFile(poiLayer,poiLayerFile.getPath() );

        byte[] fileContent = Files.readAllBytes(poiLayerFile.toPath());

        byte[] bytesPoiLayer = SerializeUtil.serializeToBytes(poiLayer );


        PoiLayer poiLayerFromFile = SerializeUtil.deserializeFromFile(PoiLayer.class, poiLayerFile);
        PoiLayer poiLayerB = SerializeUtil.deserializeFromBytes(PoiLayer.class, bytesPoiLayer);

        PoiLayer poiLayer1 = new PoiLayer( new ByteArrayBitStreamReader(  bytesPoiLayer ) );


        DebugStringUtil.toJsonFile(meta,jsonLayersPath + "_poiMeta.txt" );

        byte[] bytesPoiMetaLayer = SerializeUtil.serializeToBytes(meta );

    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}


private static PoiLayer generatePoiLayer(){
    PoiLayer poiLayer = new PoiLayer();

    Poi[] pois = generatePoiMessageList();
    poiLayer.setPoiList(pois);

    return poiLayer;
}

private static DataLayer getPoiDataLayer(PoiLayer poiLayer) {
    byte[] bytes = SerializeUtil.serializeToBytes(poiLayer);
    BitBuffer buff = new BitBuffer(bytes);

    DataLayer l = new DataLayer();
    ExternData extDat = new ExternData();
    String targetPoiLayerPackage = PoiLayer.class.getName();
    extDat.setExternDescriptor(new ExternDescriptor(NAME.NAME, VERSION.VERSION, MIN_VERSION.MIN_VERSION,targetPoiLayerPackage ));
    extDat.setData(buff);
    l.setLayer(extDat);
    return l;
}

Surprisingly the metadatalayer can be encoded and decoded, but the simpler PoiLayer fails.
I really don't understand what the SerializeUtil is doing and why it won't recognize the correct size of the datastructure which has been generated by itself.

private static PoiLayerMetadata generatePoiLayerMetadata(){
    PoiLayerMetadata poiLayerMetadata = new PoiLayerMetadata();
    NameString namestr = new NameString("airport", (short) 0, NameStringRelationType.BASE_NAME, NameStringUsageType.DEFAULT_OFFICIAL_NAME, (short)0);
    NameStringCollection names = new NameStringCollection(new NameString[]{namestr});

    PoiCategory cat = new PoiCategory();
    cat.setCategoryId(0);
    cat.setStandardCategory(PoiStandardCategory.POICAT_AIRPORT);
    cat.setCategoryNameStringCollection(names);
    cat.setSelectionEntry(false);
    cat.setShowInSearchTree(false);
    cat.setIsBrand(false);
    cat.setIsCollective(false);


    PoiCategory[] cats = new PoiCategory[]{ cat};
    poiLayerMetadata.setCategories( cats);
    ScaleRange[] sc = new ScaleRange[]{new ScaleRange(0 , 1,2)};

    poiLayerMetadata.setScales( new ScaleRangeList(1, sc));
    poiLayerMetadata.setAvailableLanguages(
            new AvailableLanguages(
                new LanguageMapping[]{
                        new LanguageMapping((short) 0, "DE", "DE", "DE", (short) 0,(short) 0, new LanguageName[]{new LanguageName((short) 0, "Deutsch")})
                }
            )
    );

    return poiLayerMetadata;
}

private static DataLayer getPoiDataLayerMetadata(PoiLayerMetadata poiMetaLayer) {

    byte[] bytes = SerializeUtil.serializeToBytes(poiMetaLayer);
    BitBuffer buff = new BitBuffer(bytes);

    DataLayer l = new DataLayer();
    ExternData extDat = new ExternData();
    String targetPoiLayerPackage = PoiLayerMetadata.class.getName();
    extDat.setExternDescriptor(new ExternDescriptor(NAME.NAME, VERSION.VERSION, MIN_VERSION.MIN_VERSION,targetPoiLayerPackage ));
    extDat.setData(buff);
    l.setLayer(extDat);
    return l;
}

public static void decodeSmartTile()  {

    File ndsFile = new File(binarySmartTilePath);
    byte[] fileContent = new byte[0];
    try {
        fileContent = Files.readAllBytes(ndsFile.toPath());
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

    final BitStreamReader reader = new ByteArrayBitStreamReader(fileContent);

    try {
        smartTile = new SmartLayerTile(reader);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
    List<String> targets =  getSmartTileDataLayerTargets();
    for(int i =0; i< targets.size(); i++){
        String target = targets.get(i);
        try {
            System.out.println("Instanciating: " + target);

            if(i ==0){
                PoiLayer poiLayer = new PoiLayer(new ByteArrayBitStreamReader(smartTile.getLayers()[i].getLayer().getData().getBuffer()));
                DebugStringUtil.toJsonFile(poiLayer, decodedJsonLayersPath +i+".txt");
            }else {

                Constructor<?> constructor = null;
                Object layerObj = null;

                constructor = Class.forName(target).getConstructor();
                layerObj = constructor.newInstance();
                Method layerReadMethod = layerObj.getClass().getMethod("read", zserio.runtime.io.BitStreamReader.class);
                layerReadMethod.invoke(layerObj, new ByteArrayBitStreamReader(smartTile.getLayers()[i].getLayer().getData().getBuffer()));
                DebugStringUtil.toJsonFile(layerObj, decodedJsonLayersPath +i+".txt");
            }

        }catch(Exception e) {
            System.out.println("ERROR decoding layers: "+e.getLocalizedMessage());
        }
    }
}


public static List<String> getSmartTileDataLayerTargets() {
    List<String> targets = new ArrayList<>();
    for (DataLayer layer : smartTile.getLayers()) {
        String target = layer.getLayer().getExternDescriptor().getTarget();
        targets.add(target);
    }
    return targets;
}

[Yesyoor]

@Mi-La
This is the entire Java project (I use intelliJ).
Maybe I did something wrong with the zserio runtime etc. At some point I lost track on where the source of the error could be. Unfortunately the documentation didn't help me out a lot eigther.

Thank you for having taken a look at it already. I feared since it is NDS related and this is about Zserio, that I wouldn't find much support here :) But your explanation about the ext data helped a little already.

Nds_workshop_decoder.zip

This is the exception when it tries to deserialize the PoiLayer (with the simple = new int[ ]{0} implementation) :

Exception in thread "main" zserio.runtime.ZserioError: SerializeUtil: java.lang.reflect.InvocationTargetException
at zserio.runtime.io.SerializeUtil.deserializeFromReader(SerializeUtil.java:313)
at zserio.runtime.io.SerializeUtil.deserializeFromFile(SerializeUtil.java:268)
at act.Main.generateTestData(Main.java:114)
at act.Main.main(Main.java:63)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at zserio.runtime.io.SerializeUtil.deserializeFromReader(SerializeUtil.java:307)
... 3 more
Caused by: java.io.IOException: ByteArrayBitStreamReader: Read value '13232203844' is out of range for varsize type!
at zserio.runtime.io.ByteArrayBitStreamReader.readVarSize(ByteArrayBitStreamReader.java:612)
at zserio.runtime.array.Array.readPacked(Array.java:330)
at zserio.runtime.array.Array.readPacked(Array.java:313)
at poi.poi.Poi.read(Poi.java:584)
at poi.poi.Poi.(Poi.java:59)
at poi.layer.PoiLayer$ZserioElementFactory_poiList.create(PoiLayer.java:180)
at poi.layer.PoiLayer$ZserioElementFactory_poiList.create(PoiLayer.java:161)
at zserio.runtime.array.PackedArrayTraits$ObjectPackedArrayTraits.read(PackedArrayTraits.java:186)
at zserio.runtime.array.Array.readPacked(Array.java:346)
at zserio.runtime.array.Array.readPacked(Array.java:313)
at poi.layer.PoiLayer.read(PoiLayer.java:136)
at poi.layer.PoiLayer.(PoiLayer.java:17)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
... 6 more

[Mi-La]

@Yesyoor
You are correct that this is really more an NDS related question and you should probably contact the NDS Support (e.g. via email) directly. And yet you probably shouldn't share NDS sources here.

However, regarding the Zserio version I'm not aware of any problems with Zserio 2.12 or 2.13, but you should be always able to use the latest release, currently new 2.14.

From a brief look at your quite complex sample, it seems to me that you are trying to deserizalize the JSON files?! This cannot work. It should be possible to read it using DebugStringUtil.fromJsonFile, but whole the JSON format is there mostly for debugging purposes. To serialize data to binary format, please use SerializeUtil.serializeToFile(someBlob, somePath) .. then you can desiralize the same file using SerializeUtil.deserializeFromFile(SomeBlob.class, somePath) and it should work.

You just cannot freely mix JSON a binary formats. Zserio is designed to work only with the binary format - either using BitStreamReader/BitStreamWriter manually, or using SerializeUtil (zserio.runtime.io.SerializeUtil) which could be more handy.

JSON was added for debugging purposes and can be used via zserio.runtime.DebugStringUtil.

Note that there was an issue with JSON deserialization in 2.13, which is already fixed, but I think it's not the case.

[Yesyoor]

@Mi-La
Thank you for the fast reply.
Don't get confused about the DebugStringUtil and SerializeUtil, I din't mix it, I serialize and deserialize with the serializeUtil and json is done with DebugStringUtil. They just both appear there because I try to find out why it can generate JSON Export, but not serialize and deserialize correrctly (which works for the metadata layer only so far)

So while trying to pin the error source down, I serialized/ deserialized empty SmartLayerTiles, Datalayers only and so on to check where the problem is.

And as described it occures only when trying to serialize and deserialize the poiLayer (no matter if embedded in a SmartLayerTile or not).

I have been decoding multiple binary files of display layers and traffic layers in the past months. But generating POIs and serializing them and deserializing turns out to be really tricky and yet I don't know why the reader gets offset.

[Mi-La]

What about the line 114 in src/act/Main.java:

            PoiLayer poiLayerFromFile = SerializeUtil.deserializeFromFile(PoiLayer.class, poiLayerFile);

Here the poiLayerFile is clearly JSON.

On lines 104-107:

File poiLayerFile = new File(jsonLayersPath + "_poi.txt");

DebugStringUtil.toJsonFile(smartTile,jsonSmartTilePath );
DebugStringUtil.toJsonFile(poiLayer,poiLayerFile.getPath() );

If you are able to provide really minimalistic (not NDS releated, because it is not public) sample which will demonstrate your problem, I can try to further help you. But in case of NDS please ask the NDS Support.

[Yesyoor]

you are right, changed it to SerializeUtil and it still won't work.

Unfortunately I also can't tell what is going wrong...just that it happens in the PoiLayer and that it is always some misinterpreted byte length/ size.

I"ll ask the support, anyways thank you for takign a look and the suggestions! :)

[Mi-La]

Ah .. I was finally able to debug your project somehow. I have probably found the source of your problem.
It seems to be caused partially by your wrong data and partially by our weak safety checks, which should clearly catch you immediately during serialization!

Poi defines position with hardcoded coordShift=0 (see the schema Position2D(0)), but you set it to 5 in Main.java on line 162 (method generatePoiMessageList).

Normally we check the parameters to meet the schema restrictions, but it seems that the check is missing in case of packed arrays :-(. Thus now when you try to serialize the Poi, it writes the position with number of bits depending on the shift value 5.

But when it tries to read it again, it uses the schema defined value 0 which cause that it reads different number of data from the stream.

Your quick fix is to set the coordShift to the value defined by the schema - i.e. 0:

        byte coordShift = 0;

And I'll create an issue that the check shall be present also in the writePacked method - the same on as the check in Poi.write:

        // check parameters
        if (position_.getShift() != (byte)(0))
        {
            throw new zserio.runtime.ZserioError("Write: Wrong parameter shift for field Poi.position: " +
                    position_.getShift() + " != " + (byte)(0) + "!");
        }
        position_.write(out);

Note that this one particular check is missing only in Java where it's missing by accident 🌋.

[Mi-La]

Created issue #616.

[Yesyoor]

@Mi-La Wow thank you so much! How did you found it? I also debugged and saw the bitoffset, but I totally got lost in those reflection classes during debugging.

I used coordshif 5 because I always stumbled upon coordshifts>0 for Incident Events.
Again, thank you so much, I would have never found this bug, I assumed it to be different zserio, runtime and schema versions or me doing smth wrong with the external descriptors - although I also wondered why it would not complain during serialization which led me to the doubt the compability of the the zersio runtime and the generated code.

[Mi-La]

@Yesyoor Great that it helped!

I have a small advantage than I know exactly what the generated code and Zserio "inners" should do .. and still I had to debug field by field and carefully check all the serialized and deserialized bits until the exception :-).

[Yesyoor]

So nice! Really well done, I tried the same but my debugger seemed to jump and I lost track. I am also quite experienced in working with bytepatterns but the reflections and nested calls made it really difficult for me. Thanks again and have a great weekend :)