SBE with components and reuse

[lapetiteclef]

We have success fully used SBE for some point to point solutions. But in a large project I now have some issues using SBE for a larger schema.

To structure reuse between schemas we use a set of SBE schemas files and preprocessing/merging them to get common types across domains. Main goal is to share the processing and transformations needed at application level (mapping to reuse at SBE component level).

I noticed one side affect. If defining a composite that is referring to other composite and types, (in my case a ref to enum). It is the Java code generation that skips generating the enum if its not refereed somewhere at the top message level. I checked the Ir parser and it looks like its only generate what defined in a message not indirectly what is refereed from the underlying components. If I add a dummy message referring the enum at top message level the underlying components generates correctly.

Is that a bug or is it not supported with a component referring to types not used at message level?

[mjpt777]

From messages we recursively descend into composites and capture all types which are defined or referenced. I've just written a quick test to confirm this. We do not generate types that have no usage.

Can you provide a test to reproduce what your issue is?

[mjpt777]

Can you also please confirm what version you are using?

[lapetiteclef]

I try to make minimal example. It was not as simple as I described above, I noticed it when trying to generate a minimal example. Not sure if this maps exactly to my enum error but its generates similar:

I updated few weeks ago from latest github version. I can update again and see if it resolves it.

<type name="string10" length="10" primitiveType="char" />

<composite name="messageHeader" description="header">
	<type name="blockLength" primitiveType="uint16" />
	<type name="templateId" primitiveType="uint16" />
	<type name="schemaId" primitiveType="uint16" />
	<type name="version" primitiveType="uint16" />
</composite>


<composite name="mycomponent2">
	<ref name="somefield" type="string10" />
</composite>

<composite name="mycomponent1">
	<ref name="somefield" type="string10" />
	<ref name="comp" type="mycomponent2" />
</composite>


<composite name="mycomponent0">
	<ref name="somefield" type="string10" />
	<ref name="comp" type="mycomponent1" />
</composite>


</types>

<message name="RequestSomething" id="1" description="REQUEST">
	<field name="id" id="1" type="int64" />
	<field name="comp" id="2" type="mycomponent0" />
</message>
<!-- enable to generate missing component 1
<message name="RequestSomethingAgain" id="2" description="REQUEST">
	<field name="id" id="1" type="int64" />
	<field name="comp" id="2" type="mycomponent1" />
</message>
 -->

[lapetiteclef]

The above was tested on SBE 1.7.1-SNAPSHOT
I noticed code was generated without enabled schema validation.
Tried with schema validation using the sbe.xsd located in sbe-tool\src\main\resources\fpl

The example above passes schema validation (after replacing message with sbe:message) but I get the same problem, component1 is missing.

[mjpt777]

Yes I believe this is a real issue when we have a chain of refs. I will investigate.

[lapetiteclef]

I checked and it looks like a minor issue in Ir.captureTypes / captureType methods. Referenced types needs to be added as its reference to not overwrite reoccurring ref names in messages.

Example of minor fix by forwarding the referencedName to captureType method:

private void captureTypes(final List<Token> tokens, final int beginIndex, final int endIndex)
{
    for (int i = beginIndex; i <= endIndex; i++)
    {
        final Token token = tokens.get(i);
        final int typeBeginIndex = i;

        switch (token.signal())
        {
            case BEGIN_COMPOSITE:
                i = captureType(tokens, i, Signal.END_COMPOSITE, token.name(), token.referencedName());
                captureTypes(tokens, typeBeginIndex + 1, i - 1);                		                		
                break;

            case BEGIN_ENUM:
                i = captureType(tokens, i, Signal.END_ENUM, token.name(), token.referencedName() );
                break;

            case BEGIN_SET:
                i = captureType(tokens, i, Signal.END_SET, token.name(), token.referencedName() );
                break;
        }
    }
}

private int captureType(final List<Token> tokens, final int index, final Signal endSignal, final String name, final String referencedName)
{
    final List<Token> typeTokens = new ArrayList<>();

    int i = index;
    Token token = tokens.get(i);
    typeTokens.add(token);
    do
    {
        token = tokens.get(++i);
        typeTokens.add(token);
    }
    while (endSignal != token.signal() || !name.equals(token.name()));

    updateComponentTokenCounts(typeTokens);
    if( referencedName == null )
    	typesByNameMap.put(name, typeTokens);
    else
    	typesByNameMap.put(referencedName, typeTokens);
    	
    return i;
}

[mjpt777]

Thanks for this. I should have some time to have a look tomorrow.

[mjpt777]

Thanks. That seems to have been the issue.

[lapetiteclef]

Good, I guess my large schemas provides some extra SBE testing.

I also have some other questions, maybe separate thread but I try here:

I tested and added a package to the type section locally. It worked well generating fully qualified types into separate package.
Example

    <types package="test.api.common">
    </types>
    <types package="test.api.common.product">
    </types>

The resulting code reuse in the SBE generated code is not important because its generated, but all surrounding transformations / code managing SBE is not. This feature reduces code duplication in developer written code.

I also added two helper functions to each message encoder:

encoder.reset() to clear out memory for message block in its wrapped buffer. We use it when reusing the message instance when code does not have access to the actual buffer

enoder.copyFrom( decoder ) to enable a copy from same message type without direct access to buffers.

What do you think about possibilities adding features described above? Or is there any other recommended way for accomplishing similar things? Change for Java code generation was quite small, but I did not look into other languages and aspects.

[mjpt777]

Best to move this to another issue. Things like your package suggestion are better address for the specification which is outside our gift to change. Other things if within the spec we can consider.

[mjpt777]

Also best to keep each issue to a single subject so it can be tracked and it makes sense in the scope of the issue name.