Nested structures and references in expression language

[red-hood]

Hi,

I would like to parse a binary stream of complex, nested structures, like it is done e.g. in PNGParsingTest, where the class Png has a non-primitive member of type Chunk.

In the test file, the description of the binary format (the arguement to prepare) is given as a complete, nested structure, together with the two class declarations in the same file.

I would prefer to keep the declaration of the two classes separate together with their binary description, and only make a reference to the Chunk type in the DSL for the Png class and have JBBP resolve the reference. Is that possible with the current state of JBBP?

Also, is there any way to automatically inherit from a binary parser description to reuse it for an inherited mapped class?

[raydac]

the DSL is very easy and linear one so that all decision points must be implemented on the Java level, sometime if you have a field with variable data content you can use var modifier and use custom var field processor but I would recommend to have several JBBPParsers for different cases working with the same JBBPBitInputStream and it allows to build more or less complex not over-engineered solutions.
what do you mean as inheritance of mapped class? it would be good just provide some example of code

[red-hood]

I also considered the approach of different parsers on one stream (like in the Ethernet/IP/TCP example), but that seems to be quite a mix of descriptional modelling and imperative programming, which I personally do not like very much.

So, to give some example of what I mean with keeping the single classes separate, but together with their binary description:

https://github.com/synnefy/test_jbbp/blob/master/src/main/java/io/synnefy/test/jbbp/Png.java

Here I just build the parser description by concatenation of the part defined in the Chunk class, and later create a parser from it. It works, but I need to manually reference the right description, which is error prone.

I personally would prefer to be a able to declare Chunk itself as a "binary" type in the DSL, which could be reused and resolved automatically like any other primitive field, e.g.:

https://github.com/synnefy/test_jbbp/blob/master/src/main/java/io/synnefy/test/jbbp/PngDefunct.java

This would then also allow automatic inheritance, in which only additional fields of a class need to be declared (if there are any).

Are there any plans to support such a feature?

I had a look at preon, which describes the binary mapping by annotations on each class member (much like JPA for databases), allows to nest and inherit classes, and builds the binary structure accordingly. Unfortunately, it currently does not seem to be maintained, nor is there good support for unsigned binary representations, so a combination of both frameworks would be great :).

[raydac]

if take in count that parser works with text script, from my point of view it is much easier to use the edge instead of implementation work with java classes, you can make script dynamically and for instance I rewrote PNG parser where format of chunk described in the chunk class

public class PngTest {

  @Bin
  private static class Chunk {
    public static final String FORMAT = "int length; int type; byte[length] data; int crc;";

    int length;
    int type;
    byte[] data;
    int crc;
  }

  @Bin
  private static class PNGFormat {
    public static final String FORMAT = "long header; chunks [_]{$"+Chunk.class.getSimpleName()+"$}";

    long header;
    Chunk [] chunks;

    private static final JBBPParser PARSER = JBBPParser.prepare(FORMAT.replace("$"+Chunk.class.getSimpleName()+"$", Chunk.FORMAT));

    public PNGFormat(final InputStream inStream) throws IOException {
      PARSER.parse(inStream).mapTo(this);
    }
  }

  public static void main(String ... args) throws Exception {
    final PNGFormat parser = new PNGFormat(ClassLoader.getSystemClassLoader().getResourceAsStream("test.png"));
    System.out.println("Header : "+parser.header);
    System.out.println("Number of chunks : "+parser.chunks.length);
  }
}

it is very hard to implement type system related to java classes and it will increase complexity of the library dramatically without big profit

[red-hood]

Thank you for the example, using the class lookup might help and reduce possibility for errors.

There is another detail which I currently do not understand well: From the examples, it seesm to me that the binary format for the output is described by the parameters for the Bin()-Annotation (like BinType, outOrder...), whereas the binary format for the input is described by the argument to Parser.prepare(). Is it possible to use the annotions for parsing, or the DSL for encoding, so to have only one source for the binary format description?

[raydac]

at first there was not @bin annotation in the library and the library could only parse data, then some functionality to write data and mapping was added and @bin annotation was added for the purpose. @bin provides many parameters but anyway it doesn't allow correctly build script, for instance it is impossible to describe how to code array length for fields but if you have more or less easy case then you can try to use the snippet to generate DSL from fields marked by @bin , the snippet will generate DSL from such classes for such case

  public static class Chunk {

    public static final String FORMAT = "int length; int type; byte[length] data; int crc;";

    @Bin(outOrder = 1)
    int length;
    @Bin(outOrder = 2)
    int type;
    @Bin(outOrder = 3, type = BinType.BYTE_ARRAY,comment = "length")
    byte[] data;
    @Bin(outOrder = 4)
    int crc;
  }

  @Bin
  public static class PNGFormat {

    public static final String FORMAT = "long header; chunks [_]{$" + Chunk.class.getSimpleName() + "$}";

    @Bin(outOrder = 1)
    long header;
    @Bin(outOrder = 2, type = BinType.STRUCT_ARRAY)
    Chunk[] chunks;

    private static final JBBPParser PARSER = JBBPParser.prepare(FORMAT.replace("$" + Chunk.class.getSimpleName() + "$", Chunk.FORMAT));

    public PNGFormat(final InputStream inStream) throws IOException {
      PARSER.parse(inStream).mapTo(this);
    }
  }

[red-hood]

Thank you, this is very helpful for me. I guess it might also be useful to add it as an example to the library itself, as other people might want also want to use it for parsing and serialization.