lists in self.data and pprint-todo

[AiyionPrime]

I recently have started working with dpkt and encountered a packettype, which can occur several times in its parent. Or the other way around, the parents data-field can contain several of the children.

@glaslos had a similar problem a few years back when he tried to unpack SCCP. #376
This appears to be a rare treat throughout the codebase, but does exist a few times like in TLS, where multiple instances of TLSRecord are stored in a list records in TLS.

https://github.com/kbandla/dpkt/blob/master/dpkt/ssl.py#L58-L73

Looking around in dpkt.py, I feel like it is discouraged to have such lists in self.data, but to pick an arbitrary? name, create a list there and have data only contain b''.
https://github.com/kbandla/dpkt/blob/master/dpkt/dpkt.py#L290-L291

Maybe someone can point me in the right direction, looking at pprint through git blame, my guess would be @obormot?

Thanks in advance, Aiyion

[obormot]

I've seen a couple ways this has been done in dpkt parsers

1. auto-parse as much as possible, tls is an example;
2. don't parse everything, but provide routines and/or classes to pack and unpack things, netbios is an example, dhcp pack options is another one.

In both cases the remaining, unparsed data buffer goes to self.data, which can be picked up by the next protocol layer, or decoded as part of the same layer.

Choosing one method over the other is up the author of the code. In general though we try to keep dpkt fast and nimble, so a good guiding principle is to avoid parsing things if it impacts performance. A good example is IP and MAC addresses - they are present in most packets, so dpkt avoids converting them and leaves it up to the developer. pprint() will display printable IP and MAC addresses on demand.

As for naming fields, going by the protocol's RFC is a bulletproof way, but that's not always available. I've seen fields named pretty arbitrarily. In most cases when we refactor decoders and decide to rename fields, we leave the old ones around as well for compatibility. Example: https://github.com/kbandla/dpkt/blob/master/dpkt/ssl.py#L343

Hope this is helpful!

[AiyionPrime]

Very much, thank you!