How to convert data to custom class instance while unpacking

[qasikfwn]

Hi, I'm a fan of the declarative nature of caterpillar and I'm evaluating if it would be a good fit to replace the existing binary parsing system in a project. One thing I'm trying to accomplish is annotating fields in a struct with a custom non-struct type even though a normal uint64 needs to be parsed in its place.

Here's an example that hopefully illustrates what I mean:

import sys

from caterpillar.byteorder import LittleEndian
from caterpillar.py import struct
from caterpillar.shortcuts import unpack_file
from caterpillar.types import uint32_t


class Custom64bitHash:
	h: int

	def __init__(self, h: int) -> None:
		# do some other stuff with `h`
		# ...
		self.h = h


@struct(kw_only=True, order=LittleEndian)
class CustomStruct:
	field_1: uint32_t
	field_2: Custom64bitHash
	field_3: uint32_t


if __name__ == "__main__":
	path = sys.argv[1]
	obj = unpack_file(CustomStruct, path)
	for x, y in obj.__dict__.items():
		print(f"{x}: {y}")

What I would like to happen is: when parsing field_2, a uint64 is read and used to init an instance of Custom64bitHash which is then actually assigned to field_2

If I try to run this as-is, I get the error:

caterpillar.exception.ValidationError: The object of type '<class 'type'>' could not be converted to a struct, because the placed annotation does not have a corresponding handler:
 type: <class 'type'>,
 annotation: <class '__main__.CustomHash'>

I've read through the documentation but I haven't been able to come up with anything for this situation.

• The annotation registry seems to be for converting non-struct classes into struct classes, but not for using struct classes (like uint64) to initialize non-struct classes.
• Hooks might be the solution, but I'm not sure how to use them in this situation. If I could write a Custom64bitHash method that manually reads the 8 bytes required and converts them to an int, that would be acceptable

For Actions I'm not sure if something like this can work:

def unpack_hash(context: _ContextLike) -> None:
	...
	# can I read a uint64 here and return a Custom64bitHash instance?


@struct(kw_only=True, order=LittleEndian)
class CustomStruct:
	field_1: uint32_t
	field_2: f[Custom64bitHash, Action(unpack=unpack_hash)]
	field_3: uint32_t

The troublesome part for all this seems to be the fact that Custom64bitHash.__init__() needs to do more processing with h than simply directly storing it.
Is there some kind of handler or method I can implement for Custom64bitHash that will allow this to work?

[MatrixEditor]

Hi @qasikfwn ,

I think what you are looking for is the Transformer class:

from caterpillar.py import Transformer, f, uint64

class Custom64bitHash:
	h: int

	def __init__(self, h: int) -> None:
		# do some other stuff with `h`
		# ...
		self.h = h

class _64bitHash(Transformer[Custom64bitHash, int, Custom64bitHash, int]):
    def __init__(self):
        # parses a uint64 first
        super().__init__(uint64)

    def encode(self, obj: Custom64bitHash, context) -> int:
        return obj.h

    def decode(self, parsed: int, context) -> Custom64bitHash:
          return Custom64bitHash(parsed)


Custom64bitHash_t = f[Custom64bitHash, _64bitHash()]

Here, uint64is parsed first and you can define a function to apply custom logic.

Also, I did look into the documentation again and this could the only component that is still missing from the reference/tutorial -> will be marked as an open issue.

[qasikfwn]

That worked, thanks!