Sending binary data

[ggallea]

I am Reading data from binary file and trying to send the data I read to kafka.
I get the following error:
cannot unmarshal object into Go struct field Message.messages.value of type []uint8

What am I doing wrong?

Thanks

[mostafa]

Hey @ggallea,

Have a look at this example.

[ggallea]

The example is serializing a string. I have numbers (float and int32) to send
I also tried with Uint8Array and it gave me the same error

[mostafa]

@ggallea
Can you send your script and some sample data, so I can use it to debug the issue?

[ggallea]

How can I serialize (types other then string) to bytes (using the serialize function in SchemaRegistry)?

[c-meier]

Hello @mostafa,

Would it be possible to directly send an ArrayBuffer?

For the moment I do the following:

const body = encoding.b64decode(".............")  // returns an ArrayBuffer
writer.produce({ messages: [{
    value: schemaRegistry.serialize({
        data: Array.from(new Uint8Array(body)),
        schemaType: SCHEMA_TYPE_BYTES,
    }),
    key: ...
}]})

It works but it means a lot of transformations.

[mostafa]

Hey @c-meier,

you can read the reasoning behind making the serdes explicit in the following PR and its linked issues:

• Export Schema Registry client and the serdes functionality #149

[c-meier]

Hello @mostafa,

My point wasn't about exporting the serdes functionality to JS (I think it works quite well). But that I think it would make more sense for SCHEMA_TYPE_BYTES to accept data as an ArrayBuffer (or maybe Uint8Array) instead of an Array[number]

EDIT: K6 seem to be using ArrayBuffer as the return type of most JS method which deal with binary data (see issue)

[mostafa]

@c-meier
Fair enough! Then I need to test it to see if and how it works. In the meantime, if you can help contribute some code, it'll be much appreciated.