job data serialization: alternative serialization methods?

[SEAPUNK]

right now, we are only allowing the job to be created if the data can be stringified with JSON

what if we allow things like msgpack?

[SEAPUNK]

job data, if stored, will need to be stored as a binary blob if we're doing things like msgpack
what about job data type specification??? deserialization?

[SEAPUNK]

okay how about this:

when creating a job, you specify in the options what the job data type and serializer is
when creating the job runner, you must be explicit in saying that the runner supports alternative job data deserializers, so binary blobs can be sent to the runner

[SEAPUNK]

otherwise, it's JSON by default

[SEAPUNK]

...or maybe not. i'm having thoughts of doing it the "websocket" way and giving you either a "string" data (in this case, parsed JSON), or "binary" data (the custom data)

or maybe in the manager stub you have to specify the supported custom deserializers?

this is a confusing topic

[SEAPUNK]

lmao no

[SEAPUNK]

If someone wants it, they'll open up an issue for it, and I'll judge the use cases, and complexity of implementation. Most likely, things like those binary blobs could be done in another way, e.g. pointing to the binary blob resource that the runner's code could download, and process like that.

Yeah, I think that's much better.