feat(dataflow): Use raw contents in joins if any message uses raw contents #4781
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/StreamTransforms.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/StreamTransforms.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Show resolved
Hide resolved
| val v = input.contents.boolContentsList.flatMap { ByteBuffer.allocate(1) | ||
| .order(ByteOrder.LITTLE_ENDIAN).put(if (it) {1} else {0}) .array().toList() }.toByteArray() | ||
| } | ||
| DataType.FP16, // Unclear if this is correct as will be stored as 4 byte floats |
There was a problem hiding this comment.
💭 It seems that Kotlin explicitly defines a Float as having 4 bytes/32 bits. Given that, we could just allocate Float.SIZE_BYTES / 2 for the FP16 type.
There was a problem hiding this comment.
The issue is the source is a FP32 even if they type says FP16 so I think the best we can do now is leave as this or try to see how Triton store FP16 in their raw input contents
There was a problem hiding this comment.
In the interests of fixing the immediate issue, it's probably fine to leave for now, or to explicitly just drop the FP16 contents on the basis that wrong answers are worse than no answers.
Agree we should revisit for FP16, and (U)INT(8|16) as those probably suffer from the same issue.
There was a problem hiding this comment.
I think the uints won't as we allocate the correct size and push the correct value as an uint. I have updated fp16 to throw an exception.
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
scheduler/data-flow/src/main/kotlin/io/seldon/dataflow/kafka/protocol.kt
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Have discussed offline. We'll leave the exception handling for now and I'll aim to revisit that soon to ensure exceptions in the data handling don't cause the entire application to keel over.
In the meantime, approving.
We should probably add docs that:
- If any message in a join uses raw payloads, all messages will be converted.
- All compliant inference servers should support raw payloads. If users use one that doesn't but don't use any raw payloads with it, that'd be safe, but it's up to users to get right.
agrski
changed the title
agrski
changed the title
Triton will fail if some inputs have raw contents and some do not.
This PR ensures we always just have raw contents if one input does and another does not when joining streams of data.
Fixes: #4776