[94] Support for strings #97
Conversation
|
What does this PR enable? Loading images into TensorFrames? ? Can you make the description a little more descriptive :-) |
| } | ||
|
|
||
| // ********** STRING ********* | ||
| // This is actually byte arrays, which corresponds to the 'binary' type in Spark. |
There was a problem hiding this comment.
what if you pass in a real string column?? wouldn't be cleaner to explicitly pass in bytearrays?
There was a problem hiding this comment.
The String type in Spark is different: it corresponds to the UTF-8 encoded representation of textual data, and it would not be accepted. The reason for that is that you would need to specify the encoding (UTF-8, UTF-16, etc.) in order to convert it to a byte array. In order to bypass this problem, tensorflow only accepts byte arrays (which they call 'strings').
Under the hood, though, Spark passes byte arrays to this function (see appendRaw below)
|
|
||
| override def convertTensor(t: tf.Tensor): MWrappedArray[Array[Byte]] = { | ||
| // TODO(tjh) implement later | ||
| ??? |
There was a problem hiding this comment.
what's happening here?
1/ does it compile?
2/ are these not needed??
There was a problem hiding this comment.
Oh yes, this is a special scala syntax for holes in the program. It should be replaced by proper exceptions.
|
Merging this PR, thanks @sueann for taking a look! |
2 participants
Adds support for ingesting byte arrays (the
binarytype in spark, which corresponds to thestringtype in tensorflow).This also refactors some of the internals to ensure that the dependencies to tensorflow and spark are contained to the encoding and the decoding. This helps significantly when checking for the correctness of the code.
Includes an integration test that reads a binary string in spark.
#94