Support NumPy for tensorflow-io

[yongtang]

In TensorFlow's guide of "Importing Data":
https://www.tensorflow.org/guide/datasets

It is possible to reading input data directly from TFRecord (TFRecordDataset), text (TextLineDataset ), csv (CsvDataset) but not with NumPy. Reading input from NumPy still have to use a not so elegant way in the example code of the TensorFlow Guide:

with np.load("/var/data/training_data.npy") as data:
  features = data["features"]
  labels = data["labels"]

# Assume that each row of `features` corresponds to the same row as `labels`.
assert features.shape[0] == labels.shape[0]

dataset = tf.data.Dataset.from_tensor_slices((features, labels))

It should be possible to implement NumPy support so that reading input from numpy could be done in a similar fashion as other input format. This potentially could also improve the performance as it may not be needed to read everything into the memory immediately (remotely related: tensorflow/tensorflow#16933).

[BryanCutler]

@yongtang Arrow supports reading numpy arrays to record batches. So I don't think it would be much effort to add this, but there would be a limit on dimensionality - for now at least. Hopefully that will change in the future.

[yongtang]

@BryanCutler I think there are two issues, one is the conversion between Apache Arrow and Numpy in memory, another is to read data from npy or npz file format. I haven't find out a way to open npy or npz from Arrow library. However, it seems like the npy or npz file format are very much straightforward, a simple parser should be enough. I will look into it and see if I could come up with something quick.

[BryanCutler]

Actually the conversion from Numpy to Arrow is zero copy, so it wouldn't consume any more memory, but you're right, Arrow doesn't support reading these files. If you're able to read them directly in the op that would be cool!

[areeh]

Performant solutions using tf.data and disk are quite painful right now, so this would be a welcome addition. I just want to mention that it's particularly useful if you support a case where data looks like:

{ 'feature1': ([...], dtype=np.float32), 'feature2': ([...], dtype=np.int16), }
etc. Somewhat similar to the example from the guide in the split between feature and label, if I'm understanding correctly

[yongtang]

@areeh PR #407 has been opened which should support your case. The PR allows to read a local numpy through the same address space.

It may have limitations but if your process is local then performance might be good for large numpy array (as there is no serialization overhead before hand).

The dict/tuple of the features has been added as well.

[areeh]

@yongtang The PR looks great, it supports everything I had in mind when I wrote the comment. Thank you

[kvignesh1420]

@yongtang can this be closed?

[yongtang]

@kvignesh1420 Ah yes thanks for the reminder 👍