I/O operations similar to Numpy

[AlexJuca]

Will Nx support I/O operations similar to those found in Numpy's npyio?

Functions like:

np.save("tmp/123", np.array([[1, 1, 0], [0, 1, 0]]))
np.load("tmp/123.npy")

if so, will these functions be packaged into their own module or called directly from Nx?

[josevalim]

We can discuss it, I am interested in the use cases. :)

[AlexJuca]

@josevalim It is common in the ML community to save numpy arrays to disk so they can used at a later time.

And from my research the functionality fit the following use cases:

1. To save predictions from a model and reuse these predictions later.
2. To save pre-processed Tensors like a corpus of text (integers) or a collection of rescaled image data (pixels) and reuse-them later.

In essence, these example use cases capture the main idea:

Saving Tensors to file so they can be loaded up quickly and reused later.

It would also be interesting to see how these saved tensors could be used in a distributed system. Perhaps saving the file to one or multiple nodes.

Or loading very large tensors saved in binary compressed format on various nodes to be processed.

[AlexJuca]

It is common to save to .csv files but loading and saving gigabytes of data using this format is slow.
Numpy solves that with a binary format that is much more efficient when loading and saving to disk.

[josevalim]

Right! For NN though, they have their own formats, so I am wondering if most tools will have their own formats in top, meaning ours won’t be used much. :)

[AlexJuca]

Numpy has it's own .npy binary format and this format seems to be used in the R community.
R also has it's own .rdata format.

We could add support for reading .npy file if it is common enough and try to convert to valid Nx tensor.

This might be interesting for python developers who have workflows that use .npy files and want to experiment with elixir.

This would essentially be like reading another format like .csv and converting to a valid tensor.

But yeah, most tools have their own specialized binary format to save large arrays.

[shoz-f]

Hello wizards.
A few days ago, I reworked my Npy module to load/save Nx.tensor into npy/npz files.

I created the original Npy module to take the tensor processed by python's tensorflow and use it in the tensorflow-lite module on Nerves. The initial specification was to load/save my own %Npy{} into a npy file.

If you are interested, please visit my github:
https://github.com/shoz-f/npy_ex

[AlexJuca]

@shoz-f Nice! I will take a look at your implementation. It would be nice for this to be part of Nx itsel in my opinion. Tell me from your experience, how common is this feature in a ML researcher or practitioners workflow?

[sekunho]

Are there any temporary workarounds for this? I don't intend to use the model elsewhere, just within Axon. But I'm having trouble thinking of an approach that allows me to reuse a trained model later on, or in a different environment/application instance. Like, training locally, and using the model in a production environment. Or maybe, just to apply the model in a different instance.

[josevalim]

@imsekun if production and dev have the same endianess, then it is a matter of :erlang.term_to_binary to serialize it and write the result to a file. Then File.read! and :erlang.binary_to_term to read it.

[sekunho]

@josevalim Oh wow TIL that existed. Awesome. Thanks so much!

[josevalim]

We currently have from_numpy and from_numpy_archive, which are Numpy specific. I will open up a new discussion about file storage for Nx.