Transform to numpy arrays

[simeneide]

Pytorch files are too platform specific. Instead, let the data be available as numpy arrays. This has the same amount of compression. In addition, we make a couple of name changes to more intuitive names.

Todo

• Convert files from pytorch to numpy
• Name changes: Call the displayed slates "slates" instead of "action", call the type of interaction (recs or search) "interaction_type" instead of "displayType". Remove ind2user and ind2item as they are scrambled anyways. Remove the popularity count in itemattr object.
• Update the dataloader
• Update quickstart (wait until we also build the pip package?)

[simeneide]

Pushed some of your changes @NegatioN . I now have two files: "datahelper.py" and "dataset_torch.py" which contain relevant code. The idea is that we can also add a "dataset_tensorflow.py" etc to reduce needed dependencies.

In addition to these I think it makes sense to add some evaluation functions, which I will write in pytorch-lightning as a first implementation. So maybe a dataset_lightning.py that can inherit the dataset from dataset_torch maybe. Is this reasonable or too complicated?

[NegatioN]

Pushed some of your changes @NegatioN . I now have two files: "datahelper.py" and "dataset_torch.py" which contain relevant code. The idea is that we can also add a "dataset_tensorflow.py" etc to reduce needed dependencies.

(I already wrote this in a comment, but:) I would aim for a numpy generator that people who don't use PyTorch could use for their own dataloaders (ex: TensorFlow?)

In addition to these I think it makes sense to add some evaluation functions, which I will write in pytorch-lightning as a first implementation.

Maybe writing the evaluation functions for the framework you're using first is the right call. But I think it's wise to put some effort into separating the eval functions into a self-contained format, so that they can be used in both Lightning/Pytorch/regular Python? I can provide more feedback on those things, or try to help out, if you sketch out how the function itself is supposed to look like 🙂

So maybe a dataset_lightning.py that can inherit the dataset from dataset_torch maybe. Is this reasonable or too complicated?

The datasets look identical for Lightning and Pytorch, right? It's only the eval functions you're worried about?

[simeneide]

Pushed some of your changes @NegatioN . I now have two files: "datahelper.py" and "dataset_torch.py" which contain relevant code. The idea is that we can also add a "dataset_tensorflow.py" etc to reduce needed dependencies.

(I already wrote this in a comment, but:) I would aim for a numpy generator that people who don't use PyTorch could use for their own dataloaders (ex: TensorFlow?)

In addition to these I think it makes sense to add some evaluation functions, which I will write in pytorch-lightning as a first implementation.

Maybe writing the evaluation functions for the framework you're using first is the right call. But I think it's wise to put some effort into separating the eval functions into a self-contained format, so that they can be used in both Lightning/Pytorch/regular Python? I can provide more feedback on those things, or try to help out, if you sketch out how the function itself is supposed to look like 🙂

So maybe a dataset_lightning.py that can inherit the dataset from dataset_torch maybe. Is this reasonable or too complicated?

The datasets look identical for Lightning and Pytorch, right? It's only the eval functions you're worried about?

I agree on all these points! But its out of scope for this PR I think. It also requires us to be a bit careful so that all functions and dataset splits are equivalent. So need to be worked on slightly separate. But we gone one step with moving data from torch to numpy. so next steps would be to also move code to be only np-dependent. :) numpy generators could a good way to "fix" this.

[NegatioN]

LGTM :)