Transform repo into using nbdev framework

[simeneide]

nbdev is a framework from fast.ai that develops everything in notebooks. It its heavily opinionated, but also gives us documentation and pip packaging "for free". Ive transformed the repo into this format.

Todo: My main issue now is that I removed the following lines from .gitattributes which means git lfs no longer works. But nbdev did not build with these present..

*.pt filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.json filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text

[NegatioN]

.DS_Store should not be commited I think?

[SofieVerrewaere]

nice to have: A little tqdm bar to see how the download progresses? Can be added to the ToDo's if you want it.

[SofieVerrewaere]

Nice work!

[SofieVerrewaere]

Would be nice to elaborate a bit more on the Dev Cycle, maybe also add a linter (black?)?

[SofieVerrewaere]

Maybe add link to: https://github.com/fastai/nbdev/ to get people started ... and add run pip install nbdev or conda install -c fastai nbdev.

[simeneide]

yes, this is the standard text from nbdev, but its interesting that they did not include these links :)

[simeneide]

@SofieVerrewaere :

nice to have: A little tqdm bar to see how the download progresses? Can be added to the ToDo's if you want it.

The underlying download library allowed to show the downloaded file size as a progress. Added that as an option.

[simeneide]

Google colab fails when loading and transforming the numpy data into a torch array. The main issue is the "slates" data. It is currently stored as int64, but we can try to reduce it to int32. Compressed saving only reduces from 1.2gb to 1.1gb, but the in-memory usage is halfed: 4555290000/9110580000 = 0.5

logging.info('Load data..')
data = {}
with np.load("{}/data.npz".format(data_dir)) as data_np:
    for key in data_np.keys():
      # Load and close for each item:
      with np.load("{}/data.npz".format(data_dir)) as data_tmp:
        print(key, data_tmp[key].nbytes)
        data[key] = data_tmp[key].astype("int32")
        #data[key] = torch.tensor(data_tmp[key], dtype=torch.int)

[simeneide]

I removed the attributes from git lfs to make nbdev work. So a bug now is that git lfs is not working properly. For usage, however, this is less relevant as users can download the files from gdrive using the buildin function.

I also created a int32 version of the data.npz file that is uploaded to gdrive, decreasing the memory footprint and makes it usable in google colab. I'll merge with these fixes.