Dandi hub example

[jbentvelsen]

This pull request contains two main parts:

1. An improved version of the custom datastore. The new version does also accept nwb files (instead of tiff only).
2. A live script (dandi_example.mlx) that shows how to access a dataset from DANDI archive and how to manually read frames from it. Then, we also show how to use the new datastore with the nwb dataset from dandi for training and inference. Note that the workflow shown there is very similar as the one in the customdatastore_example.

[jbentvelsen]

@tkuenzmw I have changed the datastore such that it also supports nwb files. I wanted to show how to use it with the dandi-sets. However, I have no idea what a good dataset on dandi is. And secondly, which network I should use with it. Do you have an idea about that? Right now I took a 'random' set from 000037 and the same network you used in the custom datastore example (as both are ophys data).

Secondly, training (at least without gpu) takes very long (over an hour, I don't know how long exactly). I am thus not sure yet whether this an useful example. @vijayiyer05 Do you think this example would still be worth including?

[jbentvelsen]

@vijayiyer05 Where/how can I make a link to open a particular live script in the Mathworks online viewer?

[vijayiyer05]

@vijayiyer05 Where/how can I make a link to open a particular live script in the Mathworks online viewer?

This tool can be used to create a link to Open in MATLAB Online from GitHub

But of course some of the larger examples may not be suitable for MATLAB Online

[tkuenzmw]

great additions to the customdatastore and Dandi Example!

[tkuenzmw]

The help information in the beginning of the file should be updated to reflect the changes in usage and capabilities

[tkuenzmw]

I think the example datafile is quite large, but it is hard to find raw ophys data. The demo itself looks good.

[stevevanhooser]

@tkuenzmw @vijayiyer05

Ready for me to give it a final test and merge? It seems to be out of draft mode and submitted for final review.

Thanks
Steve

[stevevanhooser]

Hi all -

In an effort to get out version 7 before Dec 15, @vijayiyer05 and I discussed setting the Dandi integration aside for a little bit.

In consultation with @vijayiyer05, the 50GB file is a bit unwieldy.

There are a few issues with downloading the dandi file.

• For me, the file doesn't download, even if I call the dandi-cli tool from my terminal.
• If we are going to ask the users to install the dandi-cli, then I think we ought to tell them how to do it so they don't have to read too much (easy as pip install dandi).
• The code as it is now calls the dandi command from Matlab via the system command. However, it is not on my path, and I doubt it will be on the path of many users, given that it is in the Python library. So if we're going to do that we ought to have some good solution. Not sure if telling users how to add it to their Matlab path is easier, or walking them through a download on the command line.

I made a new branch ongoing_dandihub_work that continues the dandi work.