Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ We are releasing this code to the public as a tool we expect others to use and a
### Chapter 1: Using DANDI/getting data
- [Downloading NWB files from DANDI](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/download_nwb.ipynb)
- [Reading NWB files](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/read_nwb.ipynb)
- [Streaming NWB files from DANDI](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/stream_nwb.ipynb)
- [Exploring NWB files with NWBWidgets](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/use_nwbwidgets.ipynb)
- [Streaming NWB files from DANDI](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/stream_nwb.ipynb)
- [Querying metadata across sessions from DANDI](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/basics/get_dandiset_metadata.ipynb)

### Chapter 2: Data visualization
Expand Down
9 changes: 8 additions & 1 deletion docs/basics/download_nwb.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,13 @@
"id": "a309c067",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"A newer version (0.55.1) of dandi/dandi-cli is available. You are using 0.46.6\n"
]
},
{
"name": "stdout",
"output_type": "stream",
Expand Down Expand Up @@ -192,7 +199,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
"version": "3.9.13"
}
},
"nbformat": 4,
Expand Down
61 changes: 61 additions & 0 deletions docs/basics/intro_information.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
Copy link
Collaborator

@rcpeene rcpeene Jun 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For step one, it reads as though you mean "click here to access any dandiset from the allen institute". Maybe say "access an example 2-Photon Dandiset from the allen institute openscope project"


Reply via ReviewNB

Copy link
Collaborator

@rcpeene rcpeene Jun 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

theres probably some more NWB links you could add in here


Reply via ReviewNB

"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Preliminary Knowledge"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### DANDI\n",
"Throughout these notebooks, we utilize a platform called [DANDI](https://dandiarchive.org/) (Data Archive and Neurophysiology Imaging). DANDI is a platform that allows open-source data sharing and archiving add acts as a centralized repository where research can deposit data. While we have provided some pre-loaded data from DANDI in the notebooks, the purpose of this databooks is to take any dataset off DANDI and reproduce the analysis within these notebooks. You should familiarize youself with DANDI before running through the notebooks, and explore how Dandisets are organized on the website. \n",
"\n",
"One step that may be confusing when running through the notebooks for the first time is finding a filepath for a certain file within a Dandiset. These are the steps for accessing a filepath you can try out yourself: \n",
"1) Click [here](https://dandiarchive.org/dandiset/000535?search=allen%20institute%202%20photon&pos=2) to access a Dandiset uploaded by researchers at the Allen Institute. \n",
"2) On the right side of the page, click the \"files\" tab.\n",
"3) You will now see a list of folders. Click on any folder. \n",
"4) Once you have entered the folder, you will see a list of files with 4 blue buttons to the right of each file name. Select any file you would like and click the \"i\" icon in the blue circle. \n",
"5) This will pull up a new tab with a bunch of red code. At the top of the code, you will see `\"id\" :`, `\"path\" :`, and `\"access\":`. Copy the code to the right of `\"path\" :`. It will look like this `\"sub-460654/sub-460654_ses-20190611T181840_behavior+ophys.nwb\"`. This is the filepath you will insert in various notebooks when it asks for `dandi_filepath`. \n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### NWB FILES\n",
"NWB (Neurodata Without Borders) files utilize a standardized format for storing and sharing neurophysiology data. Their purpose is to address the need for a universal data format that makes analysis accessible across different experimental techniques. NWB files contain raw data, processed data, analysis results, and metadata all organized in a uniform manner across different research projects. It would be useful to understand the format of these files before running through the notebooks because we will be accessing many different modules within various files to retreive data. Click [here](https://nwb-schema.readthedocs.io/en/latest/) to explore NWB format specification."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Understanding Data Colletion Techniques\n",
"In this databook, we will be analyzing data from two different types of experimental techniques: optical imagine (ophys) and extracellular electrophysiology (ecephys). To truly grasp what is happening during the analysis process, you should familiarize youself with these experimental modalities. \n",
"\n",
"Resouces for ophys:\n",
"1) [this paper](https://www.pnas.org/doi/epdf/10.1073/pnas.1232232100) will provide an introduction to two-photon calcium imaging. \n",
"\n",
"Resources for ecephys:\n",
"1) [this paper](https://www.nature.com/articles/s41586-020-03171-x) from the Allen Institute will provide an introduction to ecephys and the use of neuropixel probes. \n",
"2) [here](https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels) is a visualization of the neuropixel probes that may come in handy when trying to visualualize how the data is collected from the probe itself."
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
11 changes: 8 additions & 3 deletions docs/basics/read_nwb.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,10 @@
"outputs": [],
"source": [
"filename = dandi_filepath.split(\"/\")[-1]\n",
"filepath = f\"{download_loc}/{filename}\""
"filepath = f\"{download_loc}/{filename}\"\n",
"\n",
"print(filename)\n",
"print(filepath)"
]
},
{
Expand Down Expand Up @@ -221,7 +224,9 @@
"outputs": [],
"source": [
"### uncomment these to view aspects of the file\n",
"### note that not all these properties exist for all nwb files\n",
"\n",
"### not all of these exist for all NWB files (Key Errors will arise if the fields don't exist for this file)\n",
"\n",
"# nwb.identifier\n",
"# nwb.processing\n",
"# nwb.acquisition[\"events\"]\n",
Expand Down Expand Up @@ -6393,7 +6398,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
"version": "3.9.13"
}
},
"nbformat": 4,
Expand Down
32 changes: 13 additions & 19 deletions docs/basics/stream_nwb.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"id": "df1c4cce",
"metadata": {},
"outputs": [],
Expand Down Expand Up @@ -66,7 +66,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 2,
"id": "f3f97f13",
"metadata": {},
"outputs": [],
Expand All @@ -79,17 +79,10 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 3,
"id": "a51caf90",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"A newer version (0.53.0) of dandi/dandi-cli is available. You are using 0.46.6\n"
]
},
{
"name": "stdout",
"output_type": "stream",
Expand All @@ -110,15 +103,15 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 4,
"id": "d131ad56",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Retrieved file url https://dandiarchive.s3.amazonaws.com/blobs/f5f/175/f5f1752f-5227-47d5-8f75-cd71937878aa?response-content-disposition=attachment%3B%20filename%3D%22sub-699733573_ses-715093703.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20230420%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230420T201648Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=6088e2110f82ce358dec90e827468a37fd995c34e80089139ff1056659e1ccb8\n"
"Retrieved file url https://dandiarchive.s3.amazonaws.com/blobs/f5f/175/f5f1752f-5227-47d5-8f75-cd71937878aa?response-content-disposition=attachment%3B%20filename%3D%22sub-699733573_ses-715093703.nwb%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20230607%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230607T170103Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=47c78e4993c637e57e9e6296dfd5f0f52d06f4d33574280ca6c527ebd4ed2869\n"
]
}
],
Expand Down Expand Up @@ -152,9 +145,9 @@
"name": "stderr",
"output_type": "stream",
"text": [
"c:\\Users\\carter.peene\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.5.1 is already loaded.\n",
"c:\\Users\\katrina.ager\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.6.0 is already loaded.\n",
" warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n",
"c:\\Users\\carter.peene\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.2.2 because version 2.6.0-alpha is already loaded.\n",
"c:\\Users\\katrina.ager\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.2.2 because version 2.5.0 is already loaded.\n",
" warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n"
]
}
Expand Down Expand Up @@ -182,13 +175,14 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 10,
"id": "91031da2",
"metadata": {},
"outputs": [],
"source": [
"### uncomment these to view aspects of the file\n",
"### note that not all these properties exist for all nwb files\n",
"### not all of these exist for all NWB files (Key Errors will arise if the fields don't exist for this file)\n",
"\n",
"# nwb.identifier\n",
"# nwb.processing\n",
"# nwb.acquisition[\"events\"]\n",
Expand All @@ -199,14 +193,14 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 11,
"id": "3e06b964",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "770645b2edf24149ad7158e0ea7c3096",
"model_id": "f5c84701e3b14599ac2adcf1fa35d89d",
"version_major": 2,
"version_minor": 0
},
Expand Down Expand Up @@ -239,7 +233,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
"version": "3.9.13"
}
},
"nbformat": 4,
Expand Down
38 changes: 22 additions & 16 deletions docs/basics/use_nwbwidgets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
"metadata": {},
Copy link
Collaborator

@rcpeene rcpeene Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be reworded/restructured to seem a bit more natural with the rest of the text. Perhaps something like "In the cells below, the dandi api i used to facilitate the download. the method... etc. etc. and get_asset_by_path does blah"


Reply via ReviewNB

Copy link
Collaborator

@rcpeene rcpeene Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make one paragraph, fix spelling typo 'specific'


Reply via ReviewNB

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it already is one paragraph?

"source": [
"### Downloading an NWB File\n",
"To examine an NWB File locally, it must first be downloaded. `dandiset_id` and `dandi_filepath` may be changed to select a different file off of DANDI. If the file of interest already downloaded, you don't need to run the download cell again. When trying to download an embargoed file, refer to the code from the [Downloading an NWB File](./download_nwb.ipynb) notebook."
"To examine an NWB File locally, it must first be downloaded. `dandiset_id` and `dandi_filepath` may be changed to select a different file off of DANDI. If the file of interest already downloaded, you don't need to run the download cell again. When trying to download an embargoed file, refer to the code from the [Downloading an NWB File](./download_nwb.ipynb) notebook. In the cells below, the DANDI API is used to faciliate the download. The `get_dandiset()` methods retrieves a specific dataset within DANDI while the `get_asset_by_path()` method retrieves a specific file within that dataset. "
]
},
{
Expand All @@ -73,10 +73,22 @@
"execution_count": 4,
"id": "83d7e5f7",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sub-699733573_ses-715093703.nwb\n",
"./sub-699733573_ses-715093703.nwb\n"
]
}
],
"source": [
"filename = dandi_filepath.split(\"/\")[-1]\n",
"filepath = f\"{download_loc}/{filename}\""
"filepath = f\"{download_loc}/{filename}\"\n",
"\n",
"print(filename)\n",
"print(filepath)"
]
},
{
Expand All @@ -85,13 +97,6 @@
"id": "1c5db129",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"A newer version (0.53.0) of dandi/dandi-cli is available. You are using 0.46.6\n"
]
},
{
"name": "stdout",
"output_type": "stream",
Expand All @@ -104,6 +109,7 @@
"client = dandiapi.DandiAPIClient()\n",
"my_dandiset = client.get_dandiset(dandiset_id)\n",
"file = my_dandiset.get_asset_by_path(dandi_filepath)\n",
"\n",
"# this may take awhile, especially if the file to download is large\n",
"file.download(filepath)\n",
"\n",
Expand Down Expand Up @@ -132,17 +138,17 @@
"name": "stderr",
"output_type": "stream",
"text": [
"c:\\Users\\carter.peene\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.5.1 is already loaded.\n",
"c:\\Users\\katrina.ager\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.6.0 is already loaded.\n",
" warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n",
"c:\\Users\\carter.peene\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.2.2 because version 2.6.0-alpha is already loaded.\n",
"c:\\Users\\katrina.ager\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\hdmf\\spec\\namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.2.2 because version 2.5.0 is already loaded.\n",
" warn(\"Ignoring cached namespace '%s' version %s because version %s is already loaded.\"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"root pynwb.file.NWBFile at 0x2285253481232\n",
"root pynwb.file.NWBFile at 0x2179856764832\n",
"Fields:\n",
" acquisition: {\n",
" raw_running_wheel_rotation <class 'pynwb.base.TimeSeries'>,\n",
Expand Down Expand Up @@ -191,7 +197,7 @@
" session_id: 715093703\n",
" session_start_time: 2019-01-19 00:54:18-08:00\n",
" stimulus_notes: brain_observatory_1.1\n",
" subject: subject abc.EcephysSpecimen at 0x2285271628048\n",
" subject: subject abc.EcephysSpecimen at 0x2179856428960\n",
"Fields:\n",
" age: P118D\n",
" age_in_days: 118.0\n",
Expand Down Expand Up @@ -223,7 +229,7 @@
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ff394f60aa434277a364cc6bd96495c5",
"model_id": "fb037fcb861e489896b6812becc84a8e",
"version_major": 2,
"version_minor": 0
},
Expand Down Expand Up @@ -256,7 +262,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
"version": "3.9.13"
}
},
"nbformat": 4,
Expand Down
Loading