docs/basics/get_dandiset_metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b8d63bf5",
   "metadata": {
    "id": "b8d63bf5"
   },
   "source": [
    "# Getting Experimental Metadata from DANDI\n",
    "It can be helpful to view general information about the experimental sessions that produced your data. Since typically each NWB File represents one session, a dandiset's files can be examined to get an overview of each of the sessions. This can vary, depending on who produced the NWB file. In this notebook, NWB Files within one of the Allen Institute's datasets are opened and some basic information is used to make a table of the experimental sessions and their properties."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "410df685",
   "metadata": {
    "id": "410df685"
   },
   "source": [
    "### Environment Setup\n",
    "⚠️**Note: If running on a new environment, run this cell once and then restart the kernel**⚠️"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1859d9e1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "try:\n",
    "    from databook_utils.dandi_utils import dandi_stream_open\n",
    "except:\n",
    "    !git clone https://github.com/AllenInstitute/openscope_databook.git\n",
    "    %cd openscope_databook\n",
    "    %pip install -e ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2bbff7f6",
   "metadata": {
    "id": "2bbff7f6",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import h5py\n",
    "import pandas as pd\n",
    "import remfile\n",
    "\n",
    "from dandi import dandiapi\n",
    "from pynwb import NWBHDF5IO\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a874b37c",
   "metadata": {
    "id": "a874b37c"
   },
   "source": [
    "### Getting Dandiset Metadata\n",
    "To view other data, change `dandiset_id` to be the id of the dandiset you're interested in. If the dandiset is embargoed, set `dandi_api_key` to your DANDI API key. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c01caed9",
   "metadata": {
    "id": "c01caed9",
    "tags": []
   },
   "outputs": [],
   "source": [
    "dandiset_id = \"000248\"\n",
    "dandi_api_key = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5648ccf4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5648ccf4",
    "outputId": "811a0cc5-5472-488d-bc9d-4a7814fa4760",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got dandiset DANDI:000248/0.240502.2344\n"
     ]
    }
   ],
   "source": [
    "my_dandiset = dandiapi.DandiAPIClient(token=dandi_api_key).get_dandiset(dandiset_id)\n",
    "print(f\"Got dandiset {my_dandiset}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8130105e",
   "metadata": {
    "id": "8130105e"
   },
   "source": [
    "### Get NWB Info\n",
    "Below are two definitions of thefunction `get_nwb_info`. These function are tailored to our NWB Files; Our *Ophys* and our *Ecephys* datasets respectively. It retrieves a series of important metadata values from the NWB file object. It is likely that the code for accessing the fields of interest to you will be slightly different for your files. This can easily be altered to extract any other information from an NWB file you want as long as you're familiar with the internal layout of your files. However, make sure to change the `columns` field in the pandas dataframe below to properly reflect any changes to the function. `nwb_type` can be set below to 'ephys' or 'ophys' depending on the type of the NWBs of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2a410585",
   "metadata": {},
   "outputs": [],
   "source": [
    "nwb_type = \"ephys\" # or, \"ophys\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "14c5933e-07e0-4a27-be90-06ee6bf27a19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get experimental information from within ophys file\n",
    "# getattr is used because not all nwb files have all properties. If not handled like this, errors will arise\n",
    "def get_ophys_nwb_info(nwb):\n",
    "        session_time = getattr(nwb, \"session_start_time\", None)\n",
    "\n",
    "        metadata_obj = getattr(nwb, \"lab_meta_data\", {})\n",
    "        metadata = metadata_obj.get(\"metadata\", None)\n",
    "        session_id = getattr(metadata, \"ophys_session_id\", None)\n",
    "        experiment_id = getattr(metadata, \"ophys_experiment_id\", None)\n",
    "\n",
    "        fov_height = getattr(metadata, \"field_of_view_height\", None)\n",
    "        fov_width = getattr(metadata, \"field_of_view_width\", None)\n",
    "        imaging_depth = getattr(metadata, \"imaging_depth\", None)\n",
    "        group = getattr(metadata, \"imaging_plane_group\", None)\n",
    "        group_count = getattr(metadata, \"imaging_plane_group_count\", None)\n",
    "        container_id = getattr(metadata, \"experiment_container_id\", None)\n",
    "        \n",
    "        subject = getattr(nwb, \"subject\", None)\n",
    "        specimen_name = getattr(subject, \"subject_id\", None)\n",
    "        age = getattr(subject, \"age\", None)\n",
    "        sex = getattr(subject, \"sex\", None)\n",
    "        genotype = getattr(subject, \"genotype\", None)\n",
    "        \n",
    "        try: n_rois = nwb.processing[\"ophys\"][\"dff\"].roi_response_series[\"traces\"].data.shape[1]\n",
    "        except: n_rois = None\n",
    "        try: location = list(nwb.imaging_planes.values())[0].location\n",
    "        except: location = None\n",
    "        \n",
    "        intervals = getattr(nwb, \"intervals\", {})\n",
    "        stim_types = set(intervals.keys())\n",
    "        stim_tables = [intervals[table_name] for table_name in intervals]\n",
    "        # gets highest value among final \"stop times\" of all stim tables in intervals\n",
    "        session_end = max([table.stop_time[-1] for table in stim_tables if len(table) > 1])\n",
    "\n",
    "        return [session_time, specimen_name, session_id, experiment_id, container_id, group, group_count, imaging_depth, location, fov_height, fov_width, specimen_name, sex, age, genotype, stim_types, n_rois, session_end]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "aa93e289",
   "metadata": {
    "id": "aa93e289",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get experimental information from within ecephys file\n",
    "# getattr is used because not all nwb files have all properties. If not handled like this, errors will arise\n",
    "def get_ephys_nwb_info(nwb):\n",
    "        session_time = getattr(nwb, \"session_start_time\", None)\n",
    "\n",
    "        subject = getattr(nwb, \"subject\", None)\n",
    "        specimen_name = getattr(subject, \"specimen_name\", None)\n",
    "        age = getattr(subject, \"age_in_days\", None)\n",
    "        sex = getattr(subject, \"sex\", None)\n",
    "        genotype = getattr(subject, \"genotype\", None)\n",
    "\n",
    "        probes = set(getattr(nwb, \"devices\", {}).keys())\n",
    "        units = getattr(nwb, \"units\", [])\n",
    "        n_units = len(units) if hasattr(units, '__len__') else 0        \n",
    "        \n",
    "        intervals = getattr(nwb, \"intervals\", {})\n",
    "        stim_types = set(intervals.keys())\n",
    "        stim_tables = [intervals[table_name] for table_name in intervals]\n",
    "        # gets highest value among final \"stop times\" of all stim tables in intervals\n",
    "        session_end = max([table.stop_time[-1] for table in stim_tables if len(table) > 1])\n",
    "\n",
    "        return [session_time, specimen_name, sex, age, genotype, probes, stim_types, n_units, session_end]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4f7e1d",
   "metadata": {
    "id": "4f4f7e1d"
   },
   "source": [
    "### Getting Table\n",
    "Here, each relevant file in the dandiset is streamed and opened remotely to get the information of interest using the function `get_nwb_info`, defined above, and then it is added to a table of sessions and their metadata. Since some files are for specific probes rather than entire sessions, they are skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "60ad89ca",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "60ad89ca",
    "outputId": "0502693e-0cf0-44f5-aa60-3229a77487f0",
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12 files retrieved\n",
      "Examining file 1/12: 9ab6bfff-70ed-44dc-b384-96b4cef2b566\n",
      "Examining file 2/12: 15bbb781-f912-4630-b9fe-f3df864290ad\n",
      "Examining file 3/12: f7b65765-41b5-4605-9585-95338c3a9e5a\n",
      "Examining file 4/12: 6b321a1c-55c4-4de5-8a25-373f2c5a4bc8\n",
      "Examining file 5/12: 0073a783-5b41-42a8-882a-2960554d4e43\n",
      "Examining file 6/12: 1b7aaf88-eabd-46f5-8f2a-c1fecba823f2\n",
      "Examining file 7/12: 3e3d9ce2-4d45-4f41-a957-532cbdf5bc39\n",
      "Examining file 8/12: f1a26076-0a3e-43c8-b138-5320bca7a23f\n",
      "Examining file 9/12: bc340647-7b72-4ec8-aa86-ac236da36713\n",
      "Examining file 10/12: efb53d2c-2d51-4a26-9b36-5d3d3ff7e19e\n",
      "Examining file 11/12: e61edcf3-26e9-44e1-9017-3ee558197da5\n",
      "Examining file 12/12: 233e6f43-d51e-47f0-9cda-838444c274f6\n"
     ]
    }
   ],
   "source": [
    "nwb_table = []\n",
    "# skip files that aren't main session files\n",
    "files = [asset for asset in my_dandiset.get_assets() if \"probe\" not in asset.path]\n",
    "# swap this with line above for one of our ophys dandisets\n",
    "# files = [asset for asset in my_dandiset.get_assets() if \"raw\" not in asset.path]\n",
    "n_files = len(files)\n",
    "print(f\"{n_files} files retrieved\")\n",
    "\n",
    "extract_nwb_info = get_ephys_nwb_info if nwb_type == \"ephys\" else get_ophys_nwb_info\n",
    "\n",
    "for i, file in enumerate(files):\n",
    "    print(f\"Examining file {i+1}/{n_files}: {file.identifier}\")    \n",
    "    # get basic file metadata\n",
    "    row = [file.identifier, file.size, file.path]\n",
    "    \n",
    "    base_url = file.client.session.head(file.base_download_url)\n",
    "    file_url = base_url.headers[\"Location\"]\n",
    "\n",
    "    # open and read nwb file with streaming\n",
    "    rem_file = remfile.File(file_url)\n",
    "    h5py_file = h5py.File(rem_file, \"r\")\n",
    "    io = NWBHDF5IO(file=h5py_file, mode=\"r\", load_namespaces=True)\n",
    "    nwb = io.read()\n",
    "\n",
    "    # extract experimental info from within file\n",
    "    row += extract_nwb_info(nwb)\n",
    "    nwb_table.append(row)\n",
    "    del nwb\n",
    "\n",
    "    # don't run full loop if running in test environment\n",
    "    if os.environ.get(\"TESTING\", False):\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0de97d1d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "0de97d1d",
    "outputId": "09dbafe7-af3e-48d9-9094-69da1818b6c7",
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>identifier</th>\n",
       "      <th>size</th>\n",
       "      <th>path</th>\n",
       "      <th>session_time</th>\n",
       "      <th>sub_name</th>\n",
       "      <th>sub_sex</th>\n",
       "      <th>sub_age</th>\n",
       "      <th>sub_genotype</th>\n",
       "      <th>probes</th>\n",
       "      <th>stim types</th>\n",
       "      <th>#_units</th>\n",
       "      <th>session_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9ab6bfff-70ed-44dc-b384-96b4cef2b566</td>\n",
       "      <td>3308934228</td>\n",
       "      <td>sub-633229/sub-633229_ses-1199247593_ogen.nwb</td>\n",
       "      <td>2022-08-17 00:00:00-07:00</td>\n",
       "      <td>633229</td>\n",
       "      <td>F</td>\n",
       "      <td>101.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>3026</td>\n",
       "      <td>7279.58784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15bbb781-f912-4630-b9fe-f3df864290ad</td>\n",
       "      <td>2773818936</td>\n",
       "      <td>sub-631510/sub-631510_ses-1196157974_ogen.nwb</td>\n",
       "      <td>2022-08-03 00:00:00-07:00</td>\n",
       "      <td>631510</td>\n",
       "      <td>F</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2386</td>\n",
       "      <td>7339.21131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f7b65765-41b5-4605-9585-95338c3a9e5a</td>\n",
       "      <td>2717870004</td>\n",
       "      <td>sub-620334/sub-620334_ses-1189887297_ogen.nwb</td>\n",
       "      <td>2022-07-06 00:00:00-07:00</td>\n",
       "      <td>620334</td>\n",
       "      <td>M</td>\n",
       "      <td>154.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2092</td>\n",
       "      <td>7279.89479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6b321a1c-55c4-4de5-8a25-373f2c5a4bc8</td>\n",
       "      <td>3036007774</td>\n",
       "      <td>sub-620333/sub-620333_ses-1188137866_ogen.nwb</td>\n",
       "      <td>2022-06-30 00:00:00-07:00</td>\n",
       "      <td>620333</td>\n",
       "      <td>M</td>\n",
       "      <td>148.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2593</td>\n",
       "      <td>7283.08716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0073a783-5b41-42a8-882a-2960554d4e43</td>\n",
       "      <td>2108745405</td>\n",
       "      <td>sub-631570/sub-631570_ses-1194857009_ogen.nwb</td>\n",
       "      <td>2022-07-28 00:00:00-07:00</td>\n",
       "      <td>631570</td>\n",
       "      <td>F</td>\n",
       "      <td>92.0</td>\n",
       "      <td>Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>1789</td>\n",
       "      <td>7278.92195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1b7aaf88-eabd-46f5-8f2a-c1fecba823f2</td>\n",
       "      <td>2598789798</td>\n",
       "      <td>sub-625555/sub-625555_ses-1183070926_ogen.nwb</td>\n",
       "      <td>2022-06-09 00:00:00-07:00</td>\n",
       "      <td>625555</td>\n",
       "      <td>F</td>\n",
       "      <td>90.0</td>\n",
       "      <td>Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2621</td>\n",
       "      <td>7278.57204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3e3d9ce2-4d45-4f41-a957-532cbdf5bc39</td>\n",
       "      <td>3619691457</td>\n",
       "      <td>sub-625554/sub-625554_ses-1181330601_ogen.nwb</td>\n",
       "      <td>2022-06-01 00:00:00-07:00</td>\n",
       "      <td>625554</td>\n",
       "      <td>M</td>\n",
       "      <td>82.0</td>\n",
       "      <td>Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2930</td>\n",
       "      <td>7315.43515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>f1a26076-0a3e-43c8-b138-5320bca7a23f</td>\n",
       "      <td>2469141920</td>\n",
       "      <td>sub-619296/sub-619296_ses-1187930705_ogen.nwb</td>\n",
       "      <td>2022-06-29 00:00:00-07:00</td>\n",
       "      <td>619296</td>\n",
       "      <td>M</td>\n",
       "      <td>154.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>1918</td>\n",
       "      <td>7278.13709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bc340647-7b72-4ec8-aa86-ac236da36713</td>\n",
       "      <td>2709636662</td>\n",
       "      <td>sub-630506/sub-630506_ses-1192952695_ogen.nwb</td>\n",
       "      <td>2022-07-20 00:00:00-07:00</td>\n",
       "      <td>630506</td>\n",
       "      <td>F</td>\n",
       "      <td>92.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2517</td>\n",
       "      <td>7279.14674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>efb53d2c-2d51-4a26-9b36-5d3d3ff7e19e</td>\n",
       "      <td>2803551453</td>\n",
       "      <td>sub-625545/sub-625545_ses-1182865981_ogen.nwb</td>\n",
       "      <td>2022-06-08 00:00:00-07:00</td>\n",
       "      <td>625545</td>\n",
       "      <td>M</td>\n",
       "      <td>89.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2793</td>\n",
       "      <td>7279.21336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>e61edcf3-26e9-44e1-9017-3ee558197da5</td>\n",
       "      <td>2453451408</td>\n",
       "      <td>sub-619293/sub-619293_ses-1184980079_ogen.nwb</td>\n",
       "      <td>2022-06-16 00:00:00-07:00</td>\n",
       "      <td>619293</td>\n",
       "      <td>M</td>\n",
       "      <td>141.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2136</td>\n",
       "      <td>7454.53444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>233e6f43-d51e-47f0-9cda-838444c274f6</td>\n",
       "      <td>2557113684</td>\n",
       "      <td>sub-637484/sub-637484_ses-1208667752_ogen.nwb</td>\n",
       "      <td>2022-09-08 00:00:00-07:00</td>\n",
       "      <td>637484</td>\n",
       "      <td>M</td>\n",
       "      <td>92.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2373</td>\n",
       "      <td>7349.25154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              identifier        size  \\\n",
       "0   9ab6bfff-70ed-44dc-b384-96b4cef2b566  3308934228   \n",
       "1   15bbb781-f912-4630-b9fe-f3df864290ad  2773818936   \n",
       "2   f7b65765-41b5-4605-9585-95338c3a9e5a  2717870004   \n",
       "3   6b321a1c-55c4-4de5-8a25-373f2c5a4bc8  3036007774   \n",
       "4   0073a783-5b41-42a8-882a-2960554d4e43  2108745405   \n",
       "5   1b7aaf88-eabd-46f5-8f2a-c1fecba823f2  2598789798   \n",
       "6   3e3d9ce2-4d45-4f41-a957-532cbdf5bc39  3619691457   \n",
       "7   f1a26076-0a3e-43c8-b138-5320bca7a23f  2469141920   \n",
       "8   bc340647-7b72-4ec8-aa86-ac236da36713  2709636662   \n",
       "9   efb53d2c-2d51-4a26-9b36-5d3d3ff7e19e  2803551453   \n",
       "10  e61edcf3-26e9-44e1-9017-3ee558197da5  2453451408   \n",
       "11  233e6f43-d51e-47f0-9cda-838444c274f6  2557113684   \n",
       "\n",
       "                                             path              session_time  \\\n",
       "0   sub-633229/sub-633229_ses-1199247593_ogen.nwb 2022-08-17 00:00:00-07:00   \n",
       "1   sub-631510/sub-631510_ses-1196157974_ogen.nwb 2022-08-03 00:00:00-07:00   \n",
       "2   sub-620334/sub-620334_ses-1189887297_ogen.nwb 2022-07-06 00:00:00-07:00   \n",
       "3   sub-620333/sub-620333_ses-1188137866_ogen.nwb 2022-06-30 00:00:00-07:00   \n",
       "4   sub-631570/sub-631570_ses-1194857009_ogen.nwb 2022-07-28 00:00:00-07:00   \n",
       "5   sub-625555/sub-625555_ses-1183070926_ogen.nwb 2022-06-09 00:00:00-07:00   \n",
       "6   sub-625554/sub-625554_ses-1181330601_ogen.nwb 2022-06-01 00:00:00-07:00   \n",
       "7   sub-619296/sub-619296_ses-1187930705_ogen.nwb 2022-06-29 00:00:00-07:00   \n",
       "8   sub-630506/sub-630506_ses-1192952695_ogen.nwb 2022-07-20 00:00:00-07:00   \n",
       "9   sub-625545/sub-625545_ses-1182865981_ogen.nwb 2022-06-08 00:00:00-07:00   \n",
       "10  sub-619293/sub-619293_ses-1184980079_ogen.nwb 2022-06-16 00:00:00-07:00   \n",
       "11  sub-637484/sub-637484_ses-1208667752_ogen.nwb 2022-09-08 00:00:00-07:00   \n",
       "\n",
       "   sub_name sub_sex  sub_age                                     sub_genotype  \\\n",
       "0    633229       F    101.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "1    631510       F     99.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "2    620334       M    154.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "3    620333       M    148.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "4    631570       F     92.0  Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "5    625555       F     90.0  Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "6    625554       M     82.0  Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "7    619296       M    154.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "8    630506       F     92.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "9    625545       M     89.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "10   619293       M    141.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "11   637484       M     92.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "\n",
       "                                               probes  \\\n",
       "0   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "1   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "2   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "3   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "4   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "5   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "6   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "7   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "8   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "9   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "10  {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "11  {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "\n",
       "                                           stim types  #_units  session_length  \n",
       "0   {spontaneous_presentations, ICwcfg1_presentati...     3026      7279.58784  \n",
       "1   {spontaneous_presentations, ICwcfg1_presentati...     2386      7339.21131  \n",
       "2   {spontaneous_presentations, ICwcfg1_presentati...     2092      7279.89479  \n",
       "3   {spontaneous_presentations, ICwcfg1_presentati...     2593      7283.08716  \n",
       "4   {spontaneous_presentations, ICwcfg1_presentati...     1789      7278.92195  \n",
       "5   {spontaneous_presentations, ICwcfg1_presentati...     2621      7278.57204  \n",
       "6   {spontaneous_presentations, ICwcfg1_presentati...     2930      7315.43515  \n",
       "7   {spontaneous_presentations, ICwcfg1_presentati...     1918      7278.13709  \n",
       "8   {spontaneous_presentations, ICwcfg1_presentati...     2517      7279.14674  \n",
       "9   {spontaneous_presentations, ICwcfg1_presentati...     2793      7279.21336  \n",
       "10  {spontaneous_presentations, ICwcfg1_presentati...     2136      7454.53444  \n",
       "11  {spontaneous_presentations, ICwcfg1_presentati...     2373      7349.25154  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert table to pandas dataframe\n",
    "if nwb_type == \"ephys\":\n",
    "    dandiset_files = pd.DataFrame(nwb_table, columns=(\"identifier\", \"size\", \"path\", \"session_time\", \"sub_name\", \"sub_sex\", \"sub_age\", \"sub_genotype\", \"probes\", \"stim types\", \"#_units\", \"session_length\"))\n",
    "else:\n",
    "    dandiset_files = pd.DataFrame(nwb_table, columns=(\"identifier\", \"size\", \"path\", \"session_time\", \"sub_name\", \"session_id\", \"experiment_id\", \"container_id\", \"group\", \"group_count\", \"imaging_depth\", \"location\", \"fov_height\", \"fov_width\", \"specimen_name\", \"sub_sex\", \"sub_age\", \"sub_genotype\", \"stim_types\", \"#_rois\", \"session_end\"))\n",
    "dandiset_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b3f6cd66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dandiset Overview:\n",
      "12 dandiset_files\n",
      "12 subjects 7 males, 5 females\n",
      "9 sst, 3 pval, 0 wt\n"
     ]
    }
   ],
   "source": [
    "if nwb_type == \"ephys\":\n",
    "    m_count = len(dandiset_files[\"sub_sex\"][dandiset_files[\"sub_sex\"] == \"M\"])\n",
    "    f_count = len(dandiset_files[\"sub_sex\"][dandiset_files[\"sub_sex\"] == \"F\"])\n",
    "    sst_count = len(dandiset_files[dandiset_files[\"sub_genotype\"].str.count(\"Sst\") >= 1])\n",
    "    pval_count = len(dandiset_files[dandiset_files[\"sub_genotype\"].str.count(\"Pval\") >= 1])\n",
    "    wt_count = len(dandiset_files[dandiset_files[\"sub_genotype\"].str.count(\"wt/wt\") >= 1])\n",
    "    print(\"Dandiset Overview:\")\n",
    "    print(len(dandiset_files), \"dandiset_files\")\n",
    "    print(len(set(dandiset_files[\"sub_name\"])), \"subjects\", m_count, \"males,\", f_count, \"females\")\n",
    "    print(sst_count, \"sst,\", pval_count, \"pval,\", wt_count, \"wt\")\n",
    "else:\n",
    "    n_sessions = len(dandiset_files[\"session_id\"].value_counts())\n",
    "    subjects_info = dandiset_files.groupby([\"sub_name\", \"sub_sex\"]).size().reset_index().to_dict()\n",
    "    m_count = len([sex for sex in subjects_info[\"sub_sex\"].values() if sex == \"M\"])\n",
    "    f_count = len([sex for sex in subjects_info[\"sub_sex\"].values() if sex == \"F\"])\n",
    "    print(\"Dandiset Overview:\")\n",
    "    print(len(dandiset_files), \"dandiset_files\")\n",
    "    print(len(subjects_info[\"sub_name\"]), \"subjects\", m_count, \"males\", f_count,\"females\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "af5257a8-ddfc-4cf4-b6b3-b8765bd05409",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# output all session metadata to local CSV file\n",
    "dandiset_files.to_csv(\"dandiset_files.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45883f51",
   "metadata": {
    "id": "45883f51"
   },
   "source": [
    "### Selecting Files\n",
    "**Pandas** syntax can be used to filter the table above and select individual sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "db02b77e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 347
    },
    "id": "db02b77e",
    "outputId": "9b292cd9-dafd-4c9c-f78d-83e17d0689da",
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>identifier</th>\n",
       "      <th>size</th>\n",
       "      <th>path</th>\n",
       "      <th>session_time</th>\n",
       "      <th>sub_name</th>\n",
       "      <th>sub_sex</th>\n",
       "      <th>sub_age</th>\n",
       "      <th>sub_genotype</th>\n",
       "      <th>probes</th>\n",
       "      <th>stim types</th>\n",
       "      <th>#_units</th>\n",
       "      <th>session_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0073a783-5b41-42a8-882a-2960554d4e43</td>\n",
       "      <td>2108745405</td>\n",
       "      <td>sub-631570/sub-631570_ses-1194857009_ogen.nwb</td>\n",
       "      <td>2022-07-28 00:00:00-07:00</td>\n",
       "      <td>631570</td>\n",
       "      <td>F</td>\n",
       "      <td>92.0</td>\n",
       "      <td>Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>1789</td>\n",
       "      <td>7278.92195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>f1a26076-0a3e-43c8-b138-5320bca7a23f</td>\n",
       "      <td>2469141920</td>\n",
       "      <td>sub-619296/sub-619296_ses-1187930705_ogen.nwb</td>\n",
       "      <td>2022-06-29 00:00:00-07:00</td>\n",
       "      <td>619296</td>\n",
       "      <td>M</td>\n",
       "      <td>154.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>1918</td>\n",
       "      <td>7278.13709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>e61edcf3-26e9-44e1-9017-3ee558197da5</td>\n",
       "      <td>2453451408</td>\n",
       "      <td>sub-619293/sub-619293_ses-1184980079_ogen.nwb</td>\n",
       "      <td>2022-06-16 00:00:00-07:00</td>\n",
       "      <td>619293</td>\n",
       "      <td>M</td>\n",
       "      <td>141.0</td>\n",
       "      <td>Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt</td>\n",
       "      <td>{probeC, probeB, probeE, probeA, probeD, probe...</td>\n",
       "      <td>{spontaneous_presentations, ICwcfg1_presentati...</td>\n",
       "      <td>2136</td>\n",
       "      <td>7454.53444</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              identifier        size  \\\n",
       "4   0073a783-5b41-42a8-882a-2960554d4e43  2108745405   \n",
       "7   f1a26076-0a3e-43c8-b138-5320bca7a23f  2469141920   \n",
       "10  e61edcf3-26e9-44e1-9017-3ee558197da5  2453451408   \n",
       "\n",
       "                                             path              session_time  \\\n",
       "4   sub-631570/sub-631570_ses-1194857009_ogen.nwb 2022-07-28 00:00:00-07:00   \n",
       "7   sub-619296/sub-619296_ses-1187930705_ogen.nwb 2022-06-29 00:00:00-07:00   \n",
       "10  sub-619293/sub-619293_ses-1184980079_ogen.nwb 2022-06-16 00:00:00-07:00   \n",
       "\n",
       "   sub_name sub_sex  sub_age                                     sub_genotype  \\\n",
       "4    631570       F     92.0  Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "7    619296       M    154.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "10   619293       M    141.0    Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt   \n",
       "\n",
       "                                               probes  \\\n",
       "4   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "7   {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "10  {probeC, probeB, probeE, probeA, probeD, probe...   \n",
       "\n",
       "                                           stim types  #_units  session_length  \n",
       "4   {spontaneous_presentations, ICwcfg1_presentati...     1789      7278.92195  \n",
       "7   {spontaneous_presentations, ICwcfg1_presentati...     1918      7278.13709  \n",
       "10  {spontaneous_presentations, ICwcfg1_presentati...     2136      7454.53444  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_files = dandiset_files[dandiset_files[\"size\"] <= 2_500_000_000]\n",
    "# selected_files = dandiset_files[dandiset_files[\"sub_sex\"] == \"F\"]\n",
    "# selected_files = dandiset_files[dandiset_files[\"#_units\"] > 2900]\n",
    "selected_files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff4d5e44",
   "metadata": {
    "id": "ff4d5e44"
   },
   "source": [
    "### Downloading Selected Files\n",
    "To download the files, we use the same method that is explained in [Downloading an NWB File](./download_nwb.ipynb). This can be used with the paths from the selected sessions above to just download the files of interest.  Set `download_loc` to be the relative path of where the files should be downloaded. Note that if the files are large, this can take a long time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "18780a0b",
   "metadata": {
    "id": "18780a0b"
   },
   "outputs": [],
   "source": [
    "download_loc = \".\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9757fbaa",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9757fbaa",
    "outputId": "659f4fac-f9ec-4a84-8687-740fc66400b7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sub-619293/sub-619293_ses-1184980079_ogen.nwb',\n",
       " 'sub-619296/sub-619296_ses-1187930705_ogen.nwb',\n",
       " 'sub-631570/sub-631570_ses-1194857009_ogen.nwb'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_paths = set(selected_files.path)\n",
    "selected_paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9502306f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9502306f",
    "outputId": "5982baae-7a2f-4192-b3be-b03773298220",
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded file to ./sub-631570_ses-1194857009_ogen.nwb\n",
      "Downloaded file to ./sub-619293_ses-1184980079_ogen.nwb\n",
      "Downloaded file to ./sub-631570_ses-1194857009_ogen.nwb\n",
      "Downloaded file to ./sub-619296_ses-1187930705_ogen.nwb\n"
     ]
    }
   ],
   "source": [
    "for dandi_filepath in selected_paths:\n",
    "    filename = dandi_filepath.split(\"/\")[-1]\n",
    "    file = my_dandiset.get_asset_by_path(dandi_filepath)\n",
    "    file.download(f\"{download_loc}/{filename}\")\n",
    "    print(f\"Downloaded file to {download_loc}/{filename}\")"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}