1. obtain_neuron_skeletons.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0bff52f9",
   "metadata": {},
   "source": [
    "# Obtaining neuron datasets from the Janelia fruiftly brain project\n",
    "Below is the code used to obtain neuron skeletons, but also further processing to obtain only the endpoints of the skeleton segments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49a2c1c2",
   "metadata": {},
   "source": [
    "Please register and obtain your token at https://neuprint.janelia.org/help/api in order to access the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "11c9d163",
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuprint import Client\n",
    "\n",
    "TOKEN = \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6Imx1a2FibGFnb2pldmljMTk5NUBnbWFpbC5jb20iLCJsZXZlbCI6Im5vYXV0aCIsImltYWdlLXVybCI6Imh0dHBzOi8vbGgzLmdvb2dsZXVzZXJjb250ZW50LmNvbS9hLS9BT2gxNEdqdDZpdFFGR2xTSWZUTElNTjRmcEt1QzZ3QmE2Rlp0WU1XYmpKV1ZBPXM5Ni1jP3N6PTUwP3N6PTUwIiwiZXhwIjoxODA5MTE1OTEyfQ.0h6CJp8xfQEpkW8a2_gqJUBrEA5GyBiZkNvDjRpoXoY\" # <--- Paste your token here\n",
    "\n",
    "c = Client('neuprint.janelia.org', 'hemibrain:v1.2.1', TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3bd8d91",
   "metadata": {},
   "source": [
    "The function below  gives a dictionary with neural regions and their corresponding subregions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aa4577eb-9e5b-49a1-8448-edb36fdadd04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['AL(L)*', 'AL(R)*', 'AOT(R)', 'CX', 'GC', 'GF(R)', 'GNG*', 'INP', 'LH(R)*', 'LX(L)', 'LX(R)', 'MB(+ACA)(R)', 'MB(L)', 'OL(R)', 'PENP', 'POC', 'SNP(L)', 'SNP(R)', 'VLNP(R)', 'VMNP', 'mALT(L)', 'mALT(R)'])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neuprint import fetch_roi_hierarchy\n",
    "roi_dict = fetch_roi_hierarchy(False, True, 'dict')\n",
    "roi_dict['hemibrain'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "693ce1c3",
   "metadata": {},
   "source": [
    "I chose only one ('ME(R)') region to analyze, out of whole dataset and I set the criteria to 1 in-going and out-going synapse for neurons to be selected. This is because neurons may pass through and connect mutiple different regions and may share varying number of connections in each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8e2ca281",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of neurons in selected datasets:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ME(R)': 3721}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neuprint import fetch_adjacencies, NeuronCriteria as NC\n",
    "from neuprint import fetch_neurons\n",
    "import pandas as pd\n",
    "roi_list = ['ME(R)']\n",
    "dataset_lengths = {}\n",
    "neuron_df_dict = {}\n",
    "for region in roi_list:\n",
    "    if  region[-1:] == '*':\n",
    "        region = region[:-1]\n",
    "    # The criteria is set to one input synapse and oneoutput synapse, both within the selected region\n",
    "    criteria = NC(min_roi_inputs=1,min_roi_outputs=1, inputRois=[region], outputRois=[region])\n",
    "    # Fetching the neuron and region information based on the criteria\n",
    "    neuron_df, roi_counts_df = fetch_neurons(criteria)\n",
    "    # Saving and storing the information\n",
    "    dataset_lengths[region] = len(neuron_df)\n",
    "    neuron_df.to_csv('neuron_regions_information/' + region+ '_region.csv')\n",
    "    neuron_df_dict[region] = neuron_df\n",
    "\n",
    "print('Number of neurons in selected datasets:')\n",
    "dataset_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f43fc6",
   "metadata": {},
   "source": [
    "Based on the acquired neuron information, I downloaded the exact skeleton (morphological) information, which is stored a series of connected linear segments. I only used the starting and ending point of each segment for the purpose of this research:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bb8d4d7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There was total of 3721 neurons. Out of them 742 were missing and 2979 were downloaded\n"
     ]
    }
   ],
   "source": [
    "from neuprint import fetch_skeleton\n",
    "# Columns I will filter by, coordinates, radius information and neuron identity (to which neuron does a point belong to)\n",
    "columns = ['x','y','z','radius','bodyId']\n",
    "# Storing missing/present neuron ids\n",
    "missing_ids = []\n",
    "present_ids = []\n",
    "\n",
    "skeleton_df_dict = {}\n",
    "skeleton_df = pd.DataFrame(columns=columns)\n",
    "for region in roi_list:\n",
    "    for i,bodyId in enumerate(neuron_df_dict[region]['bodyId']):\n",
    "        if i % 500 == 0:\n",
    "            print('Completed',i,'out of',dataset_lengths[region],'neurons for the region',region)\n",
    "        # Try and except are used if there is an error in accesing the database\n",
    "        try:\n",
    "            # Fetching the skeleton based on the neuron bodyId\n",
    "            s =  c.fetch_skeleton(body = int(bodyId),format='pandas')\n",
    "            s['bodyId'] = str(bodyId)\n",
    "            present_ids.append(bodyId)\n",
    "            skeleton_df = pd.concat([skeleton_df,s])\n",
    "        except:\n",
    "            missing_ids.append(bodyId)\n",
    "    skeleton_df = skeleton_df[columns].copy()\n",
    "    skeleton_df.to_csv('neuron_regions_points/' + region+'_points.csv')\n",
    "    skeleton_df_dict[region] = skeleton_df\n",
    "print(\"Data succesfully obtained\")                          \n",
    "print('There was total of', dataset_lengths[region],'neurons.','Out of them',len(missing_ids),'were missing and',len(present_ids),'were downloaded')       "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e953f52",
   "metadata": {},
   "source": [
    "Each region will have a set of points, with corresponding radii and neuron identites (bodyId):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3a414b8d-cc11-4874-9e82-4d45a573ba34",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>radius</th>\n",
       "      <th>bodyId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9192.0</td>\n",
       "      <td>17552.0</td>\n",
       "      <td>9100.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>543702186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9192.0</td>\n",
       "      <td>17576.0</td>\n",
       "      <td>9100.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>543702186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9216.0</td>\n",
       "      <td>17600.0</td>\n",
       "      <td>9100.0</td>\n",
       "      <td>21.941099</td>\n",
       "      <td>543702186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9264.0</td>\n",
       "      <td>17648.0</td>\n",
       "      <td>9100.0</td>\n",
       "      <td>21.941099</td>\n",
       "      <td>543702186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9336.0</td>\n",
       "      <td>17696.0</td>\n",
       "      <td>9100.0</td>\n",
       "      <td>63.894699</td>\n",
       "      <td>543702186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15257</th>\n",
       "      <td>13954.0</td>\n",
       "      <td>11840.0</td>\n",
       "      <td>30690.0</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>7112616299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15258</th>\n",
       "      <td>13954.0</td>\n",
       "      <td>11818.0</td>\n",
       "      <td>30690.0</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>7112616299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15259</th>\n",
       "      <td>13932.0</td>\n",
       "      <td>11796.0</td>\n",
       "      <td>30712.0</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>7112616299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15260</th>\n",
       "      <td>13932.0</td>\n",
       "      <td>11774.0</td>\n",
       "      <td>30712.0</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>7112616299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15261</th>\n",
       "      <td>13932.0</td>\n",
       "      <td>11752.0</td>\n",
       "      <td>30734.0</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>7112616299</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1464000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             x        y        z     radius      bodyId\n",
       "0       9192.0  17552.0   9100.0  12.000000   543702186\n",
       "1       9192.0  17576.0   9100.0  12.000000   543702186\n",
       "2       9216.0  17600.0   9100.0  21.941099   543702186\n",
       "3       9264.0  17648.0   9100.0  21.941099   543702186\n",
       "4       9336.0  17696.0   9100.0  63.894699   543702186\n",
       "...        ...      ...      ...        ...         ...\n",
       "15257  13954.0  11840.0  30690.0  11.000000  7112616299\n",
       "15258  13954.0  11818.0  30690.0  11.000000  7112616299\n",
       "15259  13932.0  11796.0  30712.0  11.000000  7112616299\n",
       "15260  13932.0  11774.0  30712.0  11.000000  7112616299\n",
       "15261  13932.0  11752.0  30734.0  11.000000  7112616299\n",
       "\n",
       "[1464000 rows x 5 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skeleton_df_dict['ME(R)']"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}