Lightsheet_cFos_Pipeline

A set of scripts used to analyze data from whole brain lightsheet experiments.

General Organization

• main.ipynb - Notebook which generates figures associated with paper.
• functionScripts directory contains scripts with functions grouped by classify, init, plot, helper. Contains configFunctions which controls many analysis parameters.
• figureScripts directory for scripts used to generate figures for paper. Most now refactored to plot functions, remainder focused on some visualizations, including brainrender.
• Data - Data folder with Lightsheet data, Atlas folder with Allen Brain Atlas related files for organization.

Setup

• Requires Python 3.10 or newer.
• Recommend setting up Conda environment.
• environment.yml file included in base directory.
• Github Specific: file 'allGeneStructureInfo_allgenes_summary_struct.csv' can not be put on Github/CodeOcean because it is to big.
  • Download file from https://alexkwanlab.org/wp-content/uploads/2023/01/davoudian2023.zip
  • Unzip file in any directory.
  • Move file 'allGeneStructureInfo_allgenes_summary_struct.csv' into 'Atlas' folder.

Analysis Setup

• OS: Windows 11, Python 3.10.6
• Hardware: Lenovo Legion 7 Pro Laptop (Model 82N6, SKU 16ACHg6) with AMD Ryzen 7 5800H (8 cores). 32 GB of RAM.
• Packages as defined at the end of README

Installation Guide

• Download main directory 'Lightsheet_cFos_Pipeline' file to a editable directory (as the code makes writes and can be blocked)
• Install Conda - Guide for doing so here Link
• Use the Anaconda terminal to navigate to the downloaded directory
• Setup a Conda enviornment using the included ENV.yml (Look here for instructions on using this file to initialize environment - Link)
• On detailed hardware, should take around 5 to 10 minutes.

Use Guide

• Download files from the 'code' folder in Code Ocean.
• Activate conda enviornment with used name 'conda activate <YOUR_ENV_NAME>' and run 'main.ipynb'.
• Preferred Method - Use VS Code to open main.ipynb and activate generated environment in VS code (Guide here - Link).
• Configuration set by Code block with heading 'Configuration related dictionaries and function calls'.
• in saved code version, 'config.return_classifyDict_testing()' is used to generate classifyDict which will allow for quicker running of test run (mostly by reducing cross-validation count to 10 and using a simple method of feature selection which picks top 30, rather than Boruta procedure).
• On detailed hardware, this version of the code should run in about 3 minutes.
• Once complete run succeeds, switch to config.return_classifyDict_default() to perform published analysis. This can take on the order of 6 to 10 hours.

Running on your own data

• Key considerations will include generating a pandas dataframe with the correct columns and data for each, reference the output of 'initFunctions.loadLightSheetData' for this format.
• functions like 'load_lightsheet_batchCSV' and 'load_lightsheet_excel' rely on specific formats of CSV files and excel files for bringing in line with expected format. Custom functions should be written taking into account your own data and its outputs.
• specific classifications must be defined in 'helperFunctions.create_drugClass_dict()' which creates the dictionary of analyses which can be called, and which 'drug' entities are included and translated to classes. Classes can be single drug.

ENV.yml contents

name: cFosProject channels:

• defaults
• conda-forge
• nvidia dependencies:
• python-dateutil==2.8.2=pyhd3eb1b0_0
• icu==58.2=ha925a31_3
• debugpy==1.5.1=py310hd77b12b_0
• packaging==21.3=pyhd3eb1b0_0
• prompt-toolkit==3.0.20=pyhd3eb1b0_0
• jupyter==1.0.0=py310haa95532_8
• sip==6.6.2=py310hd77b12b_0
• prometheus_client==0.14.1=py310haa95532_0
• ply==3.11=py310haa95532_0
• bleach==4.1.0=pyhd3eb1b0_0
• tk==8.6.12=h2bbff1b_0
• qt-webengine==5.15.9=hb9a9bb5_4
• pyparsing==3.0.9=py310haa95532_0
• testpath==0.6.0=py310haa95532_0
• nbconvert==6.4.4=py310haa95532_0
• entrypoints==0.4=py310haa95532_0
• gstreamer==1.18.5=hd78058f_0
• libxslt==1.1.35=h2bbff1b_0
• pandocfilters==1.5.0=pyhd3eb1b0_0
• patsy==0.5.2=py310haa95532_1
• libvorbis==1.3.7=he774522_0
• scikit-image
• numpy-base==1.23.1=py310h206c741_0
• libwebp==1.2.4=h2bbff1b_0
• vc==14.2=h21ff451_1
• pywinpty==2.0.2=py310h5da7b33_0
• cryptography==37.0.1=py310h21b164f_0
• wincertstore==0.2=py310haa95532_2
• sniffio==1.2.0=py310haa95532_1
• executing==0.8.3=pyhd3eb1b0_0
• h5py
• colorama==0.4.5=py310haa95532_0
• fftw==3.3.9=h2bbff1b_1
• fonttools==4.25.0=pyhd3eb1b0_0
• ipython==8.4.0=py310haa95532_0
• pyzmq==23.2.0=py310hd77b12b_0
• pywin32==302=py310h2bbff1b_2
• numpy==1.23.1=py310h6d2d95c_0
• mistune==0.8.4=py310h2bbff1b_1000
• websocket-client==0.58.0=py310haa95532_4
• asttokens==2.0.5=pyhd3eb1b0_0
• setuptools==63.4.1=py310haa95532_0
• bzip2==1.0.8=he774522_0
• urllib3==1.26.11=py310haa95532_0
• jinja2==3.0.3=pyhd3eb1b0_0
• wcwidth==0.2.5=pyhd3eb1b0_0
• et_xmlfile==1.1.0=py310haa95532_0
• kiwisolver==1.4.2=py310hd77b12b_0
• libtiff==4.4.0=h8a3f274_0
• markupsafe==2.1.1=py310h2bbff1b_0
• brotli-bin==1.0.9=h2bbff1b_7
• libclang==12.0.0=default_h627e005_2
• sqlite==3.39.3=h2bbff1b_0
• libiconv==1.16=h2bbff1b_2
• jupyterlab==3.4.4=py310haa95532_0
• pyqt==5.15.7=py310hd77b12b_0
• glib==2.69.1=h5dc1a3c_1
• anyio==3.5.0=py310haa95532_0
• zlib==1.2.12=h8cc25b3_3
• terminado==0.13.1=py310haa95532_0
• libbrotlidec==1.0.9=h2bbff1b_7
• attrs==21.4.0=pyhd3eb1b0_0
• stack_data==0.2.0=pyhd3eb1b0_0
• ca-certificates
• libbrotlienc==1.0.9=h2bbff1b_7
• requests==2.28.1=py310haa95532_0
• imbalanced-learn
• jupyter_server==1.18.1=py310haa95532_0
• vs2015_runtime==14.27.29016=h5e58377_2
• soupsieve==2.3.1=pyhd3eb1b0_0
• wheel==0.37.1=pyhd3eb1b0_0
• jpeg==9e=h2bbff1b_0
• typing-extensions==4.3.0=py310haa95532_0
• libogg==1.3.5=h2bbff1b_1
• pyopenssl==22.0.0=pyhd3eb1b0_0
• qt-main==5.15.2=he8e5bd7_7
• statsmodels==0.13.2=py310h2bbff1b_0
• xz==5.2.6=h8cc25b3_0
• pcre==8.45=hd77b12b_0
• bottleneck==1.3.5=py310h9128911_0
• certifi
• munkres==1.1.4=py_0
• gst-plugins-base==1.18.5=h9e645db_0
• ipykernel==6.15.2=py310haa95532_0
• argon2-cffi==21.3.0=pyhd3eb1b0_0
• lerc==3.0=hd77b12b_0
• notebook==6.4.12=py310haa95532_0
• jupyter_core==4.11.1=py310haa95532_0
• openssl
• pyrsistent==0.18.0=py310h2bbff1b_0
• parso==0.8.3=pyhd3eb1b0_0
• nbclient==0.5.13=py310haa95532_0
• pysocks==1.7.1=py310haa95532_0
• qtconsole==5.3.2=py310haa95532_0
• toml==0.10.2=pyhd3eb1b0_0
• backcall==0.2.0=pyhd3eb1b0_0
• mkl==2021.4.0=haa95532_640
• pickleshare==0.7.5=pyhd3eb1b0_1003
• decorator==5.1.1=pyhd3eb1b0_0
• libbrotlicommon==1.0.9=h2bbff1b_7
• widgetsnbextension==3.5.2=py310haa95532_0
• jsonschema==4.16.0=py310haa95532_0
• cffi==1.15.1=py310h2bbff1b_0
• winpty==0.4.3=4
• pip==22.2.2=py310haa95532_0
• jupyter_console==6.4.3=pyhd3eb1b0_0
• tornado==6.2=py310h2bbff1b_0
• tzdata==2022c=h04d1e81_0
• pytz==2022.1=py310haa95532_0
• pandas==1.4.4=py310hd77b12b_0
• lz4-c==1.9.3=h2bbff1b_1
• jupyterlab_pygments==0.1.2=py_0
• scipy==1.9.1=py310h86744a3_0
• intel-openmp==2021.4.0=haa95532_3556
• matplotlib-inline==0.1.6=py310haa95532_0
• psutil==5.9.0=py310h2bbff1b_0
• brotlipy==0.7.0=py310h2bbff1b_1002
• nbclassic==0.3.5=pyhd3eb1b0_0
• freetype==2.10.4=hd328e21_0
• qtpy==2.2.0=py310haa95532_0
• qtwebkit==5.212=h3ad3cdb_4
• python-fastjsonschema==2.16.2=py310haa95532_0
• libsodium==1.0.18=h62dcd97_0
• pycparser==2.21=pyhd3eb1b0_0
• python==3.10.6=hbb2ffb3_0
• zstd==1.5.2=h19a0ad4_0
• mkl_fft==1.3.1=py310ha0764ea_0
• win_inet_pton==1.1.0=py310haa95532_0
• jedi==0.18.1=py310haa95532_1
• tqdm==4.64.1=py310haa95532_0
• blas==1.0=mkl
• prompt_toolkit==3.0.20=hd3eb1b0_0
• argon2-cffi-bindings==21.2.0=py310h2bbff1b_0
• ipython_genutils==0.2.0=pyhd3eb1b0_1
• pillow==9.2.0=py310hdc2b20a_1
• brotli==1.0.9=h2bbff1b_7
• six==1.16.0=pyhd3eb1b0_1
• zeromq==4.3.4=hd77b12b_0
• beautifulsoup4==4.11.1=py310haa95532_0
• pure_eval==0.2.2=pyhd3eb1b0_0
• charset-normalizer==2.0.4=pyhd3eb1b0_0
• defusedxml==0.7.1=pyhd3eb1b0_0
• mkl_random==1.2.2=py310h4ed8f06_0
• mkl-service==2.4.0=py310h2bbff1b_0
• cycler==0.11.0=pyhd3eb1b0_0
• jupyterlab_widgets==1.0.0=pyhd3eb1b0_1
• idna==3.3=pyhd3eb1b0_0
• traitlets==5.1.1=pyhd3eb1b0_0
• matplotlib-base==3.5.2=py310hd77b12b_0
• openpyxl==3.0.10=py310h2bbff1b_0
• matplotlib==3.5.2=py310haa95532_0
• webencodings==0.5.1=py310haa95532_1
• jupyter_client==7.3.5=py310haa95532_0
• icc_rt==2022.1.0=h6049295_2
• nest-asyncio==1.5.5=py310haa95532_0
• pygments==2.11.2=pyhd3eb1b0_0
• ipywidgets==7.6.5=pyhd3eb1b0_1
• libwebp-base==1.2.4=h2bbff1b_0
• jupyterlab_server==2.15.2=py310haa95532_0
• libffi==3.4.2=hd77b12b_4
• seaborn==0.11.2=pyhd3eb1b0_0
• json5==0.9.6=pyhd3eb1b0_0
• numexpr==2.8.3=py310hb57aa6b_0
• send2trash==1.8.0=pyhd3eb1b0_1
• babel==2.9.1=pyhd3eb1b0_0
• scikit-learn
• libxml2==2.9.14=h0ad7f3c_0
• pyqt5-sip==12.11.0=py310hd77b12b_0
• nbformat==5.5.0=py310haa95532_0
• libpng==1.6.37=h2a8f88b_0
• libdeflate==1.8=h2bbff1b_5
• shap
• boruta_py prefix: C:\Users\aboha.conda\envs\cFosProject