Rename notebooks folder to templates (#729)

* Remove _ark from the templates folder * Fix reference to clustering notebook in post clustering notebook (should point to templates) Co-authored-by: Noah F. Greenwald <noahfgreenwald@gmail.com>
angelolab · Oct 3, 2022 · 367e133 · 367e133
1 parent 7918e2b
commit 367e133
Show file tree

Hide file tree

Showing 15 changed files with 24 additions and 25 deletions.
diff --git a/README.md b/README.md
@@ -47,33 +47,33 @@ This repo contains tools for analyzing multiplexed imaging data. The assumption
 We have recorded workshop talks which complement the repository. [MIBI Workshop Playlist](https://youtube.com/playlist?list=PLjNbkEm4vA26o5YvWKeyHXF8HjTJc7yB0).
 
 #### 1. Segmentation  
-The [**segmentation notebook**](./templates_ark/1_Segment_Image_Data.ipynb) will walk you through the process of using [Mesmer](https://www.nature.com/articles/s41587-021-01094-0) to segment your image data. This includes selecting the appropriate channel(s) for segmentation, running your data through the network, and then extracting single-cell statistics from the resulting segmentation mask. [Workshop Talk - Session V - Part 1: Segmentation](https://youtu.be/4_AJxrxPYlk?t=231)
+The [**segmentation notebook**](./templates/1_Segment_Image_Data.ipynb) will walk you through the process of using [Mesmer](https://www.nature.com/articles/s41587-021-01094-0) to segment your image data. This includes selecting the appropriate channel(s) for segmentation, running your data through the network, and then extracting single-cell statistics from the resulting segmentation mask. [Workshop Talk - Session V - Part 1: Segmentation](https://youtu.be/4_AJxrxPYlk?t=231)
 
 #### 2. Pixel clustering with Pixie  
-The first step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**pixel clustering notebook**](./templates_ark/2_Cluster_Pixels.ipynb). The notebook walks you through the process of generating pixel clusters for your data, and lets you specify what markers to use for the clustering, train a model, use it to classify your entire dataset, and generate pixel cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session IV - Pixel Level Analysis](https://youtu.be/e7C1NvaPLaY)
+The first step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**pixel clustering notebook**](./templates/2_Cluster_Pixels.ipynb). The notebook walks you through the process of generating pixel clusters for your data, and lets you specify what markers to use for the clustering, train a model, use it to classify your entire dataset, and generate pixel cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session IV - Pixel Level Analysis](https://youtu.be/e7C1NvaPLaY)
 
 #### 3. Cell clustering with Pixie  
-The second step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**cell clustering notebook**](./templates_ark/3_Cluster_Cells.ipynb). This notebook will use the pixel clusters generated in the first notebook to cluster the cells in your dataset. The notebook walks you through generating cell clusters for your data and generates cell cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session V - Cell-level Analysis - Part 2: Cell Clustering](https://youtu.be/4_AJxrxPYlk?t=2704)
+The second step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**cell clustering notebook**](./templates/3_Cluster_Cells.ipynb). This notebook will use the pixel clusters generated in the first notebook to cluster the cells in your dataset. The notebook walks you through generating cell clusters for your data and generates cell cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session V - Cell-level Analysis - Part 2: Cell Clustering](https://youtu.be/4_AJxrxPYlk?t=2704)
 
 #### 4. Post Clustering Tasks
-After the Pixie Pipeline, the user can inspect and fine tune their results with the [**post clustering notebook**](./templates_ark/4_Post_Clustering.ipynb). This notebook will go over cleaning up artifacts left from clustering, and working with functional markers.
+After the Pixie Pipeline, the user can inspect and fine tune their results with the [**post clustering notebook**](./templates/4_Post_Clustering.ipynb). This notebook will go over cleaning up artifacts left from clustering, and working with functional markers.
 
 #### 5. Spatial Analysis  
 [Workshop Talk - Session VI - Spatial Analysis - Part 1: Choosing the Right Analysis Tool](https://youtu.be/HQAkS7kbzH8).
 
   1. **Pairwise Enrichment Analysis**
 
-     The [**pairwise enrichment notebook**](templates_ark/example_pairwise_spatial_enrichment.ipynb) allows the user to investigate the
+     The [**pairwise enrichment notebook**](templates/example_pairwise_spatial_enrichment.ipynb) allows the user to investigate the
      interaction between the phenotypes present in their data. In addition users can
      cluster based on phenotypes around a particular feature such as *artery* or *gland*. [Workshop Talk - Session VI - Spatial Analysis - Part 2: Pairwise Spatial Enrichment](https://youtu.be/HQAkS7kbzH8?t=619).
 
   2. **K-means Neighborhood Analysis**
 
-     The [**neighborhood analysis notebook**](templates_ark/example_neighborhood_analysis_script.ipynb) sheds light on neighborhoods made of micro-environments which consist of a collection of cell phenotypes. [Workshop Talk - Session VI - Spatial Analysis - Part 3: K-means Neighborhood Analysis](https://youtu.be/HQAkS7kbzH8?t=2117).
+     The [**neighborhood analysis notebook**](templates/example_neighborhood_analysis_script.ipynb) sheds light on neighborhoods made of micro-environments which consist of a collection of cell phenotypes. [Workshop Talk - Session VI - Spatial Analysis - Part 3: K-means Neighborhood Analysis](https://youtu.be/HQAkS7kbzH8?t=2117).
 
   1. **Spatial LDA**
 
-     The [**preprocessing**](templates_ark/LDA_Preprocessing.ipynb) and [**training / inference**](templates_ark/LDA_Training_and_Inference.ipynb) draws from language analysis, specifically topic modelling. Spatial LDA overlays a probability distribution on cells belonging to a any particular micro-environment. [Workshop Talk - Session VI - Spatial Analysis - Part 4: Spatial LDA](https://youtu.be/HQAkS7kbzH8?t=3087).
+     The [**preprocessing**](templates/LDA_Preprocessing.ipynb) and [**training / inference**](templates/LDA_Training_and_Inference.ipynb) draws from language analysis, specifically topic modelling. Spatial LDA overlays a probability distribution on cells belonging to a any particular micro-environment. [Workshop Talk - Session VI - Spatial Analysis - Part 4: Spatial LDA](https://youtu.be/HQAkS7kbzH8?t=3087).
 
 
 ### Installation Steps
@@ -127,7 +127,7 @@ You can shut down the notebooks and close docker by entering `control-c` in the
 
 **REMEMBER TO DUPLICATE AND RENAME NOTEBOOKS**
 
-If you didn't change the name of any of the notebooks within the `templates_ark` folder, they will be overwritten when you decide to update the repo. Read about updating Ark [here](#updating-the-repository)
+If you didn't change the name of any of the notebooks within the `templates` folder, they will be overwritten when you decide to update the repo. Read about updating Ark [here](#updating-the-repository)
 
 ## External Tools
 

diff --git a/ark/utils/io_utils_test.py b/ark/utils/io_utils_test.py
@@ -9,8 +9,7 @@
 
 def test_validate_paths():
     # change cwd to /scripts for more accurate testing
-    # we'll ignore templates_qc because that is only run on the commercial MIBI computer
-    os.chdir('templates_ark')
+    os.chdir('templates')
 
     # make a tempdir for testing
     with tempfile.TemporaryDirectory(dir='../data') as valid_path:

diff --git a/ark/utils/notebook_update_test.py b/ark/utils/notebook_update_test.py
@@ -23,7 +23,7 @@ def _exec_update_notebooks(base_path, update_flag=True, bad_flag=False):
     if bad_flag:
         args.append("-g")
 
-    # attempt to copy files from base_path/templates_ark to scripts
+    # attempt to copy files from base_path/templates to scripts
     try:
         # we have to append /private ahead of the base_path due to the
         # way the temp_dir gets configured
@@ -48,9 +48,9 @@ def _exec_update_notebooks(base_path, update_flag=True, bad_flag=False):
 
 
 def _make_dir_and_exec(base_dir, templates, scripts=None, update_flag=True, bad_flag=False):
-    os.mkdir(os.path.join(base_dir, "templates_ark"))
+    os.mkdir(os.path.join(base_dir, "templates"))
     for template in templates:
-        pathlib.Path(os.path.join(base_dir, "templates_ark", template[0])).write_text(template[1])
+        pathlib.Path(os.path.join(base_dir, "templates", template[0])).write_text(template[1])
 
     if scripts is not None:
         os.mkdir(os.path.join(base_dir, "scripts"))

diff --git a/ark/utils/notebooks_test.py b/ark/utils/notebooks_test.py
@@ -12,11 +12,11 @@
 
 
 SEGMENT_IMAGE_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)),
-                                       '..', '..', 'templates_ark', '1_Segment_Image_Data.ipynb')
+                                       '..', '..', 'templates', '1_Segment_Image_Data.ipynb')
 PIXEL_CLUSTER_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)),
-                                  '..', '..', 'templates_ark', '2_Cluster_Pixels.ipynb')
+                                  '..', '..', 'templates', '2_Cluster_Pixels.ipynb')
 CELL_CLUSTER_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)),
-                                 '..', '..', 'templates_ark', '3_Cluster_Cells.ipynb')
+                                 '..', '..', 'templates', '3_Cluster_Cells.ipynb')
 
 
 def _exec_notebook(nb_filename, base_folder):
@@ -31,15 +31,15 @@ def _exec_notebook(nb_filename, base_folder):
 
 # test runs with default inputs
 def test_segment_image_data():
-    _exec_notebook('1_Segment_Image_Data.ipynb', 'templates_ark')
+    _exec_notebook('1_Segment_Image_Data.ipynb', 'templates')
 
 
 def test_example_pairwise_spatial_enrichment():
-    _exec_notebook('example_pairwise_spatial_enrichment.ipynb', 'templates_ark')
+    _exec_notebook('example_pairwise_spatial_enrichment.ipynb', 'templates')
 
 
 def test_example_neighborhood_analysis():
-    _exec_notebook('example_neighborhood_analysis_script.ipynb', 'templates_ark')
+    _exec_notebook('example_neighborhood_analysis_script.ipynb', 'templates')
 
 
 # test folder inputs for image segmentation

diff --git a/start_docker.sh b/start_docker.sh
@@ -8,7 +8,7 @@ while test $# -gt 0
 do
   case "$1" in
     -n|--develop-notebook-templates)
-      JUPYTER_DIR='templates_ark'
+      JUPYTER_DIR='templates'
       shift
       ;;
     -u|--update)

diff --git a/templates_ark/1_Segment_Image_Data.ipynb → templates/1_Segment_Image_Data.ipynb b/templates_ark/1_Segment_Image_Data.ipynb → templates/1_Segment_Image_Data.ipynb
diff --git a/templates_ark/2_Cluster_Pixels.ipynb → templates/2_Cluster_Pixels.ipynb b/templates_ark/2_Cluster_Pixels.ipynb → templates/2_Cluster_Pixels.ipynb
diff --git a/templates_ark/3_Cluster_Cells.ipynb → templates/3_Cluster_Cells.ipynb b/templates_ark/3_Cluster_Cells.ipynb → templates/3_Cluster_Cells.ipynb
diff --git a/templates_ark/4_Post_Clustering.ipynb → templates/4_Post_Clustering.ipynb b/templates_ark/4_Post_Clustering.ipynb → templates/4_Post_Clustering.ipynb
@@ -6,7 +6,7 @@
    "metadata": {},
    "source": [
     "# Post-clustering tasks\n",
-    "This notebook allows the user to inspect and fine-tune the output of the [clustering](https://github.com/angelolab/ark-analysis/blob/main/templates_ark/3_Cluster_Cells.ipynb) notebook. There are two parts of this notebook. \n",
+    "This notebook allows the user to inspect and fine-tune the output of the [clustering](https://github.com/angelolab/ark-analysis/blob/main/templates/3_Cluster_Cells.ipynb) notebook. There are two parts of this notebook. \n",
     "1. Clustering cleanup: If there are clusters that were not properly separated during Pixie cell clustering, this provides the option of defining manual thresholds based on marker intensity to combine or separate specific clusters\n",
     "2. Marker thresholding: For markers that were not directly used for clustering, but whose expression is important for phenotyping individual cells, this provides the user with visualization to determine accurate thresholds for positive/negative classification"
    ]
@@ -400,7 +400,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.13"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,

diff --git a/templates_ark/LDA_Preprocessing.ipynb → templates/LDA_Preprocessing.ipynb b/templates_ark/LDA_Preprocessing.ipynb → templates/LDA_Preprocessing.ipynb
diff --git a/...ates_ark/LDA_Training_and_Inference.ipynb → templates/LDA_Training_and_Inference.ipynb b/...ates_ark/LDA_Training_and_Inference.ipynb → templates/LDA_Training_and_Inference.ipynb
diff --git a/...xample_neighborhood_analysis_script.ipynb → ...xample_neighborhood_analysis_script.ipynb b/...xample_neighborhood_analysis_script.ipynb → ...xample_neighborhood_analysis_script.ipynb
diff --git a/...example_pairwise_spatial_enrichment.ipynb → ...example_pairwise_spatial_enrichment.ipynb b/...example_pairwise_spatial_enrichment.ipynb → ...example_pairwise_spatial_enrichment.ipynb
diff --git a/templates_ark/example_visualization.ipynb → templates/example_visualization.ipynb b/templates_ark/example_visualization.ipynb → templates/example_visualization.ipynb
diff --git a/update_notebooks.sh b/update_notebooks.sh
@@ -24,7 +24,7 @@ then
   if [ $update -ne 0 ]
   then
     # check for each template's existance
-    for f in "$PWD"/templates_ark/*.ipynb
+    for f in "$PWD"/templates/*.ipynb
     do
       # get basename of notebook
       name=$(basename "$f")
@@ -55,10 +55,10 @@ then
       fi
     done
   else
-    cp -n "$PWD/templates_ark"/*\.ipynb "$PWD/scripts/."
+    cp -n "$PWD/templates"/*\.ipynb "$PWD/scripts/."
   fi
 else
   # since there is no scripts directory, just make one and copy from templates
   mkdir "$PWD/scripts"
-  cp "$PWD"/templates_ark/*\.ipynb "$PWD/scripts/."
+  cp "$PWD"/templates/*\.ipynb "$PWD/scripts/."
 fi