Merge branch 'main' into update-grid-styling

.github/workflows/tests.yml

@@ -1,4 +1,4 @@
-name: testing
+name: CI
 
 on: [push, pull_request]
 
@@ -19,7 +19,7 @@ jobs:
     - name: Install other packages needed for api tests
       run: pip install httpx
     - name: Run tests and collect coverage
-      run: cd python && pytest --cov protocaas --cov-report=xml --cov-report=term tests/
+      run: cd python && pytest --cov dendro --cov-report=xml --cov-report=term tests/
     - uses: codecov/codecov-action@v3
       with:
         token: ${{ secrets.CODECOV_TOKEN }}

README.md

@@ -1,20 +1,20 @@
-[![testing](https://github.com/scratchrealm/protocaas/actions/workflows/tests.yml/badge.svg)](https://github.com/scratchrealm/protocaas/actions/workflows/tests.yml)
-[![codecov](https://codecov.io/gh/scratchrealm/protocaas/graph/badge.svg?token=B2DUYR34RZ)](https://codecov.io/gh/scratchrealm/protocaas)
+[![testing](https://github.com/scratchrealm/dendro/actions/workflows/tests.yml/badge.svg)](https://github.com/scratchrealm/dendro/actions/workflows/tests.yml)
+[![codecov](https://codecov.io/gh/scratchrealm/dendro/graph/badge.svg?token=B2DUYR34RZ)](https://codecov.io/gh/scratchrealm/dendro)
 
-# Protocaas (prototype v3) - Neuroscience Analysis Web App
+# Dendro (prototype v3) - Neuroscience Analysis Web App
 
-Protocaas is a **prototype** web application designed for scientists in research labs who want to efficiently manage and conduct neurophysiology data analysis. The current focus is spike sorting of electrophysiology data. Whether you are working with your own data or utilizing public resources from the [DANDI Archive](https://dandiarchive.org/), Protocaas provides a user-friendly platform for organizing, running, and sharing neuroscientific processing jobs.
+Dendro is a **prototype** web application designed for scientists in research labs who want to efficiently manage and conduct neurophysiology data analysis. The current focus is spike sorting of electrophysiology data. Whether you are working with your own data or utilizing public resources from the [DANDI Archive](https://dandiarchive.org/), Dendro provides a user-friendly platform for organizing, running, and sharing neuroscientific processing jobs.
 
 :warning: **Please note:** This software is currently in the prototype phase and is not recommended for production use.
 
 If you're a lab working with electrophysiology data and looking for a solution to spike sort your data, consider trying this software as a beta tester. You can use it in the cloud *or* with your local resources. Reach out to one of the authors if interested.
 
-* [Access the live site](https://protocaas.vercel.app) (it's a prototype!)
+* [Access the live site](https://dendro.vercel.app) (it's a prototype!)
 * [Learn how to host a compute resource](./doc/host_compute_resource.md)
 
 ## System Requirements
 
-To use Protocaas, you only need a web browser. No additional software installation is required. Simply open your preferred web browser and navigate to the [live site](https://protocaas.vercel.app) to get started.
+To use Dendro, you only need a web browser. No additional software installation is required. Simply open your preferred web browser and navigate to the [live site](https://dendro.vercel.app) to get started.
 
 If you want to [host your own compute resource](./doc/host_compute_resource.md) for processing, you will need a Linux machine with optional access to a Slurm cluster or AWS resources.
 
@@ -26,19 +26,19 @@ The first step in uploading your raw electrophysiology data to DANDI is to conve
 
 Once your data is in NWB format, create a new dandiset on [DANDI Archive](https://dandiarchive.org/) and then [follow these instructions to upload your data](https://www.dandiarchive.org/handbook/13_upload). If you are not ready to make your data public, you can choose to create an embargoed (private) dandiset and share it with collaborators. If you are only testing the system, you can use the [staging site](https://gui-staging.dandiarchive.org/).
 
-### Create a Protocaas project
+### Create a Dendro project
 
 Note: If you are working with an embargoed dandiset, then you will need to first provide your DANDI API key (see below).
 
-Each Protocaas project is associated with a dandiset. To create a new project, go to the [protocaas web interface](https://protocaas.vercel.app) and log in using your GitHub account. You can then navigate to the desired dandiset and click "Create a new protocaas project for this dandiset". Provide a project name. You are now ready to import data from the dandiset and start spike sorting.
+Each Dendro project is associated with a dandiset. To create a new project, go to the [dendro web interface](https://dendro.vercel.app) and log in using your GitHub account. You can then navigate to the desired dandiset and click "Create a new dendro project for this dandiset". Provide a project name. You are now ready to import data from the dandiset and start spike sorting.
 
-### Import files from DANDI into a Protocaas project
+### Import files from DANDI into a Dendro project
 
-Once you have created a Protocaas project associated with a dandiset (see above) you can import NWB files (aka DANDI assets) by clicking on the "DANDI import" tab on the left. Use the checkboxes to select the desired files and then click "Import selected assets". The import process can take a minute or two depending on the number of files you are importing. Once the import is complete, you can click on the "Files" tab to see the imported files.
+Once you have created a Dendro project associated with a dandiset (see above) you can import NWB files (aka DANDI assets) by clicking on the "DANDI import" tab on the left. Use the checkboxes to select the desired files and then click "Import selected assets". The import process can take a minute or two depending on the number of files you are importing. Once the import is complete, you can click on the "Files" tab to see the imported files.
 
 ### Run spike sorting
 
-Once you have imported NWB files from DANDI into your Protocaas project, you can begin processing your data. At this point you will need to select a compute resource. By default a very limited default compute resource will be used, which is okay for quick tests. For more serious processing, you will need to [set up your own compute resource](./doc/host_compute_resource.md). You can select the compute resource by clicking the "Settings" button on the "Project home" tab.
+Once you have imported NWB files from DANDI into your Dendro project, you can begin processing your data. At this point you will need to select a compute resource. By default a very limited default compute resource will be used, which is okay for quick tests. For more serious processing, you will need to [set up your own compute resource](./doc/host_compute_resource.md). You can select the compute resource by clicking the "Settings" button on the "Project home" tab.
 
 Once you have selected the compute resource for your project, you can select files for spike sorting using the checkboxes and then click "Run spike sorting". Choose the desired spike sorter, set the desired sorting parameters, and click "Submit". Note that in order to run spike sorting, your NWB files will need to have the appropriate ElectricalSeries data objects in the acquisition section.
 
@@ -58,7 +58,7 @@ To upload the sorting results to DANDI, select the generated files and click "Up
 
 You can obtain your DANDI API key from [https://dandiarchive.org/](https://dandiarchive.org/) or, for the staging site, [https://gui-staging.dandiarchive.org/](https://gui-staging.dandiarchive.org/). Click the button in the upper-right corner.
 
-Then, in the [Protocaas web interface](https://protocaas.vercel.app/), click the key icon in the upper-right and paste in the API key.
+Then, in the [Dendro web interface](https://dendro.vercel.app/), click the key icon in the upper-right and paste in the API key.
 
 ### Hosting a compute resource
 
@@ -70,19 +70,19 @@ More detailed instructions will be forthcoming. For now you can take a look at t
 
 ### Projects, Files and Jobs
 
-Protocaas organizes datasets into projects, files and jobs, streamlining your data management process. Each project is associated a dandiset and a compute resource and comprises files and processing jobs. You can choose to make your projects public or private.
+Dendro organizes datasets into projects, files and jobs, streamlining your data management process. Each project is associated a dandiset and a compute resource and comprises files and processing jobs. You can choose to make your projects public or private.
 
 Project files serve as either pointers to external data sources (e.g., DANDI assets) or as the output of specific processing jobs. These files are typically formatted in NWB (Neurodata Without Borders) format. To get started, you can use the DANDI import tool to seamlessly import data from DANDI repositories. Once imported, you can define processing jobs, such as spike sorting, that take these raw data files as input and generate new project files as output. Project files are immutable.
 
 ### Files and Jobs are Tightly Linked
 
-The full provenance history of each file is stored within a Protocaas project. Each generated file is associated with a unique job that generated it, and each job has links to its input and output files. If a file is deleted, then all jobs that link to that job (either as an input or an output) are also deleted. Similarly, if a job is deleted, then the linked files are also deleted. Thus, in a pipeline, deleting a single file can have a cascading effect in deleting files and jobs throughout the project. In this way, Protocaas files always have a full provenance record.
+The full provenance history of each file is stored within a Dendro project. Each generated file is associated with a unique job that generated it, and each job has links to its input and output files. If a file is deleted, then all jobs that link to that job (either as an input or an output) are also deleted. Similarly, if a job is deleted, then the linked files are also deleted. Thus, in a pipeline, deleting a single file can have a cascading effect in deleting files and jobs throughout the project. In this way, Dendro files always have a full provenance record.
 
 Files and jobs can also be automatically deleted if a new job is queued that would overwrite existing files. Note that existing files (and jobs) are deleted when the new job is *queued* (not *run*).
 
 ### Processing Apps
 
-Protocaas processing tools are organized into plugin apps which are containerized executable programs. At this point, there are only [a few processing apps available](https://github.com/scratchrealm/pc-spike-sorting), including:
+Dendro processing tools are organized into plugin apps which are containerized executable programs. At this point, there are only [a few processing apps available](https://github.com/scratchrealm/pc-spike-sorting), including:
 
 - Spike sorting using Kilosort 2.5
 - Spike sorting using Kilosort 3

api/index.py

@@ -1,6 +1,6 @@
 from fastapi import FastAPI
 
-# Here's the reason that all the other Python files are in ../python/protocaas/api_helpers
+# Here's the reason that all the other Python files are in ../python/dendro/api_helpers
 # I was noticing very long build times (~15 minutes)...
 # Apparently, vercel treats every .py file in /api as a lambda function.
 # So it was building each and every one of them, even though index.py should be the only one.
@@ -12,10 +12,10 @@
 import sys
 print(f'This dir: {thisdir}')
 sys.path.append(thisdir + "/../python")
-from protocaas.api_helpers.routers.processor.router import router as processor_router
-from protocaas.api_helpers.routers.compute_resource.router import router as compute_resource_router
-from protocaas.api_helpers.routers.client.router import router as client_router
-from protocaas.api_helpers.routers.gui.router import router as gui_router
+from dendro.api_helpers.routers.processor.router import router as processor_router
+from dendro.api_helpers.routers.compute_resource.router import router as compute_resource_router
+from dendro.api_helpers.routers.client.router import router as client_router
+from dendro.api_helpers.routers.gui.router import router as gui_router
 
 from fastapi.middleware.cors import CORSMiddleware
 
@@ -26,7 +26,7 @@
 origins = [
     "http://localhost:3000",
     "http://localhost:5173",
-    "https://protocaas.vercel.app"
+    "https://dendro.vercel.app"
 ]
 
 app.add_middleware(

doc/compute_resource_prerequisites.md

@@ -1,6 +1,6 @@
-# Protocaas compute resource prerequisites
+# Dendro compute resource prerequisites
 
-Suppose you have a Linux machine and would like to use it as a protocaas compute resource. Here are the recommended steps to prepare it with the necessary software.
+Suppose you have a Linux machine and would like to use it as a dendro compute resource. Here are the recommended steps to prepare it with the necessary software.
 
 ## Use Linux
 
@@ -30,6 +30,6 @@ or add this command to your ~/.bashrc file to automatically start in this enviro
 
 ## Install docker or singularity
 
-To use your computer as a protocaas compute resource, you'll most likely need to install either docker or singularity. Docker is simpler to install, whereas singularity is better for shared environments or compute clusters.
+To use your computer as a dendro compute resource, you'll most likely need to install either docker or singularity. Docker is simpler to install, whereas singularity is better for shared environments or compute clusters.
 
 To install docker server (or docker engine): https://docs.docker.com/engine/install/

doc/for_developers.md

@@ -1,9 +1,30 @@
-## Running the tests (vscode action)
+# Dendro for developers
+
+Here are some notes for developers. This is not a complete guide, but it should help you get started. Please contact the authors if you have any questions. We are happy to accept questions and pull requests.
+
+* [Developing the Python package](#developing-the-python-package)
+* [Developing the frontend](#developing-the-frontend)
+
+# Developing the Python package
+
+## Running the Python tests (vscode action)
 
 ```bash
+# Install dendro
+cd python
+pip install -e .
+```
+
+```bash
+# Install the dependencies needed for testing
 pip install pytest pytest-asyncio pytest-cov pyright flake8
 ```
 
+```bash
+# Also install the dependencies for the API:
+pip install -r requirements.txt
+```
+
 In vscode:
 
 Command palette => Run Task => Test
@@ -33,3 +54,60 @@ The rules being ignored are in .flake8
 
 You can run `cd python && flake8 --config ../.flake8` to see the errors.
 
+# Developing the frontend
+
+## Setup
+
+Install a modern version of nodejs (e.g. v18).
+
+Install yarn (`npm install -g yarn`).
+
+```bash
+# Install the dependencies
+cd dendro
+yarn install
+```
+
+## Running a development frontend server without hosting a local API
+
+**This is the simplest method if you are not working on the API.**
+
+```bash
+# Run the development server without the API
+yarn dev
+
+# Then open http://localhost:5173/?deployed-api=1
+```
+
+It is important to use the `deployed-api` query parameter, otherwise the frontend will attempt to connect to an API running on localhost:5172.
+
+In order for all functionality to work, you will need to create a .env file in the root of the project with the following contents:
+
+```bash
+VITE_PUBNUB_SUBSCRIBE_KEY=...
+VITE_GITHUB_CLIENT_ID=...
+VITE_DEFAULT_COMPUTE_RESOURCE_ID=...
+```
+
+Contact the authors for the appropriate values of these environment variables.
+
+## Running a development frontend server together with a local API
+
+**Use this method only if you are working on the API.**
+
+For this, you will need to
+* Install the Python requirements as described above.
+* Sign up for a vercel account
+* Install the vercel command-line client
+* Set up the appropriate cloud services
+    - S3 bucket
+    - MongoDB database
+    - PubNub account
+    - GitHub OAuth app
+* Set up the vercel app and configure the appropriate environment variables
+    - OUTPUT_BUCKET_CREDENTIALS, OUTPUT_BUCKET_BASE_URL, OUTPUT_BUCKET_URI, GITHUB_CLIENT_SECRET, VITE_GITHUB_CLIENT_ID, ADMIN_USER_IDS, VITE_PUBNUB_SUBSCRIBE_KEY, PUBNUB_PUBLISH_KEY, MONGO_URI, VITE_DEFAULT_COMPUTE_RESOURCE_ID
+
+```bash
+# Run the development server with the API
+vercel dev
+```

doc/host_compute_resource.md

@@ -1,6 +1,6 @@
-# Hosting a Protocaas compute resource
+# Hosting a Dendro compute resource
 
-Each Protocaas project comes equipped with a dedicated compute resource for executing analysis jobs. The default setting uses a compute resource provided by the author with limitations on CPU, memory, and concurrent jobs, shared among all users. This public resource should only be used for testing with small jobs. Contact one of the authors if you would like to run more intensive processing or configure your own compute resources.
+Each Dendro project comes equipped with a dedicated compute resource for executing analysis jobs. The default setting uses a compute resource provided by the author with limitations on CPU, memory, and concurrent jobs, shared among all users. This public resource should only be used for testing with small jobs. Contact one of the authors if you would like to run more intensive processing or configure your own compute resources.
 
 Prerequisites
 
@@ -11,7 +11,7 @@ Clone this repo, then
 
 ```bash
 # install
-cd protocaas/python
+cd dendro/python
 pip install -e .
 ```
 
@@ -20,14 +20,14 @@ pip install -e .
 export COMPUTE_RESOURCE_DIR=/some/path
 export CONTAINER_METHOD=singularity # or docker
 cd $COMPUTE_RESOURCE_DIR
-protocaas register-compute-resource
+dendro register-compute-resource
 # Open the provided link in a browser and log in using GitHub
 ```
 
 ```bash
 # Start the compute resource
 cd $COMPUTE_RESOURCE_DIR
-protocaas start-compute-resource
+dendro start-compute-resource
 # Leave this open in a terminal. It is recommended that you use a terminal multiplexer like tmux or screen.
 ```
 
@@ -71,7 +71,7 @@ You will need to provide the following fields when configuring the app in the we
 * Job queue: the name of the AWS Batch job queue to use
 * Job definition: the name of the AWS Batch job definition to use
 
-You will also need to provide your AWS credentials in the `.protocaas-compute-resource-node.yaml` in the directory where yur compute resource node daemon is running.
+You will also need to provide your AWS credentials in the `.dendro-compute-resource-node.yaml` in the directory where yur compute resource node daemon is running.
 
 Don't forget to restart your compute resource node after making changes to the web interface.
 

index.html

@@ -2,9 +2,9 @@
 <html lang="en">
   <head>
     <meta charset="UTF-8" />
-    <link rel="icon" type="image/svg+xml" href="/protocaas.png" />
+    <link rel="icon" type="image/svg+xml" href="/dendro.png" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>protocaas</title>
+    <title>dendro</title>
   </head>
   <body style="overflow: hidden; margin: 0; padding: 0">
     <div id="root"></div>

package.json

@@ -1,5 +1,5 @@
 {
-  "name": "protocaas",
+  "name": "dendro",
   "private": true,
   "version": "0.0.0",
   "scripts": {

python/.gitignore

@@ -1,7 +1,7 @@
 script_jobs
 example-data
 tmp
-.protocaas-compute-resource-node.yaml
+.dendro-compute-resource-node.yaml
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

python/README.md

@@ -1 +1 @@
-# protocaas
+# dendro