Merge pull request #171 from project-improv/docs

initial draft of docs

.github/workflows/pypi.yaml

@@ -5,6 +5,8 @@ on:
   push:
     tags: 
       - 'v[0-9]+.[0-9]+.[0-9]+'
+    branches:
+      - 'main'
 
 jobs:
   build:

.github/workflows/test-pypi.yaml

@@ -2,7 +2,11 @@
 # based on https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
 name: Publish Python 🐍 distribution 📦 to TestPyPI
 
-on: push
+on: 
+  push:
+    branches:
+      - 'main'
+      - 'dev'
 
 jobs:
   build:

README.md

@@ -1,30 +1,39 @@
 # improv
+[![PyPI](https://img.shields.io/pypi/v/improv?style=flat-square?style=flat-square)](https://pypi.org/project/improv)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/improv?style=flat-square)](https://pypi.org/project/improv)
+[![docs](https://github.com/project-improv/improv/actions/workflows/docs.yaml/badge.svg?style=flat-square)](https://project-improv.github.io/)
+[![tests](https://github.com/project-improv/improv/actions/workflows/CI.yaml/badge.svg?style=flat-square)](https://project-improv.github.io/)
+[![Coverage Status](https://coveralls.io/repos/github/project-improv/improv/badge.svg?branch=main)](https://coveralls.io/github/project-improv/improv?branch=main)
+[![PyPI - License](https://img.shields.io/pypi/l/improv?style=flat-square)](https://opensource.org/licenses/MIT)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square)](https://github.com/psf/black)
+
 A flexible software platform for real-time and adaptive neuroscience experiments.
 
-improv is a streaming software platform designed to enable adaptive experiments. By analyzing data, such as 2-photon calcium images, as it comes in, we can obtain information about the current brain state in real time and use it to adaptively modify an experiment as data collection is ongoing. 
+_improv_ is a streaming software platform designed to enable adaptive experiments. By analyzing data, such as 2-photon calcium images, as it comes in, we can obtain information about the current brain state in real time and use it to adaptively modify an experiment as data collection is ongoing. 
 
-![](https://dibs-web01.vm.duke.edu/pearson/assets/videos/zebrafish/improvGif.gif)
+![](https://dibs-web01.vm.duke.edu/pearson/assets/improv/improvGif.gif)
 
 This video shows raw 2-photon calcium imaging data in zebrafish, with cells detected in real time by [CaImAn](https://github.com/flatironinstitute/CaImAn), and directional tuning curves (shown as colored neurons) and functional connectivity (lines) estimated online, during a live experiment. Here only a few minutes of data have been acquired, and neurons are colored by their strongest response to visual simuli shown so far.
 We also provide up-to-the-moment estimates of the functional connectivity by fitting linear-nonlinear-Poisson models online, as each new piece of data is acquired. Simple visualizations offer real-time insights, allowing for adaptive experiments that change in response to the current state of the brain.
 
 
 ### How improv works
 
-<img src="figures/improv_design.png" width=85%>
+<img src="https://dibs-web01.vm.duke.edu/pearson/assets/improv/improv_design.png" width=85%>
+
 improv allows users to flexibly specify and manage adaptive experiments to integrate data collection, preprocessing, visualization, and user-defined analytics. All kinds of behavioral, neural, or modeling data can be incorporated, and input and output data streams are managed independently and asynchronously. With this design, streaming analyses and real-time interventions can be easily integrated into various experimental setups. improv manages the backend engineering of data flow and task execution for all steps in an experimental pipeline in real time, without requiring user oversight. Users need only define their particular processing pipeline with simple text files and are free to define their own streaming analyses via Python classes, allowing for rapid prototyping of adaptive experiments.  
   <br />
   <br />
 
-<img src="https://dibs-web01.vm.duke.edu/pearson/assets/images/zebrafish/actor_model.png" width=60%>
+<img src="https://dibs-web01.vm.duke.edu/pearson/assets/improv/actor_model.png" width=60%>
 
-improv's design is based on a steamlined version of the actor model for concurrent computation. Each component of the system (experimental pipeline) is considered an 'actor' and has a unique role. They interact via message passing, without the need for a central broker. Actors are implemented as user-defined classes that inherit from improv's Actor class, which supplies all queues for message passing and orchestrates process execution and error handling. Messages between actors are composed of keys that correspond to items in a shared, in-memory data store. This both minimizes communication overhead and data copying between processes. 
+_improv_'s design is based on a streamlined version of the actor model for concurrent computation. Each component of the system (experimental pipeline) is considered an 'actor' and has a unique role. They interact via message passing, without the need for a central broker. Actors are implemented as user-defined classes that inherit from _improv_'s `Actor` class, which supplies all queues for message passing and orchestrates process execution and error handling. Messages between actors are composed of keys that correspond to items in a shared, in-memory data store. This both minimizes communication overhead and data copying between processes. 
 
 
 
 ## Installation
 
-For installation instructions, please consult the [wiki](https://github.com/project-improv/improv/wiki/Installation) on our github.
+For installation instructions, please consult the [docs](https://project-improv.github.io/improv/installation.html) on our github.
 
 ### Contact
-To get in touch, feel free to reach out on Twitter <a href="http://twitter.com/annedraelos" target="_blank">@annedraelos</a>. 
+To get in touch, feel free to reach out on Twitter <a href="http://twitter.com/annedraelos" target="_blank">@annedraelos</a> or <a href="http://twitter.com/jmxpearson" target="_blank">@jmxpearson</a>. 

demos/bubblewrap/README.md

@@ -1,6 +1,6 @@
 # Running Bubblewrap demo
 ## Ubuntu users
-After installing improv, install additional dependencies (JAX on CPU by default) navigating to `demos/bubblewrap` and doing
+After installing _improv_, install additional dependencies (JAX on CPU by default) navigating to `demos/bubblewrap` and doing
 
 - `pip install -r requirements.txt`
 

docs/_config.yml

@@ -7,7 +7,7 @@
 # Book settings
 title                       : improv documentation  # The title of the book. Will be placed in the left navbar.
 author                      : improv team  # The author of the book
-copyright                   : "2022"  # Copyright year to be placed in the footer
+copyright                   : "2024"  # Copyright year to be placed in the footer
 logo                        : logo.png  # A path to the book logo
 
 # Force re-execution of notebooks on each build.
@@ -37,6 +37,9 @@ html:
   use_repository_button: true
 
 sphinx:
+  config:
+    bibtex_reference_style: author_year
+
   extra_extensions:
   - 'autoapi.extension'
   - 'sphinx.ext.napoleon'

docs/_static/custom.css

@@ -0,0 +1,3 @@
+figcaption p {
+    text-align: left
+}

docs/_toc.yml

@@ -4,8 +4,11 @@
 format: jb-book
 root: intro
 chapters:
-- file: markdown
-- file: notebooks
-- file: markdown-notebooks
-- file: content
+- file: installation
+- file: running
+- file: demos
+- file: design
+- file: actors
+- file: signals
+- file: bibliography
 - file: autoapi/index

docs/actors.md

@@ -0,0 +1,24 @@
+(page:actors)=
+# Writing actors
+
+## Overview
+Nearly any experiment implemented in _improv_ will require the creation of custom actors. As explained in [](page:design:pipeline_spec), these actors are defined by Python classes and represent distinct, independent steps in the processing pipeline. The best way to learn to write actors is by example: the [demos](https://github.com/project-improv/improv/tree/main/demos) give several complete, runnable pipelines.
+
+Here, we explain some of the theory behind how actors work, as well as some more advanced functionality.
+
+## Basic expectations: the `AbstractActor` class
+
+The `AbstractActor` class defines the template from which all actors must inherit. You will never directly use an `AbstractActor`, but all other actors are based on it. You can look up the documentation in the [](autoapi/index), but most of the methods associated with the class are internal, handling communication with the store and server, and you won't need to deal directly with them. The three methods work knowing about are:
+- `setup`: handles all the setup work the class needs to do _before_ the experiment starts
+- `run`: the data processing step executed repeatedly during the experiment; handles getting input data from the store, putting output data into the store, and informing other actors where the results are
+- `stop`: handles all cleanup necessary when the class is stopped
+
+However, the basic `AbstractActor` lacks many features we might want: it doesn't properly handle signals from the server (including `stop`!) and so requires writing a lot of tedious code to check for other things that might be happening in the system. As a result, _improv_ provides the `ManagedActor` class, to which we turn next.
+
+## Practical actor implementations with `ManagedActor`
+The key benefit the `ManagedActor` class (which is aliased to `Actor`) offers over the more general `AbstractActor` is the addition of the `RunManager` context manager. This context manager is used within `run` and handles communication with the server, including calling the `setup` and `stop` methods when the actor receives those signals. In a `ManagedActor`, the actual processing logic is located in the `runStep` function, which must be defined for any valid actor subclass.[^async_note] Again, examples of actors subclassing `Actor` (aka, `ManagedActor`) are available in the `actors` subfolder of each demo in [demos](https://github.com/project-improv/improv/tree/main/demos).
+
+Internally, actors communicate with each other and with the server via [multiprocessing queues](https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues), which are highly performant but restricted to processes on the same machine. For actors located on other machines, or across networks, there are [other actors](https://github.com/project-improv/improv/blob/main/demos/sample_actors/zmqActor.py) that communicate using [ZMQ](https://zeromq.org)[^zmq_note]. In any event, the details should be transparent to users, and implementations are subject to change without notice, so users should not depend on these internals.
+
+[^async_note]: In addition, there are asynchronous versions of the `ManagedActor` and `RunManager`, and these may become the defaults aliased to `Actor` in future versions, so users should not rely on details of these implementations.
+[^zmq_note]: And this option may become the default in future versions.

docs/bibliography.md

@@ -0,0 +1,3 @@
+# Bibliography
+```{bibliography}
+```

docs/demos.ipynb

@@ -0,0 +1,133 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "(page:demos)=\n",
+    "# Running the demos\n",
+    "To run the demos, you'll first need a [source code installation](page:installation:source_build) of _improv_. Demos are located in separate subfolders of the `demos` folder. For instance, the `minimal` subfolder contains the files "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "tags": [
+     "remove-input"
+    ],
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[34mactors\u001b[m\u001b[m\n",
+      "minimal.yaml\n",
+      "minimal_spawn.yaml\n",
+      "\n",
+      "../demos/minimal/actors:\n",
+      "sample_generator.py\n",
+      "sample_processor.py\n",
+      "sample_spawn_processor.py\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "!ls -R  ../demos/minimal | grep -v '.pyc'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `minimal.yaml` and `minimal_spawn.yaml` files each define an _improv_ pipeline, differing only in the method they use to launch subprocesses [^fork_vs_spawn]. In the simpler case, `minimal.yaml` reads\n",
+    "\n",
+    "[^fork_vs_spawn]: This is a technical distinction that may be important on some systems. A concise, helpful explainer is available in [this StackOverflow answer](https://stackoverflow.com/questions/64095876/multiprocessing-fork-vs-spawn)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "remove-input"
+    ],
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!cat ../demos/minimal/minimal.yaml"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The file requests two actors, `Generator` and `Processor`. The file tells _improv_ that `Generator` is defined in the `Generator` class of `actors.sample_generator`, and similarly `Processor` is defined in the `Processor` class inside the `actors.sample_processor` module. See [](page:design:pipeline_spec) and [](page:actors) for details.\n",
+    "\n",
+    "In addition, there is a single connection between the two actors: `Generator.q_out` (the output of `Generator`) should be connected to the input of `Processor` (`Processor.q_in`).\n",
+    "\n",
+    "```{note}\n",
+    "In the example above (and all the demos), the relevant actors are found within the `actors` subfolder of the directory containing the YAML file defining the pipeline. More generally, actors can be located anywhere, and additional diretories can be specified via the `--actor-path` command line argument to `improv run`. See [here](page:running:options) for more details.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The minimal demo can easily be run from the command line as detailed in [](page:running):\n",
+    "```bash\n",
+    "improv run demos/minimal/minimal.yaml\n",
+    "```\n",
+    "<img src=\"https://dibs-web01.vm.duke.edu/pearson/assets/improv/improv_quick_start.gif\">\n",
+    "<br>\n",
+    "<br>\n",
+    " "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Details for running each actor can be found in the `README` file within each demo folder. \n",
+    "\n",
+    "````{warning}\n",
+    "Many demos have additional dependencies that are not part of the typical _improv_ installation. In these cases, additional packages may be installed by running\n",
+    "```bash\n",
+    "pip install -r requirements.txt\n",
+    "```\n",
+    "within the demo folder. Again, see the individual README files within each demo folder for details.\n",
+    "````"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "improv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}