Skip to content

Commit

Permalink
Merge pull request #445 from oist/update-installation-docs
Browse files Browse the repository at this point in the history 
Update installation docs
  • Loading branch information
@emuemuJP
emuemuJP committed Mar 30, 2023
2 parents 45a2701 + b1fd2c4 commit c36adda
Show file tree
Hide file tree
Showing 2 changed files with 161 additions and 128 deletions.
25 changes: 15 additions & 10 deletions docs/sphinx-usage.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,18 @@ pip install -r requirements.txt

## Usage

- output html
- output html to _build (use cache)
```
make html
```
- output html to _build (all rebuild)
```
# "-E" is an option for all-rebuilding.
$env:SPHINXOPTS="-E"; make html; $env:SPHINXOPTS=""
```
- output html to _build (use cache)
```
make html
```

- output html to _build (all rebuilding)
- *"-E" is an option for all-rebuilding.*
- linux, mac
```
export SPHINXOPTS="-E"; make html; export SPHINXOPTS=""
```
- windows
```
$env:SPHINXOPTS="-E"; make html; $env:SPHINXOPTS=""
```
264 changes: 146 additions & 118 deletions docs/utils/add_algorithm.md
View file
Original file line number Diff line number Diff line change
@@ -1,127 +1,141 @@
Procedure for adding a new algorithm
=================
# Procedure for adding a new algorithm

How to add algorithms
```{contents}
:depth: 3
```

* [1. Create algorithm function file](#1-create-algorithm-function-file)
* [2. Register your algorithm](#2-register-your-algorithm)
* [3. Describe function processing](#3-describe-function-processing)
* [3.1 import](#31-import)
* [3.2 Input & Output handle](#32-input--output-handle)
* [3.3 Drawing output results](#33-drawing-output-results)
* [4. How to add parameters](#4-how-to-add-parameters)
## Example of Algorithm Addition Procedure

## 1. Create algorithm function file
First, create a python file at the appropriate location in the following directory.
Here is an example of how to create a function named `new_algorithm`. The file should be created at the following location [optinist/wrappers/optinist_wrapper](https://github.com/oist/optinist/tree/main/optinist/wrappers/optinist_wrapper).
Below we describe an example procedure for adding a new algorithm.

Create it under the name **new_algorith.py**.
**Prerequisite**
- Sample Algorithm Name ... `new_algo`
- Sample Algorithm Function Name ... `new_algo_func`
- {OPTINIST_SRC_DIR} ... Replace with the actual source storage directory path.

```python:new_algorithm.py
def new_algorithm():
return
```

[algorithm list](https://github.com/oist/optinist/tree/develop/optinist/wrappers)

- \_\_init__.py - ①
- caiman_wrapper
- \_\_init__.py - ②
- caiman_mc.py
- caiman_cnmf.py
- suite2p_wrapper
- \_\_init__.py
- suite2p_file_convert.py
- suite2p_registration.py
- suite2p_roi.py
- suite2p_cnmf.py
- optinist_wrapper
- \_\_init__.py
- basic neural analysis
- dimension reduction
- neural population analysis
- neural decoding
- `new_algorithm.py` - ＊

<br />

## 2. Register your algorithm
①: To be able to use the created **new_algorithm** on the GUI, it is necessary to register it in the \_\_init__.py. For example, [optinist/wrappers/\_\_init__.py](https://github.com/oist/optinist/blob/main/optinist/wrappers/__init__.py) in ① is as follows.
These are reading one level down, \_\_init__.py.
### 1. Prepare necessary directories and files for the algorithm

First, prepare the necessary directories and files for the algorithm.

```python
from .caiman_wrapper import caiman_wrapper_dict
from .suite2p_wrapper import suite2p_wrapper_dict
from .optinist_wrapper import optinist_wrapper_dict
- {OPTINIST_SRC_DIR}/optinist/wrappers/
- \_\_init__.py
- xxxx_wrapper/
- yyyy_wrapper/
- ...
- `new_algo_wrapper/` (*1)
- \_\_init__.py (*2)
- `new_algo_func.py` (*3)
- ... (*4)

wrapper_dict = {}
wrapper_dict.update(**caiman_wrapper_dict)
wrapper_dict.update(**suite2p_wrapper_dict)
wrapper_dict.update(**optinist_wrapper_dict)
- Explanation:
- (*1) The name is arbitrary, but `{algorithm name}_wrapper` is the standard.
- (*2) Empty first.
- (*3) Empty first.
- (*4) Prepare other files to be added.

### 2. Algorithm implementation

#### Import statement description

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/`new_algo_func`.py

```python
from optinist.api.dataclass.dataclass import *
```

- Explanation:
- If the required dataclass does not exist, you can add your own.
- see. [DataClass](#dataclass)


②: In [optinist/wrappers/caiman_wrapper/\_\_init__py](https://github.com/oist/optinist/blob/main/optinist/wrappers/caiman_wrapper/__init__.py) function is defined concretely and is written as follows. It can be registered as a function by writing `function name: {'function': function name}`.
#### Define the Input/Output of the function and implement the logic.

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/`new_algo_func`.py

The function code is described below.

```python
from .motion_correction import caiman_mc
from .cnmf import caiman_cnmf
def new_algo_func( # (*1)
image_data: ImageData, # (*2)
params: dict=None # (*3)
) -> dict(fluo=FluoData): # (*4)
import numpy as np
info = {
"fluo": FluoData(np.random.rand(100, 20), file_name="fluo"),
"image": ImageData(np.random.rand(10, 100, 100), file_name="image"),
"heatmap": HeatMapData(np.random.rand(20, 20), file_name="heatmap")
}
return info
```

caiman_wrapper_dict = {
'caiman': {
'caiman_mc': {
'function': caiman_mc,
},
'caiman_cnmf': {
'function': caiman_cnmf,
- Explanation:
- (*1) Function name can be any content.
- (*2) The first argument specifies the input data type. (This is also reflected in the GUI.)
- (*3) The second argument receives the function parameters.
- see. [Function Parameter Definitions](#function-parameter-definitions)
- (*4) The return value is a dictionary type. (This is also reflected in the GUI.)

#### Definition of information to be displayed in the GUI

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/\_\_init__.py

```python
from .new_algo_func import new_algo_func

new_algo_wrapper_dict = { # (*1)
'new_algo': { # (*2)
'new_algo_func': { # (*3)
'function': new_algo_func, # (*4)
'conda_name': 'new_algo', # (*5)
'conda_yaml': 'new_algo_env.yaml', # (*5)
},
}
}

```

- Explanation:
- (*1) The variable name is arbitrary, but `{algorithm_name}_wrapper_dict` is the standard.
- (*2) Algorithm name can be any text (display label to GUI)
- (*3) Algorithm function name can be any text (display label to GUI)
- (*4) Algorithm function name specifies the actual function name
- (*4, 5) The conda setting is optional (to be defined when using conda with snakemake)


### 3. Algorithm Registration

Actually register the **new_algorithm** function in [optinist/wrappers/optinist_wrapper/\_init__.py](https://github.com/oist/optinist/blob/main/optinist/wrappers/optinist_wrapper/__init__.py).
Register the created algorithm to the application by adding the following settings.

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/\_\_init__.py

```python
from .basic_neural_analysis import basic_neural_analysis_wrapper_dict
from .dimension_reduction import dimension_reduction_wrapper_dict
from .neural_population_analysis import neural_population_analysis_wrapper_dict
from .neural_decoding import neural_decoding_wrapper_dict

# ↓↓new add↓↓
from .new_algorithm import new_algorithm
# ↑↑new add↑↑

optinist_wrapper_dict = {
'optinist': {
'basic_neural_analysis': basic_neural_analysis_wrapper_dict,
'dimension_reduction': dimension_reduction_wrapper_dict ,
'neural_population_analysis': neural_population_analysis_wrapper_dict,
'neural_decoding': neural_decoding_wrapper_dict,
# ↓↓new add↓↓
'new_algorithm': {
'function': new_algorithm
}
# ↑↑new add↑↑
}
}
from .xxxx_wrapper import xxxx_wrapper_dict
from .yyyy_wrapper import yyyy_wrapper_dict
...
from .new_algo_wrapper import new_algo_wrapper_dict # <-- Add

wrapper_dict = {}
wrapper_dict.update(**xxxx_wrapper_dict)
wrapper_dict.update(**yyyy_wrapper_dict)
...
wrapper_dict.update(**new_algo_wrapper_dict) # <-- Add
```

After the registration process up to this point, restart the application browser to confirm that the algorithm has been added.


Restart the GUI and check TreeView, you can actually see the **new_algorithm**.
<p align="center">
<img width="300px" src="../_static/add_algorithm/new_algorithm.png" alt="new_algorithm" />
</p>
## Detailed Explanations

### DataClass

## 3. Describe function processing
### 3.1 import
Next, data inputs and outputs are defined.
Optinist defines several DataClasses to ensure consistency between Input and Output types. The main data types are as follows. These correspond to the color of each Node's handle.

Optinist support datatype.

- ImageData
- TimeSeriesData
- FluoData
Expand All @@ -131,34 +145,53 @@ Optinist support datatype.
- ScatterData
- BarData

### Input & Output handle

In the following example, the **new_algo_func** function takes **ImageData** and returns **FluoData**.

### 3.2 Input & Output handle
As an example, the **new_algorithm** function takes **ImageData** and returns **FluoData**.
The `from optinist.api.dataclass.dataclass import *` statement is the file where the dataclass is defined. params is necessary because it contains parameters for this function.
```python
from optinist.api.dataclass.dataclass import *

def new_algorithm(
def new_algo_func(
image_data: ImageData,
params: dict=None
) -> dict(fluo=FluoData):
return
```

Restart the GUI and put **new_algorithm**, and you will see that the handle color has changed.
<p align="center">
<img width="200px" src="../_static/add_algorithm/input_output.png" alt="input_output" />
</p>
Restart the Application and place **new_algo_func** on the GUI , and you will see that the handle color has changed.

![](../_static/add_algorithm/input_output.png)

### Function Parameter Definitions

Function input parameters (input on GUI) can be defined in the following file.

- {OPTINIST_SRC_DIR}/optinist/config/{algorithm_function_name}.yaml

- Sample:
```yaml
new_algo_params_1:
filtersize1: 10
filtersize2: 20
new_algo_params_2:
filtersize3: 30
filtersize4: 40
```

- Explanation:
- {algorithm_function_name} must match the actual function name.


### Drawing output results

### 3.3 Drawing output results
- Above we described the node input and output handle, here we describe the visualization of the result.
- The output of the function is a dictionary. (Here we use the variable name **info**.)
- First, the **fluo** variable that is the return value of the **new_algorithm function** is output by Wrap with **FluoData**. The name of the key in this case must match the **fluo** of the return value when declaring the function.
- The output of the function is a dictionary. (Here we use the variable **info**.)
- First, the **fluo** variable that is the return value of the **new_algo_func function** is output by Wrap with **FluoData**. The name of the key in this case must match the **fluo** of the return value when declaring the function.
- In addition, variables to be visualized are wrapped with their data types and output. In this example, **ImageData** and **HeatMap** are output.

```python
def new_algorithm(
def new_algo_func(
image_data: ImageData,
params: dict=None
) -> dict(fluo=FluoData):
Expand All @@ -171,20 +204,15 @@ def new_algorithm(
return info
```

Restart the GUI, connect imageNode and run it, and you will see the output as follows.

** Note: This process takes only 2~3 seconds, so if the process does not finish, there may be an error. If the error cannot be resolved, please post a message on slack or an issue.

<p align="center">
<img width="300px" src="../_static/add_algorithm/run.png" alt="run" />
</p>
Restart the Application, connect imageNode and run it, and you will see the output as follows.

<p align="center">
<img width="240px" src="../_static/add_algorithm/visualize_output.png" alt="output" />
</p>
- Note:
- This is a quick process (only a few seconds), so if the process does not terminate, an error may have occurred. If the error persists, please submit a question to the issue.

![](../_static/add_algorithm/run.png)

### 4. How to add parameters
The parameters are stored under [optinist/config](https://github.com/oist/optinist/tree/main/optinist/config) with **the same file name as the function name**.
The file name is `new_algorithm.yaml` and can be registered by creating it.
![](../_static/add_algorithm/visualize_output.png)

<style>
img { width: 50%; }
</style>

0 comments on commit c36adda

Please sign in to comment.