Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update installation docs #445

Merged
merged 1 commit into from
Mar 30, 2023
Merged

Update installation docs #445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions docs/sphinx-usage.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,18 @@ pip install -r requirements.txt

## Usage

- output html
- output html to _build (use cache)
```
make html
```
- output html to _build (all rebuild)
```
# "-E" is an option for all-rebuilding.
$env:SPHINXOPTS="-E"; make html; $env:SPHINXOPTS=""
```
- output html to _build (use cache)
```
make html
```

- output html to _build (all rebuilding)
- *"-E" is an option for all-rebuilding.*
- linux, mac
```
export SPHINXOPTS="-E"; make html; export SPHINXOPTS=""
```
- windows
```
$env:SPHINXOPTS="-E"; make html; $env:SPHINXOPTS=""
```
264 changes: 146 additions & 118 deletions docs/utils/add_algorithm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,127 +1,141 @@
Procedure for adding a new algorithm
=================
# Procedure for adding a new algorithm

How to add algorithms
```{contents}
:depth: 3
```

* [1. Create algorithm function file](#1-create-algorithm-function-file)
* [2. Register your algorithm](#2-register-your-algorithm)
* [3. Describe function processing](#3-describe-function-processing)
* [3.1 import](#31-import)
* [3.2 Input & Output handle](#32-input--output-handle)
* [3.3 Drawing output results](#33-drawing-output-results)
* [4. How to add parameters](#4-how-to-add-parameters)
## Example of Algorithm Addition Procedure

## 1. Create algorithm function file
First, create a python file at the appropriate location in the following directory.
Here is an example of how to create a function named `new_algorithm`. The file should be created at the following location [optinist/wrappers/optinist_wrapper](https://github.com/oist/optinist/tree/main/optinist/wrappers/optinist_wrapper).
Below we describe an example procedure for adding a new algorithm.

Create it under the name **new_algorith.py**.
**Prerequisite**
- Sample Algorithm Name ... `new_algo`
- Sample Algorithm Function Name ... `new_algo_func`
- {OPTINIST_SRC_DIR} ... Replace with the actual source storage directory path.

```python:new_algorithm.py
def new_algorithm():
return
```

[algorithm list](https://github.com/oist/optinist/tree/develop/optinist/wrappers)

- \_\_init__.py - ①
- caiman_wrapper
- \_\_init__.py - ②
- caiman_mc.py
- caiman_cnmf.py
- suite2p_wrapper
- \_\_init__.py
- suite2p_file_convert.py
- suite2p_registration.py
- suite2p_roi.py
- suite2p_cnmf.py
- optinist_wrapper
- \_\_init__.py
- basic neural analysis
- dimension reduction
- neural population analysis
- neural decoding
- `new_algorithm.py` - ＊

<br />

## 2. Register your algorithm
①: To be able to use the created **new_algorithm** on the GUI, it is necessary to register it in the \_\_init__.py. For example, [optinist/wrappers/\_\_init__.py](https://github.com/oist/optinist/blob/main/optinist/wrappers/__init__.py) in ① is as follows.
These are reading one level down, \_\_init__.py.
### 1. Prepare necessary directories and files for the algorithm

First, prepare the necessary directories and files for the algorithm.

```python
from .caiman_wrapper import caiman_wrapper_dict
from .suite2p_wrapper import suite2p_wrapper_dict
from .optinist_wrapper import optinist_wrapper_dict
- {OPTINIST_SRC_DIR}/optinist/wrappers/
- \_\_init__.py
- xxxx_wrapper/
- yyyy_wrapper/
- ...
- `new_algo_wrapper/` (*1)
- \_\_init__.py (*2)
- `new_algo_func.py` (*3)
- ... (*4)

wrapper_dict = {}
wrapper_dict.update(**caiman_wrapper_dict)
wrapper_dict.update(**suite2p_wrapper_dict)
wrapper_dict.update(**optinist_wrapper_dict)
- Explanation:
- (*1) The name is arbitrary, but `{algorithm name}_wrapper` is the standard.
- (*2) Empty first.
- (*3) Empty first.
- (*4) Prepare other files to be added.

### 2. Algorithm implementation

#### Import statement description

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/`new_algo_func`.py

```python
from optinist.api.dataclass.dataclass import *
```

- Explanation:
- If the required dataclass does not exist, you can add your own.
- see. [DataClass](#dataclass)


②: In [optinist/wrappers/caiman_wrapper/\_\_init__py](https://github.com/oist/optinist/blob/main/optinist/wrappers/caiman_wrapper/__init__.py) function is defined concretely and is written as follows. It can be registered as a function by writing `function name: {'function': function name}`.
#### Define the Input/Output of the function and implement the logic.

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/`new_algo_func`.py

The function code is described below.

```python
from .motion_correction import caiman_mc
from .cnmf import caiman_cnmf
def new_algo_func( # (*1)
image_data: ImageData, # (*2)
params: dict=None # (*3)
) -> dict(fluo=FluoData): # (*4)
import numpy as np
info = {
"fluo": FluoData(np.random.rand(100, 20), file_name="fluo"),
"image": ImageData(np.random.rand(10, 100, 100), file_name="image"),
"heatmap": HeatMapData(np.random.rand(20, 20), file_name="heatmap")
}
return info
```

caiman_wrapper_dict = {
'caiman': {
'caiman_mc': {
'function': caiman_mc,
},
'caiman_cnmf': {
'function': caiman_cnmf,
- Explanation:
- (*1) Function name can be any content.
- (*2) The first argument specifies the input data type. (This is also reflected in the GUI.)
- (*3) The second argument receives the function parameters.
- see. [Function Parameter Definitions](#function-parameter-definitions)
- (*4) The return value is a dictionary type. (This is also reflected in the GUI.)

#### Definition of information to be displayed in the GUI

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/`new_algo_wrapper`/\_\_init__.py

```python
from .new_algo_func import new_algo_func

new_algo_wrapper_dict = { # (*1)
'new_algo': { # (*2)
'new_algo_func': { # (*3)
'function': new_algo_func, # (*4)
'conda_name': 'new_algo', # (*5)
'conda_yaml': 'new_algo_env.yaml', # (*5)
},
}
}

```

- Explanation:
- (*1) The variable name is arbitrary, but `{algorithm_name}_wrapper_dict` is the standard.
- (*2) Algorithm name can be any text (display label to GUI)
- (*3) Algorithm function name can be any text (display label to GUI)
- (*4) Algorithm function name specifies the actual function name
- (*4, 5) The conda setting is optional (to be defined when using conda with snakemake)


### 3. Algorithm Registration

Actually register the **new_algorithm** function in [optinist/wrappers/optinist_wrapper/\_init__.py](https://github.com/oist/optinist/blob/main/optinist/wrappers/optinist_wrapper/__init__.py).
Register the created algorithm to the application by adding the following settings.

- Target file
- {OPTINIST_SRC_DIR}/optinist/wrappers/\_\_init__.py

```python
from .basic_neural_analysis import basic_neural_analysis_wrapper_dict
from .dimension_reduction import dimension_reduction_wrapper_dict
from .neural_population_analysis import neural_population_analysis_wrapper_dict
from .neural_decoding import neural_decoding_wrapper_dict

# ↓↓new add↓↓
from .new_algorithm import new_algorithm
# ↑↑new add↑↑

optinist_wrapper_dict = {
'optinist': {
'basic_neural_analysis': basic_neural_analysis_wrapper_dict,
'dimension_reduction': dimension_reduction_wrapper_dict ,
'neural_population_analysis': neural_population_analysis_wrapper_dict,
'neural_decoding': neural_decoding_wrapper_dict,
# ↓↓new add↓↓
'new_algorithm': {
'function': new_algorithm
}
# ↑↑new add↑↑
}
}
from .xxxx_wrapper import xxxx_wrapper_dict
from .yyyy_wrapper import yyyy_wrapper_dict
...
from .new_algo_wrapper import new_algo_wrapper_dict # <-- Add

wrapper_dict = {}
wrapper_dict.update(**xxxx_wrapper_dict)
wrapper_dict.update(**yyyy_wrapper_dict)
...
wrapper_dict.update(**new_algo_wrapper_dict) # <-- Add
```

After the registration process up to this point, restart the application browser to confirm that the algorithm has been added.


Restart the GUI and check TreeView, you can actually see the **new_algorithm**.
<p align="center">
<img width="300px" src="../_static/add_algorithm/new_algorithm.png" alt="new_algorithm" />
</p>
## Detailed Explanations

### DataClass

## 3. Describe function processing
### 3.1 import
Next, data inputs and outputs are defined.
Optinist defines several DataClasses to ensure consistency between Input and Output types. The main data types are as follows. These correspond to the color of each Node's handle.

Optinist support datatype.

- ImageData
- TimeSeriesData
- FluoData
Expand All @@ -131,34 +145,53 @@ Optinist support datatype.
- ScatterData
- BarData

### Input & Output handle

In the following example, the **new_algo_func** function takes **ImageData** and returns **FluoData**.

### 3.2 Input & Output handle
As an example, the **new_algorithm** function takes **ImageData** and returns **FluoData**.
The `from optinist.api.dataclass.dataclass import *` statement is the file where the dataclass is defined. params is necessary because it contains parameters for this function.
```python
from optinist.api.dataclass.dataclass import *

def new_algorithm(
def new_algo_func(
image_data: ImageData,
params: dict=None
) -> dict(fluo=FluoData):
return
```

Restart the GUI and put **new_algorithm**, and you will see that the handle color has changed.
<p align="center">
<img width="200px" src="../_static/add_algorithm/input_output.png" alt="input_output" />
</p>
Restart the Application and place **new_algo_func** on the GUI , and you will see that the handle color has changed.

![](../_static/add_algorithm/input_output.png)

### Function Parameter Definitions

Function input parameters (input on GUI) can be defined in the following file.

- {OPTINIST_SRC_DIR}/optinist/config/{algorithm_function_name}.yaml

- Sample:
```yaml
new_algo_params_1:
filtersize1: 10
filtersize2: 20
new_algo_params_2:
filtersize3: 30
filtersize4: 40
```

- Explanation:
- {algorithm_function_name} must match the actual function name.


### Drawing output results

### 3.3 Drawing output results
- Above we described the node input and output handle, here we describe the visualization of the result.
- The output of the function is a dictionary. (Here we use the variable name **info**.)
- First, the **fluo** variable that is the return value of the **new_algorithm function** is output by Wrap with **FluoData**. The name of the key in this case must match the **fluo** of the return value when declaring the function.
- The output of the function is a dictionary. (Here we use the variable **info**.)
- First, the **fluo** variable that is the return value of the **new_algo_func function** is output by Wrap with **FluoData**. The name of the key in this case must match the **fluo** of the return value when declaring the function.
- In addition, variables to be visualized are wrapped with their data types and output. In this example, **ImageData** and **HeatMap** are output.

```python
def new_algorithm(
def new_algo_func(
image_data: ImageData,
params: dict=None
) -> dict(fluo=FluoData):
Expand All @@ -171,20 +204,15 @@ def new_algorithm(
return info
```

Restart the GUI, connect imageNode and run it, and you will see the output as follows.

** Note: This process takes only 2~3 seconds, so if the process does not finish, there may be an error. If the error cannot be resolved, please post a message on slack or an issue.

<p align="center">
<img width="300px" src="../_static/add_algorithm/run.png" alt="run" />
</p>
Restart the Application, connect imageNode and run it, and you will see the output as follows.

<p align="center">
<img width="240px" src="../_static/add_algorithm/visualize_output.png" alt="output" />
</p>
- Note:
- This is a quick process (only a few seconds), so if the process does not terminate, an error may have occurred. If the error persists, please submit a question to the issue.

![](../_static/add_algorithm/run.png)

### 4. How to add parameters
The parameters are stored under [optinist/config](https://github.com/oist/optinist/tree/main/optinist/config) with **the same file name as the function name**.
The file name is `new_algorithm.yaml` and can be registered by creating it.
![](../_static/add_algorithm/visualize_output.png)

<style>
img { width: 50%; }
</style>