Skip to content

Commit

Permalink
Minor documentation tweaks (#111)
Browse files Browse the repository at this point in the history 
* Formatting updates

* Remove out-of-date model details

Refer others to the training repo instead
  • Loading branch information
@tayden
tayden committed May 7, 2024
1 parent a816072 commit 54b90db
Show file tree
Hide file tree
Showing 5 changed files with 130 additions and 86 deletions.
17 changes: 0 additions & 17 deletions docs/about.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,26 +105,9 @@ flowchart TD

## Model Training

The LRASPP MobileNetV3-Large[^1] model was trained using a stochastic gradient descent
optimizer with the learning rate set to $0.35$, and L2 weight decay set to
$3 \times 10^{-6}$. The model was trained for a total of 100 epochs. A cosine annealing
learning rate schedule[^2] was used to improve accuracy. The loss function used for
training was Focal Tversky Loss[^3], with parameters
$\alpha=0.7, \beta=0.3, \gamma=4.0 / 3.0$.

The model was trained on an AWS p3.8xlarge instance with 4 Nvidia Tesla V100 GPUS and
took 18 hours to finish. At the end of training, the model parameters which achieved
the best IoU~$\mu$~ score on the validation data split were saved for inference. It is
these parameters that were used to calculate the final performance statistics for the
model on the test split.

Source code for model training is available on GitHub
at [:material-github: hakai-ml-train](https://github.com/tayden/hakai-ml-train).

[^1]: [A. Howard et al., ‘Searching for MobileNetV3’. arXiv, 2019.](https://arxiv.org/abs/1905.02244)
[^2]: [I. Loshchilov and F. Hutter, ‘SGDR: Stochastic Gradient Descent with Warm Restarts’. arXiv, 2016.](https://arxiv.org/abs/1608.03983)
[^3]: [N. Abraham and N. M. Khan, ‘A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentation’. arXiv, 2018.](https://arxiv.org/abs/1810.07842)

### Training overview

```mermaid
Expand Down
60 changes: 39 additions & 21 deletions docs/beginner_guide/execution.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,37 +16,46 @@ Open anaconda prompt by searching “Anaconda Prompt” in search tool bar.

![Change Environments 1](images/change_environments1.png)

The default environment is called `(base)` which will appear in brackets before each line of text.
The default environment is called `(base)` which will appear in brackets before each
line of text.

![Change Environments 2](images/change_environments2.png)

To change to the new environment that you created in Part 1, type: `conda activate KelpSegmentation` and press enter.
In this command, "KelpSegmentation" was the name that you chose for your new environment.
Now you will see that the environment has changed from `(base)` to `(KelpSegmentation)` as KelpSegmentation is now appearing in brackets.
To change to the new environment that you created in Part 1,
type: `conda activate KelpSegmentation` and press enter.
In this command, "KelpSegmentation" was the name that you chose for your new
environment.
Now you will see that the environment has changed from `(base)` to `(KelpSegmentation)`
as KelpSegmentation is now appearing in brackets.

!!! note

If you used a different name for your environment, be sure to replace `conda activate KelpSegmentation` with
`conda activate <your-environment-name-here>`
If you used a different name for your environment, be sure to replace
`conda activate KelpSegmentation` with `conda activate <your-environment-name-here>`

You can see a list of all installed environments by typing and entering the command: `conda env list`.
You can see a list of all installed environments by typing and entering the
command: `conda env list`.

***

## Run Segmentation Tool

![Run Tool](images/run_tool.png)

Type `kom find-kelp` and copy and paste the location of the image in which you would like to detect kelp. Add a space
and then type or paste the location you would like the output file to be saved on your computer with the name of the
Type `kom find-kelp` and copy and paste the location of the image in which you would
like to detect kelp. Add a space
and then type or paste the location you would like the output file to be saved on your
computer with the name of the
output file.

In this example the orthomosaic to be classified is named “Kelp_Bed_Orthomosaic_2022.tif” and its location is in the C
In this example the orthomosaic to be classified is named
“Kelp_Bed_Orthomosaic_2022.tif” and its location is in the C
drive, documents folder, Drone_Images subfolder.

`C:\Users\sbs33.DESKTOP-K0SDD1B\Documents\Drone_Images\Kelp_Bed_Orthomosaic_2022.tif`

The output results will be sent to the same Drone_Images folder and will have the name “Kelp_Bed_Orthomosiac_2022_output.tif”,
The output results will be sent to the same Drone_Images folder and will have the name
“Kelp_Bed_Orthomosiac_2022_output.tif”,
where .tif is the file type.

`C:\Users\sbs33.DESKTOP-K0SDD1B\Documents\Drone_Images\Kelp_Bed_Orthomosaic_2022_output.tif`
Expand All @@ -57,29 +66,38 @@ So, the entire command in the Anaconda prompt is:
kom find-kelp C:\Users\sbs33.DESKTOP-K0SDD1B\Documents\Drone_Images\Kelp_Bed_Orthomosaic_2022.tif C:\Users\sbs33.DESKTOP-K0SDD1B\Documents\Drone_Images\Kelp_Bed_Orthomosaic_2022_output.tif
```

Press ++enter++ and wait for the segmentation to run; the time may vary depending on the size of the image.
Press ++enter++ and wait for the segmentation to run; the time may vary depending on the
size of the image.

!!! note

You can always get help documentation for the `kom find-kelp` tool by typing into the prompt `kom find-kelp --help`.
You can always get help documentation for the `kom find-kelp` tool by typing into
the prompt `kom find-kelp --help`.

This help documentation is the same as what is shown under the :doc:`Command Line Reference <./cli>`.
This help documentation is the same as what is shown under
the :doc:`Command Line Reference <./cli>`.

!!! tip

If you manually type in the path to your file, use the `<tab>` key to have your operating system autocomplete the paths for you.
If you manually type in the path to your file, use the `<tab>` key to have your
operating system autocomplete the paths for you.

*e.g.*, if you type `kom find-kelp C:\\Desk`, then press the `<tab>` key, your operating system should complete the
command for you so it reads: `kom find-kelp C:\\Desktop`.
*e.g.*, if you type `kom find-kelp C:\\Desk`, then press the `<tab>` key, your
operating system should complete the command for you so it
reads: `kom find-kelp C:\\Desktop`.

If the partial path is ambiguous, tapping `<tab>` multiple times will cycle through the possible path options.
If the partial path is ambiguous, tapping `<tab>` multiple times will cycle through
the possible path options.

You can continue doing this as you type in the location to autocomplete the text for long path names.
It is faster, and less error prone than typing the whole path yourself.
You can continue doing this as you type in the location to autocomplete the text
for long path names. It is faster, and less error prone than typing the whole
path yourself.

![Complete Segmentation](images/complete_segmentation.png)

Once the processing bar is at 100%, you can open the results in an image processing or spatial analysis software such as QGIS or ArcGIS. Review the results for errors and edit as needed.
Once the processing bar is at 100%, you can open the results in an image processing or
spatial analysis software such as QGIS or ArcGIS. Review the results for errors and edit
as needed.

**Part 2 Kelp Segmentation is now complete!**

Expand Down
64 changes: 42 additions & 22 deletions docs/beginner_guide/post_processing.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Methods for Post-Processing Kelp-O-Matic Output[^1]

This document describes the methods used for generating analysis ready data on species-level kelp
extent from outputs from the Kelp-O-Matic tool. Final outputs consist of polygon features (.shp)
that contain species-level classification of emergent canopy kelp present in drone imagery.
This document describes the methods used for generating analysis ready data on
species-level kelp extent from outputs from the Kelp-O-Matic tool. Final outputs
consist of polygon features (.shp) that contain species-level classification of
emergent canopy kelp present in drone imagery.

These instructions assume a basic level of experience with the ArcMAP GIS interface. For questions,
These instructions assume a basic level of experience with the ArcMAP GIS interface. For
questions,
please contact [Luba Reshitnyk](email:luba.reshitnyk@hakai.org) at
the [Hakai Institute](https://hakai.org).

Expand All @@ -31,12 +33,15 @@ graph TB

## Detailed Workflow

This section assumes that you have obtained output from the Kelp-O-Matic (.tif) and have a new
instance of ArcMap (version 10.8). These same steps can be done using other GIS software (e.g. QGIS)
This section assumes that you have obtained output from the Kelp-O-Matic (.tif) and have
a new
instance of ArcMap (version 10.8). These same steps can be done using other GIS
software (e.g. QGIS)

??? tip "Tip: ArcMap Keyboard Shortcuts"

Keyboard shortcuts can make the editing process much faster. To enable some, go to `Customize > Customize Mode > Keyboard…`.
Keyboard shortcuts can make the editing process much faster. To enable some, go
to `Customize > Customize Mode > Keyboard…`.

Some useful shortcuts include:

Expand All @@ -47,11 +52,13 @@ instance of ArcMap (version 10.8). These same steps can be done using other GIS
3. Freehand tool (set to ++ctrl+r++)
1. Toggle to freehand tool to manually digitize kelp feature polygons.
4. Clip (set to ++ctrl+w++)
1. Quickly draw a polygon then click ctrl+W to delete the polygon features within the newly drawn polygon. Then delete that polygon.
1. Quickly draw a polygon then click ctrl+W to delete the polygon features
within the newly drawn polygon. Then delete that polygon.

### Load Kelp-O-Matic Raster

Load the output `.tif` file from Kelp-O-Matic into ArcMap. The raster will have the following
Load the output `.tif` file from Kelp-O-Matic into ArcMap. The raster will have the
following
attributes:

| Output value (gridcode) | Class |
Expand All @@ -77,10 +84,12 @@ Convert this raster into polygons using the `Raster to Polygon` tool in ArcMap.
2. Type = "float"
3. Click `OK`
2. Calculate the area for each polygon
1. Right-click the "area" header in the attribute table, select `Calculate Geometry...`
1. Right-click the "area" header in the attribute table,
select `Calculate Geometry...`
1. Property = "area"
2. Units = "square meters"
3. Coordinate system = choose an appropriate projection that preserves area (e.g. NAD83 BC
3. Coordinate system = choose an appropriate projection that preserves area (
e.g. NAD83 BC
Albers Equal Area EPSG = 3005)
4. Click "OK"
3. Add a new "species" field
Expand All @@ -95,7 +104,8 @@ Convert this raster into polygons using the `Raster to Polygon` tool in ArcMap.
#### Delete small polygons that are unlikely to be kelp

1. Start editing this feature polygon layer
1. In the Editor toolbar click `Editor > start editing`. Select the feature layer you will be
1. In the Editor toolbar click `Editor > start editing`. Select the feature layer
you will be
editing
2. Right click layer in the table of contents, Select `Open attribute table`.
Click `Select by attribute`
Expand All @@ -105,7 +115,8 @@ Convert this raster into polygons using the `Raster to Polygon` tool in ArcMap.

#### Manually delete non-kelp polygons

There may be areas where KoM classifier falsely detected kelp and these polygons need to be removed
There may be areas where KoM classifier falsely detected kelp and these polygons need to
be removed
manually.

1. Select using the edit tool or “Select by Lasso” tool
Expand All @@ -114,29 +125,38 @@ manually.

#### Add kelp not detected by KoM

If there are locations where emergent canopy kelp was missed by the KoM classifier (example in
figure below) manually digitize these features using the Freehand Tool. You will need to manually
classify these new regions to species (change value in the attribute table). A 1:100 scale is
If there are locations where emergent canopy kelp was missed by the KoM classifier (
example in
figure below) manually digitize these features using the Freehand Tool. You will need to
manually
classify these new regions to species (change value in the attribute table). A 1:100
scale is
recommended.

![Missed Kelp](./images/post_processing2.png)

### Review Species Attributes

There may be some areas where the KoM classifier misclassified the species in a given area. These
polygons can be edited manually. If you're just interested in presence/absence don't worry about
There may be some areas where the KoM classifier misclassified the species in a given
area. These
polygons can be edited manually. If you're just interested in presence/absence don't
worry about
this step.

1. Select all polygon features classified as giant kelp by opening the attribute table and “select
by attribute”. Select features representing giant kelp by typing: `"gridcode" = 2`. At the bottom
1. Select all polygon features classified as giant kelp by opening the attribute table
and “select
by attribute”. Select features representing giant kelp by typing: `"gridcode" = 2`.
At the bottom
of the attribute table click on the "Show selected records" tab.
2. Right-click the "species" field you created earlier. Select "Field Calculator".
3. In the text box enter "Macrocystis pyrifera" (with quotes included) and click OK. This will fill
3. In the text box enter "Macrocystis pyrifera" (with quotes included) and click OK.
This will fill
out that attribute for all the selected polygons.

Repeat steps 1 - 3 for bull kelp by using `"gridcode" = 3` and "Nereocystis luetkeana".

1. Review all the polygon species classification and manually change ones that the KoM classifier
1. Review all the polygon species classification and manually change ones that the KoM
classifier
misclassified by changing the text in the "species" field for that polygon feature.
2. Save edits.
3. Delete the "gridcode" attribute once you are done.
Expand Down
57 changes: 38 additions & 19 deletions docs/beginner_guide/setup.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# Set-up and Installation

Before you can run the Kelp-O-Matic tool, you will need to set up some programs which will allow the tool to run. Once
Before you can run the Kelp-O-Matic tool, you will need to set up some programs which
will allow the tool to run. Once
set-up is complete the tool can be run very easily from ones terminal.

## Install Anaconda

If not already installed, download Anaconda on your computer from the [Anaconda download website](https://www.anaconda.com/products/distribution).
Follow the prompts to install the most recent version of Anaconda for your operating system.
If not already installed, download Anaconda on your computer from
the [Anaconda download website](https://www.anaconda.com/products/distribution).
Follow the prompts to install the most recent version of Anaconda for your operating
system.

![Install Anaconda](images/install_anaconda.png)

Expand All @@ -28,18 +31,24 @@ In the prompt window, create a new Anaconda environment by typing the following
conda create -n KelpSegmentation python=3.10
```

This will create an isolated Python environment for packages and their dependencies to be installed in. The name of the
environment is `KelpSegmentation` in this example, but you can choose any name you like. This command also installs Python version 3.10 into your new environment.
This will create an isolated Python environment for packages and their dependencies to
be installed in. The name of the
environment is `KelpSegmentation` in this example, but you can choose any name you like.
This command also installs Python version 3.10 into your new environment.

??? question "Why do we use a new environment?"

Using a new Conda environment is a good practice to get into when installing Python packages. It ensures that the software
you install does not conflict with other software on your computer, as well as makes it easy to remove when you don't want it anymore.
Using a new Conda environment is a good practice to get into when installing Python
packages. It ensures that the software you install does not conflict with other
software on your computer, as well as makes it easy to remove when you don't want
it anymore.

## Change Environments

Now, we're going to activate the environment we just created. By default, the Anaconda Prompt always starts in the `(base)` environment.
You can tell this is the case, because `(base)` will appear in brackets before each line of text.
Now, we're going to activate the environment we just created. By default, the Anaconda
Prompt always starts in the `(base)` environment.
You can tell this is the case, because `(base)` will appear in brackets before each line
of text.

![Change Environments 1](images/change_environments1.png)

Expand All @@ -51,32 +60,40 @@ conda activate KelpSegmentation

!!! note

If you used a different name for your environment, be sure to replace `conda activate KelpSegmentation` with
`conda activate <your-environment-name-here>`
If you used a different name for your environment, be sure to replace
`conda activate KelpSegmentation` with `conda activate <your-environment-name-here>`

You can see a list of all installed environments by typing and entering the command: `conda env list`.
You can see a list of all installed environments by typing and entering the
command: `conda env list`.

![Change Environments 2](images/change_environments2.png)

You should see that the environment has changed from `(base)` to `(KelpSegmentation)` in the prompt.
You should see that the environment has changed from `(base)` to `(KelpSegmentation)` in
the prompt.

***

## Install packages

Now that we're in the new environment, we can install the packages that the Kelp-O-Matic tool needs to run, followed by Kelp-O-Matic itself.
Now that we're in the new environment, we can install the packages that the Kelp-O-Matic
tool needs to run, followed by Kelp-O-Matic itself.

At this stage, you can directly reference the [installation instructions](../installation.md) if you find that easier, or continue with the following instructions.
At this stage, you can directly reference
the [installation instructions](../installation.md) if you find that easier, or continue
with the following instructions.

### Install PyTorch

Kelp-O-Matic relies on PyTorch to run. PyTorch is a machine learning library for Python.
PyTorch can be installed to use your CPU only, or can make use of an NVIDIA GPUs you may have in your machine to accelerate processing.
PyTorch can be installed to use your CPU only, or can make use of an NVIDIA GPUs you may
have in your machine to accelerate processing.

The most up-to-date and reliable instructions for installing PyTorch can be found on the [PyTorch website](https://pytorch.org).
The most up-to-date and reliable instructions for installing PyTorch can be found on
the [PyTorch website](https://pytorch.org).
Typically, the installation commands are:

##### GPU

```console
conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia
```
Expand All @@ -89,15 +106,17 @@ conda install pytorch torchvision cpuonly -c pytorch

### Install Kelp-O-Matic

Now that PyTorch is installed, we can install Kelp-O-Matic itself. To do so, run the command:
Now that PyTorch is installed, we can install Kelp-O-Matic itself. To do so, run the
command:

```console
conda install -c conda-forge kelp-o-matic
```

**Part 1 set-up is now complete!**

You are now ready to run the Kelp-O-Matic tool. Now that the setup steps are completed, you will not
You are now ready to run the Kelp-O-Matic tool. Now that the setup steps are completed,
you will not
have to repeat them again unless installing on a different computer.

Continue to [Part 2: Running the Segmentation tool](./execution.md)
Expand Down
Loading

0 comments on commit 54b90db

Please sign in to comment.