User Guide

1. 1. User Guide

The user guide mainly explains buttons and functions in QA.

1. 1. 1. Image List Page

After dividing WSIs into smaller image tiles, the user uploads them into QA in *Image List Page*.


The user follows instructions in order shown on the *Image List Page*

1. 1. 1. 1. Make Patches

Images tiles are sub-divided into 256 x 256 amenable to deep learning. The user can decide whether they want to remove white backgrounds when dividing these image tiles. This function is useful when there are large areas of white back ground. ("E.g. TMA spot")

1. 1. 1. 1. (Re)Train Model 0

A u-net consisting of a block depth of 5 layers and 113,306 parameters was trained on these patches in an auto-encoding fashion,

 such that model weights were optimized to reproduce the input as output with high fidelity. This helped create an initialized 
 base model that was subsequently used in the downstream supervised training.

1. 1. 1. 1. Embed Patches

A UMAP embedding plot is generated by *Embed Patches* with the current latest model. The user can *View Embedding* plot which helps to select patches for annotation which are dispersed in the model space.

1. 1. 1. Embedding Page

--- The embedding page is a 2D representation of all patches in the system, where patches perceived to be similar by the model are plotted closely together, and updates as the model trains.


When hovering over a dot, a preview of the patch is shown, and subsequently clicking on the dot takes the user to the annotation page centered around that patch.

1. 1. 1. 1. Information bar

The Information bar contains meta data information for the embedding page.


User could adjust the numbers of patches (*Patch Number*) plotted on the embedding table by *Embed Slider*.

• Latest ID* is the newest model number trained by the user; *Current ID* is the model number that the current

embedding plot employs. The user can change *Selected ID* to *Re_embed* a new embedding plot as the user is interesting to looking at the embedding plot of other model numbers. For example, the user usually will *Re_embed* as there is a later classier avaible. The user is also able to *Select ID* and then *View Embedding*, which is very useful to compare the difference between embedding plots of different models

Note: *Re_embed* does not automatically update the embedding plot, user needs to click *View Embedding* to update the page.

1. 1. 1. Annotation Page

--- User makes annotation in this page. To aid in this process, a number of common image annotation tools are provided, such as brushes and erasers of various sizes, along with polygon style annotation tools and region filling tools. Deep learning model can be trained after certain amount of training and testing samples are added to the dataset. The webpage will update to show deep learning output. User switches layers to view or hide annotation/prediction result. We are presenting a brief introduction to each function. 

1. 1. 1. 1. gitNavigation Bar

The *Navigation Bar* consists of function buttons, layer switches, and status indicators.


1. 1. 1. 1. 1. function buttons:

- *Retrain DL*: after user uploads some annotation to the dataset (at least on train set and one testing set), the user could use the annotation data to train a classifier. When the training process is finished, a u-net model can be applied to make suggestions which can be accepted or modified. There are two options for *Retrain DL*: *From base* is to apply the annotation dataset on the autoencoder (model 0); *From last* is to apply the annotation dataset on the last previous model, which can be used when the previous model gives satisfied prediction already. - *Download*: the user can export *Trained DL*, *DL Result Image*, and *Human Annotation Image*. *Train DL* is the download link for the latest trained model (path file). *DL Result Image* is to download the binary prediction mask on the image, where white areas represent regions of interest. *Human Annotation Image* is to download the annotation mask of the current image. - *Previous image*, *Next image*: user could direct to the previous/next image in the image list with out exiting the page.

1. 1. 1. 1. 1. layer switches:

User can use these layer switches to decide what to look at in the annotation page. All switches have related short cut keys. - *Image Information (I)*: the switch for the information box with short cut key *I* - *Annotation (Q)*: the switch for *annotation layer* with short cut key *Q*; see gif demo - *Prediction (W)*: the switch for *prediction layer* with short cut key *W*; see gif demo Note: When hiding *annotation layer*, the user can not make annotations. It is very helpful to switch layers frequently to check the boundary annotation when making manual annotation or revising predictions.


1. 1. 1. 1. 1. status indicators:

The "traffic light" status indicators reveals how the program runs in the backend to some extent. The status indicators have

• green*, *yellow*, and *red* colors, which representing *Done*, *Running*, and *Not available*. QA could communicate with user through these traffic lights.

- *Model*: indicates whether classifier is ready or not. when user presses the *Retrain DL*, the *Model* light turns from *green* to *yellow*. If an error happens in the backend when training the model, it will turn "red", so that the user would know that there is an error happening in the backend. - *Superpixel*: indicates whether the superpixel segmentation is ready in the backend. - *Prediction*: indicates whether the there is a prediction layer ready for user to modify. When a new *Model* is ready (turning to green), the *prediction* will turn to *yellow*, generating the prediction layer suggested by the new model. A few seconds later, the *Prediction* turns *green*, indicating that the annotation suggestion of the new model is ready to use.

1. 1. 1. 1. Information Window

The *Information Window* contains meta data information for the image tile.


It shows *project name*, *Image name* and *Image size*. Besides, *crop size* can adjust the size of the selection rectangle and the *Zoom factor* can zoom in/out the annotation page. The user can view the image annotation percentage as well.


The user can show/hide the *information Window* in the *status indicators* bar

1. 1. 1. 1. Annotation Window

• Annotation Window* is where the user makes annotation by adaptively utilizing various annotation tool provide by QA.

First, user selects a square region which is then loaded into the high magnification *Annotation Window*. Then, the user annotates positive regions in turquoise and non-target areas in fuchsia using common image markup tools. After annotating at least 3 patches, the user can train a DL classifier to generate annotation suggestions in white overlay. The user may then import the classifier’s suggestions into annotation window and edit, if needed, before accepting.


The followings are the introductions to each function in the annotation window toolbar. Each functions has a specialized hot key which helps users annotate. - *Freehand* (A): in *Freehand* mode, user can delineate the boundary of the primitives by moving the mouse


- *Superpixels* (S): A superpixel is defined as a group of adjacent pixels sharing similar characteristics in terms of chromatic, texture, or deep learned feature values. QA provides *Superpixels* mode, which enabled one-click selection for a subset of primitives, notably improving annotation efficiency.


- *Flood fill* (D): The user can annotate a closed set of image by one click in *Flood fill* mode.


- *Eraser* (F): The user can de-annotated pixels with erasers of different sizes provided in *Eraser* mode.


- *Import DL result into annotator* (G): When the user has train a DL classifier to generate annotation suggestions white overlay, the user may then *import* the classifier’s suggestions into annotation window and edit. - *Upload complete annotation* (H): Thw user can *upload* the annotated patches into *training* or *testing* dataset.


- *Annotate positive region* (Z), *Annotate negative region* (X), and *Annotate Unknown region* (C): Under theses modes, the user could annotate pixels with different labels. The user annotates regions of interest as *positive region*, marking non-primitive regions as *negative region*. QA also provides the option of marking an area as *unknown region* when users could not identify a region. - *Undo* (V), *Redo* (B), and *Reset* (N): QA also keeps track of user's annotation operations. Therefore, the user could easily *Redo* or *Undo* any annotations if misannotating some areas. Also, user can remove all annotations in the annotation