User Guide
The user guide mainly explains buttons and functions in QA.
After dividing WSIs into smaller image tiles, the user uploads them into QA in Image List Page.
The user follows instructions in order shown on the Image List Page
Images tiles are sub-divided into 256 x 256 amenable to deep learning. The user can decide whether they want to remove white backgrounds when dividing these image tiles. This function is useful when there are large areas of white back ground. ("E.g. TMA spot")
A u-net consisting of a block depth of 5 layers and 113,306 parameters was trained on these patches in an auto-encoding fashion, such that model weights were optimized to reproduce the input as output with high fidelity. This helped create an initialized base model that was subsequently used in the downstream supervised training.
A UMAP embedding plot is generated by Embed Patches with the current latest model. The user can View Embedding plot which helps to select patches for annotation which are dispersed in the model space.
The embedding page is a 2D representation of all patches in the system, where patches perceived to be similar by the model are plotted closely together, and updates as the model trains.
When hovering over a dot, a preview of the patch is shown, and subsequently clicking on the dot takes the user to the annotation page centered around that patch.
The Information bar contains meta data information for the embedding page.
User could adjust the numbers of patches (Patch Number) plotted on the embedding table by Embed Slider. Latest ID is the newest model number trained by the user; Current ID is the model number that the current embedding plot employs. The user can change Selected ID to Re_embed a new embedding plot as the user is interesting to looking at the embedding plot of other model numbers. For example, the user usually will Re_embed as there is a later classier avaible. The user is also able to Select ID and then View Embedding, which is very useful to compare the difference between embedding plots of different models
Note: Re_embed does not automatically update the embedding plot, user needs to click View Embedding to update the page.
User makes annotation in this page. To aid in this process, a number of common image annotation tools are provided, such as
brushes and erasers of various sizes, along with polygon style annotation tools and region filling tools. Deep learning model
can be trained after certain amount of training and testing samples are added to the dataset.
The webpage will update to show deep learning output. User switches layers to view or hide annotation/prediction result.
We are presenting a brief introduction to each function.
The Navigation Bar consists of function buttons, layer switches, and status indicators.
- Retrain DL: after user uploads some annotation to the dataset (at least on train set and one testing set), the user could use the annotation data to train a classifier. When the training process is finished, a u-net model can be applied to make suggestions which can be accepted or modified. There are two options for Retrain DL: From base is to apply the annotation dataset on the autoencoder (model 0); From last is to apply the annotation dataset on the last previous model, which can be used when the previous model gives satisfied prediction already.
- Download: the user can export Trained DL, DL Result Image, and Human Annotation Image. Train DL is the download link for the latest trained model (path file). DL Result Image is to download the binary prediction mask on the image, where white areas represent regions of interest. Human Annotation Image is to download the annotation mask of the current image.
- Previous image, Next image: user could direct to the previous/next image in the image list with out exiting the page.
User can use these layer switches to decide what to look at in the annotation page. All switches have related short cut keys.
- Image Information (I): the switch for the information box with short cut key I
- Annotation (Q): the switch for annotation layer with short cut key Q; see gif demo
- Prediction (W): the switch for prediction layer with short cut key W; see gif demo Note: When hiding annotation layer, the user can not make annotations. It is very helpful to switch layers frequently to check the boundary annotation when making manual annotation or revising predictions.
The "traffic light" status indicators reveals how the program runs in the backend to some extent. The status indicators have green, yellow, and red colors, which representing Done, Running, and Not available. QA could communicate with user through these traffic lights.
- Model: indicates whether classifier is ready or not. when user presses the Retrain DL, the Model light turns from green to yellow. If an error happens in the backend when training the model, it will turn "red", so that the user would know that there is an error happening in the backend.
- Superpixel: indicates whether the superpixel segmentation is ready in the backend.
- Prediction: indicates whether the there is a prediction layer ready for user to modify. When a new Model is ready (turning to green), the prediction will turn to yellow, generating the prediction layer suggested by the new model. A few seconds later, the Prediction turns green, indicating that the annotation suggestion of the new model is ready to use.
The Information Window contains meta data information for the image tile.
It shows project name, Image name and Image size. Besides, crop size can adjust the size of the selection rectangle and the Zoom factor can zoom in/out the annotation page. The user can view the image annotation percentage as well.
The user can show/hide the information Window in the status indicators bar
Annotation Window is where the user makes annotation by adaptively utilizing various annotation tool provide by QA. First, user selects a square region which is then loaded into the high magnification Annotation Window. Then, the user annotates positive regions in turquoise and non-target areas in fuchsia using common image markup tools. After annotating at least 3 patches, the user can train a DL classifier to generate annotation suggestions in white overlay. The user may then import the classifier’s suggestions into annotation window and edit, if needed, before accepting.
The followings are the introductions to each function in the annotation window toolbar. Each functions has a specialized hot key which helps users annotate.
- Freehand (A): in Freehand mode, user can delineate the boundary of the primitives by moving the mouse
- Superpixels (S): A superpixel is defined as a group of adjacent pixels sharing similar characteristics in terms of chromatic, texture, or deep learned feature values. QA provides Superpixels mode, which enabled one-click selection for a subset of primitives, notably improving annotation efficiency.
- Flood fill (D): The user can annotate a closed set of image by one click in Flood fill mode.
- Eraser (F): The user can de-annotated pixels with erasers of different sizes provided in Eraser mode.
- Import DL result into annotator (G): When the user has train a DL classifier to generate annotation suggestions white overlay, the user may then import the classifier’s suggestions into annotation window and edit.
- Upload complete annotation (H): Thw user can upload the annotated patches into training or testing dataset.
- Annotate positive region (Z), Annotate negative region (X), and Annotate Unknown region (C): Under theses modes, the user could annotate pixels with different labels. The user annotates regions of interest as positive region, marking non-primitive regions as negative region. QA also provides the option of marking an area as unknown region when users could not identify a region.
- Undo (V), Redo (B), and Reset (N): QA also keeps track of user's annotation operations. Therefore, the user could easily Redo or Undo any annotations if misannotating some areas. Also, user can remove all annotations in the annotation window by Reset button.
