Skip to content

Commit

Permalink
Merge pull request #26 from petebankhead/main
Browse files Browse the repository at this point in the history 
Updating stardist scripts, building instructions
  • Loading branch information
@petebankhead
petebankhead committed Aug 7, 2021
2 parents b5db07a + 2154fe3 commit e90b144
Show file tree
Hide file tree
Showing 2 changed files with 226 additions and 55 deletions.
250 changes: 195 additions & 55 deletions docs/advanced/stardist.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,36 +16,40 @@ This page describes how to start using StarDist 2D directly within QuPath as an
(And if you use it in combination with QuPath, be sure to :doc:`cite the QuPath paper too <../intro/citing>`!)


.. warning::

This is a provisional feature, based upon work hastily created for the NEUBIAS webinar on QuPath in April 2020, and released under mild duress from attendees.
You can find both the StarDist and QuPath webinars on the `NEUBIAS YouTube channel <https://www.youtube.com/c/neubias>`_.

If it already works for you, great!
If not, please be patient -- and be aware that it may change substantially as QuPath is developed further.


Getting the StarDist/TensorFlow extension
=========================================

StarDist requires TensorFlow + an extra QuPath extension, neither of which are currently included in the main QuPath distribution.
Getting the StarDist extension
==============================

To get them, you need to follow the instructions in the ReadMe at http://github.com/qupath/qupath-extension-tensorflow
To use StarDist with QuPath, you'll need to add an extension.
You can find it with installation instructions at http://github.com/qupath/qupath-extension-stardist


Getting pretrained models
=========================

You will need a StarDist model that has been pretrained on a wide range of images to identify nuclei that are (somewhat) similar to the nuclei in your images.
StarDist is powered by a deep learning model that is trained to detect specific kinds of nuclei.
Different models can be trained to detect nuclei within different kinds of image.

Currently, you can find several such models from the authors of StarDist at `<https://github.com/mpicbg-csbd/stardist-imagej/tree/master/src/main/resources/models/2D>`_.
You can download three example models originally trained by StarDist's authors from https://github.com/qupath/models/tree/main/stardist

The two we will consider here are:

* *dsb2018_heavy_augment.zip* - for fluorescence images (one detection channel)
* *he_heavy_augment.zip* - for brightfield H&E (three detection channels, RGB)
* *dsb2018_heavy_augment.pb* - for fluorescence images (one detection channel)
* *he_heavy_augment.pb* - for brightfield H&E (three detection channels, RGB)

You can download and unzip the models so they can be used with QuPath.

.. admonition:: No TensorFlow?

StarDist usually requires a powerful machine learning library called `TensorFlow <https://www.tensorflow.org>`_.
This is the main reason running StarDist in QuPath v0.2 was rather complicated.

As an alternative, it's now possible to run StarDist using `OpenCV <http://opencv.org>`_ instead.
This requires that the models are in a slightly different format (*.pb* files),
but because OpenCV is *already* included within QuPath it makes installation
much easier.

If you still want to use TensorFlow instead, see :ref:`stardist-advanced`.


.. tip::

Expand All @@ -72,24 +76,24 @@ Detecting nuclei
QuPath's current early StarDist support is **only available by scripting** and is rather limited in terms of reporting progress.
You can run it and... wait.

The following script applies the *he_heavy_augment* StarDist model to a brightfield H&E image:
The following script applies the *he_heavy_augment.pb* StarDist model to a brightfield H&E image:

.. code-block:: groovy
import qupath.tensorflow.stardist.StarDist2D
// Specify the model directory (you will need to change this!)
def pathModel = '/path/to/he_heavy_augment'
import qupath.ext.stardist.StarDist2D
def stardist = StarDist2D.builder(pathModel)
// Specify the model file (you will need to change this!)
var pathModel = '/path/to/he_heavy_augment.pb'
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5) // Prediction threshold
.normalizePercentiles(1, 99) // Percentile normalization
.pixelSize(0.5) // Resolution for detection
.build()
// Run detection for the selected objects
def imageData = getCurrentImageData()
def pathObjects = getSelectedObjects()
var imageData = getCurrentImageData()
var pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
return
Expand All @@ -109,28 +113,28 @@ The following script applies the *he_heavy_augment* StarDist model to a brightfi
You can copy the model directory outside QuPath (e.g. in Windows Explorer, Mac Finder) and then paste it in the script editor.
QuPath will paste in the path to the directory.

Be sure to keep the quotation marks around the path!
If you do this, make sure to add quotation marks around the path!


The following script applies the *dsb2018_heavy_augment* model to the DAPI channel of a fluorescence image:
The following script applies the *dsb2018_heavy_augment.pb* model to the DAPI channel of a fluorescence image:

.. code-block:: groovy
import qupath.tensorflow.stardist.StarDist2D
import qupath.ext.stardist.StarDist2D
// Specify the model directory (you will need to change this!)
def pathModel = '/path/to/dsb2018_heavy_augment'
// Specify the model file (you will need to change this!)
var pathModel = '/path/to/dsb2018_heavy_augment.pb'
def stardist = StarDist2D.builder(pathModel)
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5) // Probability (detection) threshold
.channels('DAPI') // Specify detection channel
.normalizePercentiles(1, 99) // Percentile normalization
.pixelSize(0.5) // Resolution for detection
.build()
// Run detection for the selected objects
def imageData = getCurrentImageData()
def pathObjects = getSelectedObjects()
var imageData = getCurrentImageData()
var pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
return
Expand All @@ -155,13 +159,13 @@ The following script applies the *dsb2018_heavy_augment* model to the DAPI chann
Customizing detection
=====================

The 'builder' used in the scripts above introduce many ways to easy customize how the StarDist detection is applied and the final outputs.
The 'builder' used in the scripts above introduce many ways to easily customize how the StarDist detection is applied and the final outputs.

One of the most important options is the ``pixelSize``, which is defined in terms of the pixel calibration units for the image (often µm, but sometimes 'pixels' if the calibration information is not available).
QuPath will automatically rescale the image as required before input to TensorFlow.
QuPath will automatically rescale the image as required before input to the model.

The 'best' value will depend upon the resolution of the images used to train the StarDist model.
You can experiment with different values.
You may need to experiment with different values.

.. tip::

Expand All @@ -175,21 +179,21 @@ Another customization is to include the probability estimates as measurements fo

.. code-block:: groovy
import qupath.tensorflow.stardist.StarDist2D
import qupath.ext.stardist.StarDist2D
// Specify the model directory (you will need to change this!)
def pathModel = '/path/to/he_heavy_augment'
// Specify the model file (you will need to change this!)
var pathModel = '/path/to/he_heavy_augment.pb'
def stardist = StarDist2D.builder(pathModel)
var stardist = StarDist2D.builder(pathModel)
.threshold(0.1) // Prediction threshold
.normalizePercentiles(1, 99) // Percentile normalization
.pixelSize(0.5) // Resolution for detection
.includeProbability(true) // Include prediction probability as measurement
.build()
// Run detection for the selected objects
def imageData = getCurrentImageData()
def pathObjects = getSelectedObjects()
var imageData = getCurrentImageData()
var pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
return
Expand Down Expand Up @@ -224,12 +228,12 @@ A similar distance-based expansion can also be used with StarDist, with optional

.. code-block:: groovy
import qupath.tensorflow.stardist.StarDist2D
import qupath.ext.stardist.StarDist2D
// Specify the model directory (you will need to change this!)
def pathModel = '/path/to/dsb2018_heavy_augment'
// Specify the model file (you will need to change this!)
var pathModel = '/path/to/dsb2018_heavy_augment.pb'
def stardist = StarDist2D.builder(pathModel)
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5) // Probability (detection) threshold
.channels('DAPI') // Select detection channel
.normalizePercentiles(1, 99) // Percentile normalization
Expand All @@ -242,8 +246,8 @@ A similar distance-based expansion can also be used with StarDist, with optional
.build()
// Run detection for the selected objects
def imageData = getCurrentImageData()
def pathObjects = getSelectedObjects()
var imageData = getCurrentImageData()
var pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
return
Expand All @@ -259,7 +263,7 @@ A similar distance-based expansion can also be used with StarDist, with optional

StarDist + QuPath cell detection and measurement

StarDist stops after nuclei have been detected; everything after that is 'pure QuPath'.
StarDist stops after nuclei have been detected; everything after that is specific to QuPath.

Nevertheless, the script above provides access to some additional features not (currently) available in QuPath's default cell detection.
These include:
Expand All @@ -286,15 +290,15 @@ These include:



More options
------------
More detection options
----------------------

There are more options available than those described here in detail.
Here is an example showing more of them:
There are even more options available than those described above.
Here is an example showing most of them:

.. code-block:: groovy
def stardist = StarDist2D.builder(pathModel)
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5) // Probability (detection) threshold
.channels('DAPI') // Select detection channel
.normalizePercentiles(1, 99) // Percentile normalization
Expand All @@ -309,13 +313,149 @@ Here is an example showing more of them:
.nThreads(4) // Limit the number of threads used for (possibly parallel) processing
.simplify(1) // Control how polygons are 'simplified' to remove unnecessary vertices
.doLog() // Use this to log a bit more information while running the script
.createAnnotations() // Generate annotation objects using StarDist, rather than detection objects
.constrainToParent(false) // Prevent nuclei/cells expanding beyond any parent annotations (default is true)
.classify("Tumor") // Automatically assign all created objects as 'Tumor'
.build()
.. tip::

In the event that you want to measure 'a ring around the nucleus' and avoid all the computational kerfuffle involved in preventing overlaps, use ``ignoreCellOverlaps(true)``.


.. _stardist-advanced:

Advanced options
================


Include preprocessing
---------------------

One of the most useful extra options to the builder is `preprocessing`, which makes it possible to perform some additional pixel operations before StarDist is used.

For example, rather than normalizing each image tile individually (as `normalizePercentiles` will do), we can normalize pixels using fixed values, for example with

.. code-block:: groovy
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5) // Prediction threshold
.preprocess( // Extra preprocessing steps, applied sequentially
ImageOps.Core.subtract(100),
ImageOps.Core.divide(100)
)
// .normalizePercentiles(1, 99) // Percentile normalization (turned off here)
.pixelSize(0.5) // Resolution for detection
.includeProbability(true) // Include prediction probability as measurement
.build()
Furthermore, we can use preprocessing to convert images to become more compatible with pretrained StarDist models.
For example, `dsb2018_heavy_augment.pb` works very well for fluorescence microscopy images, but only supports a single input channel.
However, by applying :ref:`color deconvolution<Separating stains>` as preprocessing we can convert a 3-channel brightfield image to a single-channel image that *looks* much more like a fluorescence image.
If needed, we can add extra things like filters to reduce noise as well.

.. code-block:: groovy
// Get current image - assumed to have color deconvolution stains set
var imageData = getCurrentImageData()
var stains = imageData.getColorDeconvolutionStains()
// Set everything up with single-channel fluorescence model
var pathModel = '/path/to/dsb2018_heavy_augment.pb'
var stardist = StarDist2D.builder(pathModel)
.preprocess( // Extra preprocessing steps, applied sequentially
ImageOps.Channels.deconvolve(stains),
ImageOps.Channels.extract(0),
ImageOps.Filters.median(2),
ImageOps.Core.divide(1.5)
)
.pixelSize(0.5)
.includeProbability(true)
.threshold(0.5)
.build()
.. tip::

These tricks were first described in `this forum post <https://forum.image.sc/t/stardist-extension/37696/6>`_.


Include classification
----------------------

StarDist can do more than simply detect nuclei: it can classify them as well.
There is an example notebook `here <https://github.com/stardist/stardist/blob/master/examples/other2D/multiclass.ipynb>`_ that describes how to train a model capable of both detection and classification.

The QuPath StarDist extension can use these models as well.
It only requires a change to input a map linking StarDist prediction labels to QuPath classifications.

.. code::
// Define model and resolution
var pathModel = "/path/to/classification/model.pb"
double pixelSize = 0.5
// Define a classification map, connecting prediction labels and classification names
var classifications = [
0: 'Background',
1: 'Stroma',
2: 'Tumor'
]
var stardist = StarDist2D.builder(pathModel)
.threshold(0.5)
.simplify(0)
.classificationNames(classifications) // Include names so that classifications can be applied
.keepClassifiedBackground(false) // Optionally keep detections that are classified as background (default is false)
.normalizePercentiles(1, 99)
.pixelSize(pixelSize)
.build()
// Run detection for the selected objects
var imageData = getCurrentImageData()
var pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
return
}
stardist.detectObjects(imageData, pathObjects)
println 'Done!'
.. tip::

QuPath will attempt to untangle where the classifications are in the outputs of the model.
For this to work, the number of rays predicted by StarDist should be greater than the number of distinct classifications.


Use TensorFlow
--------------

It is still possible to use StarDist with TensorFlow rather than OpenCV.
See the `QuPath TensorFlow Extension <http://github.com/qupath/qupath-extension-tensorflow>`_ for details and installation instructions.

You will need alternative pretrained models in TensorFlow's *SavedModel* format.
Unzipped examples from the `stardist-imagej repository <https://github.com/stardist/stardist-imagej/tree/master/src/main/resources/models/2D>`_ should work.

You will also need to give QuPath the path to the *folder* containing the model files in this case, e.g.

.. code-block:: groovy
var pathModel = '/path/to/dsb2018_heavy_augment' // A folder, not a file
Use CUDA
--------

If you have a recent NVIDIA graphics card, you *might* benefit from running StarDist using CUDA.
This is possible with both OpenCV and TensorFlow -- although I have found it easier to set up with OpenCV.

To do this, build QuPath from source as described in :ref:`building-gpu`.
If your GPU-friendly build of QuPath is able to detect CUDA support through OpenCV, it will be used automatically.



Differences from StarDist Fiji
==============================

Expand Down

0 comments on commit e90b144

Please sign in to comment.