QuPath Annotation Transfer

[abadgerw]

I have annotations that were made on one slide in QuPath that I would like to transfer onto a serial section. I've tried other methods like QuPath Interactive Image Alignment and Warpy but was told Valis performed better registration. I was reading the documentation and see that Valis is typically used to create a single image from 2 serial sections. Could it be used for my purpose of just tabulating registration parameters in order to transfer annotations to the right place?

I'm not a very savvy coder so any assistance you can provide would be greatly appreciated. Thanks!

[cdgatenbee]

Hi @abadgerw,
Yes, Valis should be able to transfer your annotations to another serial section. You would register the slides, read in your annotation points (xy coordinates, in pixel units), and then use the corresponding Slide object to warp the annotated points. This would look something like this:

from valis import registration

# Perform registration
registrar = registration.Valis(slide_src_dir, results_dst_dir)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

# Read in your annotations here and name as annotation_pt_xy. 
# Be sure they are in pixel units, not physical units. See below on how convert them to pixel units if the annotations are in physical units 
# annotation_img_f is the name of the slide the annotations came from
# target_img_f  is the name of the slide you want to transfer the annotations to 

annotation_source_slide = registrar.get_slide(annotation_img_f)
target_slide = registrar.get_slide(target_img_f)

annotations_on_target_slide_xy = annotation_source_slide.warp_xy_from_to(annotation_pt_xy, target_slide)

However, if your annotations are in physical units you'll need to first convert them to pixel units. This can be done using each Slide's metadata:

# Convert to pixel units. 
# annotation_source_slide.units will show you the unit, e.g. µm
# annotation_source_slide.resolution will show you the number of physical units per pixel, i.e. 0.5 would be 0.5 µm/pixel
pt_px = annotation_pt_xy_um/annotation_source_slide.resolution

# Warp points
annotations_on_target_slide_xy_px = annotation_source_slide.warp_xy_from_to(pt_xy, target_slide)

#Convert back to physical units
annotations_on_target_slide_xy_um = target_slide.resolution*annotations_on_target_slide_xy_px

You may also find the the 2nd example in the "Warping Points" section of the README helpful https://valis.readthedocs.io/en/latest/#warping-points. The main difference is that in that example the points are being warped to the registered image space to extract a ROI, but it sounds like you may instead want to transfer the points to their position on a different unwarped image. Hopefully the above example will work for you, but if not please let me know.

Best,
-Chandler

[abadgerw]

Thank you so much! I'll try this, see how it goes, and report back.

One potentially naive question before starting: What format would be best to export the image with annotations as? Should I export a full labeled image with annotations or labeled tiles? They annotations are made in QuPath and they have lots of options to choose from: https://qupath.readthedocs.io/en/stable/docs/advanced/exporting_annotations.html

[cdgatenbee]

Hi @abadgerw,
Sorry, I guess I had assumed that the annotations were Shape vertices that could be warped and then used to draw the new annotations on a different image (e.g. GeoJSON). However, it sounds like you'd like to warp a binary or labeled image instead. That can be done too, but it would just be different than the example. I'd be happy to help put together some example code on how to do this, but could you tell me a little more about your annotations? How large is the annotation image, and can QuPath save the full image? Or would it be necessary to save only individual ROI? Using the full size labeled image would be the most straight forward, since valis can read the annotation image from file, warp it using the registration parameters, and then save it as a new image. If that's not possible you could also warp the individual ROI, but it would probably be a little more complicated. Either way, let me know and I'll try to put an example together.

Best,
-Chandler

[abadgerw]

Thanks, Chandler! It looks like I can export the annotations from QuPath as GeoJSON using the QuPath code on this webpage (https://qupath.readthedocs.io/en/stable/docs/advanced/exporting_annotations.html).

Which approach do you think would work best: exporting the annotations and then warping those onto a serial section or exporting a full labeled image and warping that to place the annotations onto the new images? Since the serial sections are commonly rotated, would the latter be more accurate since features from the serial sections could be used to register the images before annotation transfer?

Attached is an example image with annotations:

The files are .svs images and the size is upwards of 500mb typically. It looks like QuPath can export the full image and annotations without downsampling as an ome.tif using the script on the following webpage (https://qupath.readthedocs.io/en/stable/docs/advanced/exporting_annotations.html). Here is the output of that: https://drive.google.com/file/d/1Lxf1szuTj_R9ksN24SJQyEZfHB370QC_/view?usp=sharing

Happy to have a chat via teleconference if that would be helpful. Thanks again for all your help!

[cdgatenbee]

Hi @abadgerw,
I think maybe the cleanest and most memory efficient way to do this would be to warp the GeoJSON points as described above, save them, and then load into QuPath. That way the only extra file you'll need to save is the warped annotation vertices, which probably wouldn't take up too much space. Plus, the image you are transferring the annotations to won't have to be warped at all.

Alternatively, if you'd like to use the labeled image, you can specify that the image you're using for annotations as the "reference_img_f" (i.e. registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=annotation_img_f ) then you can just warp and save the image(s) you want to transfer the annotation to. Since it's been aligned to the annotated image the labels you already have should all line up without the need to warp the labeled image itself. So it would just look like this

from valis import registration

# Perform registration
# annotation_img_f is the name of the slide the annotations came from
registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=annotation_img_f)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

# target_img_f  is the name of the slide you want to transfer the annotations to 
# dst_f  is the file name of warped slide. Since it's been aligned to the annotated image the labels should 

target_slide = registrar.get_slide(target_img_f)
target_slide.warp_and_save_slide(dst_f)

# Load dst_f in QuPath with the original labeled image

The downside here is that the warped slide could have some deformations from the non-rigid registration, but hopefully not bad.

A third option would be to warp the labeled image onto the unwarped target image, but unfortunately I don't have something like that coded up just yet :( I could see it being useful in cases like this though, so if it's something you think you would use let me know and I'll try to put something together.

Best,
-Chandler

[abadgerw]

Thank! I think I may be confused on the differences and potential pros/cons. As background, originally, I have been using the QuPath interactive image alignment feature. I would first rotate the annotated image so that it approximates the target image and then use the affine transformation to get a better match. I'd extract the values and input it into the following script:

Align Images and Transfer Annotations.txt

However, I was hoping to get better registration between the image with the annotations and the target slide as the annotations commonly are not exactly in the place I'd like. Therefore, I figured something that is a bit more advanced to the affine transformation would do the trick.

Therefore, would warping the labeled image onto the unwarped image (option 3) be the most similar to the approach I have been working with? Is there any downside to just warping the annotation vertices (option 1) rather than the labeled image itself (option 3)?

My apologies for all the naive questions. Thanks for all your help and support with this! I've just gotten VALIS downloaded onto our cluster computer and will be able to have a go and see how it all shakes out.

[cdgatenbee]

Hi @abadgerw, glad to hear you were able to get valis installed on the cluster but sorry for not more clearly articulating the pros and cons of each approach. But, I think warping the either the annotation vertices or the labeled image would be most similar to your current approach. I can't think of what the downsides would be of just warping the vertices, except possibly the polygon becoming "invalid". But it's hard to know beforehand, and I think it would be fixable, using something like Shapley to re-order the vertices (if necessary). Or QuPath may have some way to handle it? Sorry, I just haven't worked with QuPath's annotations before, but I think i may put together an example annotation and try this out to see how it goes.

If you'd like to warp the labeled image and apply it to the unwarped target image, you could take a slightly different approach. Instead of having "reference_img_f" be the image associated with the annotations, you could make it the target image. That way you would be aligning your annotated image to the target image, and then apply those same transformations to the labeled image. This would work ok if it's just a pair of images, but if you have more than two to align you would have to stick to doing pairwise registrations from your annotated image to the target image. At least until I've put together code that can warp one image onto another unwarped image. Anyhow, that would look like this:

from valis import registration, slide_io

# Perform registration
# target_img_f  is the name of the slide you want to transfer the annotations to, and it will not get warped
registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=target_img_f)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

# annotation_img_f is the name of the slide the annotations are based on
# labeled_img_f is the name of the labeled slide, e.g. MS018C_Iba1-labels.ome.tif
annotation_slide = registrar.get_slide(annotation_img_f)
warped_labeled_img = annotation_slide.warp_slide(src_f=labeled_img_f)

# Create some basic metadata. Only really necessary if you want to include the physical units and/or channel names
bf_dtype = slide_io.vips2bf_dtype(warped_labeled_img.format)
xyzct = slide_io.get_shape_xyzct((warped_labeled_img.width, warped_labeled_img.height), warped_labeled_img.bands)
px_phys_size = annotation_slide.reader.metadata.pixel_physical_size_xyu
# Note that you can add channel names if your labeled image has a different classification in each channel
new_ome = slide_io.create_ome_xml(xyzct, bf_dtype, is_rgb=False, pixel_physical_size_xyu=px_phys_size)
ome_xml = new_ome.to_xml()

# dst_f  is the file name of warped labeled image.
slide_io.save_ome_tiff(warped_labeled_img, dst_f=dst_f, ome_xml=ome_xml)

After that you should hopefully be able to load the registered labeled image into QuPath and thereby apply the annotations to your unwarped target image.

Please let me know how it goes, and I'm more than happy to help if you run into some issues or have other questions.

Best,
-Chandler

[cdgatenbee]

Hi @abadgerw,
I thought you might like to know that valis now offers the ability to warp one image onto another, which you can now use to transfer your annotations as labeled images to load in QuPath. Here's some example code, which is also included in the examples folder.

"""
Example showing how to warp an annotation image to go onto another image.

`slide_src_dir` is where the slides to be registered are located
`results_dst_dir` is where to save the results
`labeled_img_f` is the filename of actual annotation image
"""

import os
import pathlib
from valis import registration, slide_io, viz, warp_tools



# Perform registration. Can optionally set the reference image to be the same as the one the annotations are based on (i.e. `reference_img_f`)
registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=reference_img_f)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

# Load labeled image saved as `labeled_img_f`
labeled_img_reader_cls = slide_io.get_slide_reader(labeled_img_f)
labeled_img_reader = labeled_img_reader_cls(labeled_img_f)
labeled_img = labeled_img_reader.slide2vips(level=0)

# Have reference slide warp the labeled image onto the others and save the results
reference_slide = registrar.get_slide(reference_img_f)

annotations_dst_dir = os.path.join(registrar.dst_dir, "annotations")
pathlib.Path(annotations_dst_dir).mkdir(exist_ok=True, parents=True)

# Create and save an annotation image for each slide
for slide_obj in registrar.slide_dict.values():
    if slide_obj == reference_slide:
        continue

    # Warp the labeled image from the annotation image to this different slide #
    transferred_annotation_img = reference_slide.warp_img_from_to(labeled_img,
                                    to_slide_obj=slide_obj,
                                    interp_method="nearest")

    # Create image metadata. Note that you could add channel names if your labeled image has a different classification in each channel
    bf_dtype = slide_io.vips2bf_dtype(transferred_annotation_img.format)
    xyzct = slide_io.get_shape_xyzct((transferred_annotation_img.width, transferred_annotation_img.height), transferred_annotation_img.bands)
    px_phys_size = reference_slide.reader.metadata.pixel_physical_size_xyu
    new_ome = slide_io.create_ome_xml(xyzct, bf_dtype, is_rgb=False, pixel_physical_size_xyu=px_phys_size)
    ome_xml = new_ome.to_xml()

    # Save the labeled image as an ome.tiff #
    dst_f = os.path.join(annotations_dst_dir, f"{slide_obj.name}_annotations.ome.tiff")
    slide_io.save_ome_tiff(transferred_annotation_img, dst_f=dst_f, ome_xml=ome_xml)

    # Save a thumbnail with the annotation on top of the image #
    small_annotation_img = warp_tools.resize_img(transferred_annotation_img, slide_obj.image.shape[0:2])
    small_annotation_img_np = warp_tools.vips2numpy(small_annotation_img)
    small_img_with_annotation = viz.draw_outline(slide_obj.image, small_annotation_img_np)
    thumbnail_dst_f = os.path.join(annotations_dst_dir, f"{slide_obj.name}_annotated.png")
    warp_tools.save_img(thumbnail_dst_f, small_img_with_annotation)

Here are examples of what the thumbnails might look like:

I haven't yet tried warping the annotation vertices output by QuPath (still plan to though), but hopefully this will be helpful when you're trying to transfer annotations. If you have any questions, please don't hesitate to ask.

Best,
-Chandler

[abadgerw]

Chandler,

Thank you so much! I'm currently out of office for 2 weeks but will delve into applying these fantastic tools upon my return and will report back.

Thanks again for all your help and support with this!

[yluo0512]

@cdgatenbee
Thanks for this great package! I been exploring valis, and have couple questions.

1. can we do pair-wise image registration in valis? For example, now I have 20 H&E images with annotation, and I want to batch align these H&E images to their paired Masson's trichrome images, and transfer these H&E's annotations to Masson't trichrome. Is that possible?

[cdgatenbee]

Hi @yluo0512,

Glad to hear you've been enjoying valis :) To answer your question, yes, it is possible to do pairwise alignments. There are two approaches you can use:

1. For each sample, put each pair of images in a single folder and pass the name of the folder to the src_dir argument when you create the Valis object. Just be sure that the directory contains only the images you want to register, as valis will try align everything that is in there. Or,
2. Pass a list of the pair of images' filenames to img_list and set src_dir to your working directory when you create the Valis object. In this scenario, valis will only try to align the images in img_list, and since that list contains the full path to each image, the images can be located anywhere (e.g. your H&E image in a folder with 19 other H&E images). Also, src_dir isn't really used in that case, so it doesn't have to be your working directory, it can actually be whatever you'd like.

Since you'll be transferring the annotations from the H&E images, you might want to set reference_img_f to be the H&E filename, and the align_to_reference=True when you initialize the Valis object (not required for this to work, though). Once complete, you should be able to warp and save the annotation image as shown in the example above.

I hope that helps, and if you have any more questions, please don't hesitate to ask.

Best,
Chandler

[yluo0512]

Hi @yluo0512,

Glad to hear you've been enjoying valis :) To answer your question, yes, it is possible to do pairwise alignments. There are two approaches you can use:

1. For each sample, put each pair of images in a single folder and pass the name of the folder to the src_dir argument when you create the Valis object. Just be sure that the directory contains only the images you want to register, as valis will try align everything that is in there. Or,
2. Pass a list of the pair of images' filenames to img_list and set src_dir to your working directory when you create the Valis object. In this scenario, valis will only try to align the images in img_list, and since that list contains the full path to each image, the images can be located anywhere (e.g. your H&E image in a folder with 19 other H&E images). Also, src_dir isn't really used in that case, so it doesn't have to be your working directory, it can actually be whatever you'd like.

Since you'll be transferring the annotations from the H&E images, you might want to set reference_img_f to be the H&E filename, and the align_to_reference=True when you initialize the Valis object (not required for this to work, though). Once complete, you should be able to warp and save the annotation image as shown in the example above.

I hope that helps, and if you have any more questions, please don't hesitate to ask.

Best, Chandler

@cdgatenbee
Hi Chandler,

Thank you very much for your response! I will give it a go based on your suggestions.
The problem is that each of image has 3 sections. I found an example image online https://global.discourse-cdn.com/business4/uploads/imagej/original/3X/8/6/869d5e6f3bd5b99796f050c6214c87c05cfbc4e8.jpeg
Since each image has multiple sections, I often found the image alignment might work well for one section but might not be for others. I think about splitting H&E images into 3 part, and align them with the according paired Masson's trichrome section. But because the annotation was done based on each individual image (so one image(contains 3 section) has one geojson file), so I couldn't actually split the geojson file into three part. Therefore, even I split the image into three parts, I can't transfer the annotation from one to other since I do not have according section's geojson file.

I would love to hear your opinion on this! Or does any Valis's functionality could help with this problem? Thank you!

[cdgatenbee]

Hi @yluo0512 ,
I see, that is a bit more complicated than I had realized. I think something you could do is if you know the point at which you cut each image, you could subtract those values from the warped coordinates in the geojson file. For example, say you are transferring the annotations for the 2nd (middle) section, and sliced the H&E image image at x= 100. After registering the images, you first subtract 100 from each x value in geojson file, and then warp the coordinates. The idea is basically that you will be shifting the annotations to their position in the cut image before warping. Do you think something like this would work?

Best,
Chandler

[abadgerw]

@cdgatenbee my apologies for the delay in getting back to trying these options out. Valis has been downloaded onto our cluster computer and I've attempted both the warping of annotation points and warping of the annotated image scripts you highlighted above and have run into a couple of issues.

For warping the annotation points, I ran the following:

from valis import registration
slide_src_dir = "/hpcdata/li_mdis/li_mdis_data/waldmanad/Valis/GeoJSON"
results_dst_dir = "./slide_registration_example_GeoJSON"
registered_slide_dst_dir = "./slide_registration_example_GeoJSON/registered_slides"

registrar = registration.Valis(slide_src_dir, results_dst_dir)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

annotation_pt_xy = "MS040_SC_T.geojson"
annotation_img_f = "MS040_SC_T_TMEM119_40X.svs"
target_img_f = "MS040_SC_T_PLP_40X.svs"

annotation_source_slide = registrar.get_slide(annotation_img_f)
target_slide = registrar.get_slide(target_img_f)

annotations_on_target_slide_xy = annotation_source_slide.warp_xy_from_to(annotation_pt_xy, target_slide)

After the last command, I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sysapps/cluster/software/Anaconda3/2022.05/envs/valis-1.0.0rc12/lib/python3.10/site-packages/valis/registration.py", line 1135, in warp_xy_from_to
    warp_tools.warp_xy_from_to(xy=xy,
  File "/sysapps/cluster/software/Anaconda3/2022.05/envs/valis-1.0.0rc12/lib/python3.10/site-packages/valis/warp_tools.py", line 2330, in warp_xy_from_to
    xy_in_reg_space = warp_xy(xy, M=from_M,
  File "/sysapps/cluster/software/Anaconda3/2022.05/envs/valis-1.0.0rc12/lib/python3.10/site-packages/valis/warp_tools.py", line 2176, in warp_xy
    warped_xy = _warp_xy_numpy(xy, M, transformation_src_shape_rc=transformation_src_shape_rc,
  File "/sysapps/cluster/software/Anaconda3/2022.05/envs/valis-1.0.0rc12/lib/python3.10/site-packages/valis/warp_tools.py", line 2074, in _warp_xy_numpy
    in_src_xy = xy/src_sxy
TypeError: ufunc 'divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

For warping the annotated image, I ran the following:

import os
import pathlib
from valis import registration, slide_io, viz, warp_tools

slide_src_dir = "/hpcdata/li_mdis/li_mdis_data/waldmanad/Valis/OME"
results_dst_dir = "./slide_registration_example_OME"
registered_slide_dst_dir = "./slide_registration_example_OME/registered_slides"

labeled_img_f = "MS040_SC_T_TMEM119_40X-labels-multi.ome.tif"
reference_img_f = "MS040_SC_T_TMEM119_40X.svs"

registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=reference_img_f, align_to_reference=True)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

labeled_img_reader_cls = slide_io.get_slide_reader(labeled_img_f)
labeled_img_reader = labeled_img_reader_cls(labeled_img_f)
labeled_img = labeled_img_reader.slide2vips(level=0)

annotations_dst_dir = os.path.join(registrar.dst_dir, "annotations")
pathlib.Path(annotations_dst_dir).mkdir(exist_ok=True, parents=True)

for slide_obj in registrar.slide_dict.values():
    if slide_obj == reference_slide:
        continue

    transferred_annotation_img = reference_slide.warp_img_from_to(labeled_img,
                                    to_slide_obj=slide_obj,
                                    interp_method="nearest")

    bf_dtype = slide_io.vips2bf_dtype(transferred_annotation_img.format)
    xyzct = slide_io.get_shape_xyzct((transferred_annotation_img.width, transferred_annotation_img.height), transferred_annotation_img.bands)
    px_phys_size = reference_slide.reader.metadata.pixel_physical_size_xyu
    new_ome = slide_io.create_ome_xml(xyzct, bf_dtype, is_rgb=False, pixel_physical_size_xyu=px_phys_size)
    ome_xml = new_ome.to_xml()

    dst_f = os.path.join(annotations_dst_dir, f"{slide_obj.name}_annotations.ome.tiff")
    slide_io.save_ome_tiff(transferred_annotation_img, dst_f=dst_f, ome_xml=ome_xml)

    small_annotation_img = warp_tools.resize_img(transferred_annotation_img, slide_obj.image.shape[0:2])
    small_annotation_img_np = warp_tools.vips2numpy(small_annotation_img)
    small_img_with_annotation = viz.draw_outline(slide_obj.image, small_annotation_img_np)
    thumbnail_dst_f = os.path.join(annotations_dst_dir, f"{slide_obj.name}_annotated.png")
    warp_tools.save_img(thumbnail_dst_f, small_img_with_annotation)

For the last set of commands starting with the "for" my terminal stops responding right away and needs to be force quit.

If I remove the following bit of code it runs but very slowly (it is currently at 4.7% after 47 minutes of running which makes me think I may be doing something wrong?):

    small_annotation_img = warp_tools.resize_img(transferred_annotation_img, slide_obj.image.shape[0:2])
    small_annotation_img_np = warp_tools.vips2numpy(small_annotation_img)
    small_img_with_annotation = viz.draw_outline(slide_obj.image, small_annotation_img_np)
    thumbnail_dst_f = os.path.join(annotations_dst_dir, f"{slide_obj.name}_annotated.png")
    warp_tools.save_img(thumbnail_dst_f, small_img_with_annotation)

Edit: It ultimately stopped responding at 6.2% after 62 minutes and led me to force quit.

Here is a link with the files I was using in case that is helpful: https://drive.google.com/drive/folders/1tND01fV9JDP7oPQB31Mz5zxCCyvSKFas?usp=sharing

Thanks for all your help!