Tiled rendering for large COG (geotiff) files, with an optional second image for before/after comparison

[jo-chemla]

What to build

Note I created historical-satellite, an oss app based on maplibre which does offer the below suggested feature-set of split-mode, see docs for screens: Off, Overlay or Side-by-Side (2x1 default, but can go up to 4x2). Adding such geospatial raster time-series annotation capabilities to labelme would be tremendously helpful to change detection/classification by experts and annotators.

This request can probably be split 3-fold:

• the correct reading of georeferencing metadata from geotiff or geo-companion sidecars like jgw for jpg, jpeg, for an eventual alignment of 2 rasters with not necessarily identical footprint/extent or resolution.
• the viewport-aware streaming of the massive image overview/tiles within the hierarchy of tiled raster image files like COG - probably easier to scope to COG only for now.
• the tagging of drawn features with the before / after imagery ids and eventually dates, so the obersavtion that analysts and annotators do is both the footprint (or point) of the change, eg destruction, construction etc, and the imagery pair on which this change is observed.

#2418 describes the cost of failing on a large GeoTIFF and PR #2750 by @tongwentao fixes that by rejecting oversized images from the header before decoding. You noted in 2418 that tiled or pyramid rendering is a separate, larger feature request. This is that request: after 2750, a 37296 x 49319 GeoTIFF fails in milliseconds instead of minutes, but it still cannot be opened.

Additional context

Use case: land-use change monitoring or damage assessment on dated satellite imagery.

Below is an inspection of that request by an agent in case it's useful

Agent inspection

1. Tiled display. For TIFFs with internal tiles and overviews, which is what a COG (Cloud Optimized GeoTIFF) is, read only the overview level and tiles needed for the current viewport instead of decoding the whole image into a QImage. tifffile already reads tiles and overview pages lazily, so this stays within current dependencies. The ImageTooLargeError path introduced in 2750 is the natural branch point: when the image is too large for a single QImage but has internal tiles or overviews, open it through the tiled reader instead of raising. Shapes remain in full-resolution pixel coordinates so the JSON format does not change.
2. Remote COG by URL. Open https:// and s3:// COG URLs with HTTP range requests, same code path as above. Useful for teams whose imagery already lives in object storage.
3. Second image overlay. An optional "compare image", loaded through the same path, with a hotkey to flicker and a slider for opacity. When both images are GeoTIFFs, align them by their geotransforms, so two acquisitions with different footprints or resolutions can be compared without pre-warping to a common grid. Non-overlapping areas simply show one image. When no georeference is present, fall back to requiring the same pixel size. Polygons stay on the primary image and a compare_image_path field in the JSON records which image was used. Reading the geotransform is the same step GeoJSON export for annotations #2209 needs for GeoJSON export.

Drafted with AI assistance from Claude Fable 5.1

[tongwentao]

Thanks for bringing this up and for referencing PR #2750!

I agree that tiled/pyramid rendering would be a valuable next step for handling large GeoTIFF files in Labelme.

PR #2750 focuses on avoiding unnecessary decoding when an image exceeds Qt's size limits. Supporting tiled rendering would be a natural follow-up, although it would require a broader change to the image-loading and rendering pipeline.

I think starting with local tiled GeoTIFF/COG support, while keeping annotations in full-resolution image coordinates, would be a reasonable first step.

I'd be interested in exploring this further, depending on the maintainers' thoughts on the overall approach.

[wkentaro]

My general rule (for myself) is that labelme core stays a general 2D image annotator. Features specific to one domain should live in plugins (which don't exist yet #2155 ). Otherwise there is no clear line, and the next requests are point clouds and 3D.

Under that rule, this request splits in two.

Tiled rendering of large images goes in core. Pathology slides, microscopy, and large scans hit the same limit as GeoTIFF, so it isn't geo-specific. I think it should be kept up to:

• local files only, no https:// or s3:// URLs
• tiled TIFF/COG files with internal overviews
• shapes stay in full-resolution pixel coordinates
• no change to the JSON format
• small PRs that can be reviewed one at a time

The geo-specific parts go in a plugin. That covers reading georeferencing (geotransforms, .jgw sidecars), aligning two images by their geo coordinates, opening remote COGs, and tagging shapes with before/after imagery IDs and dates. labelme has no plugin system yet. I'm designing one in #2155, starting with exporters, importers, and AI backends, followed by an image source hook. A geo plugin would build on that hook and on core tiled rendering. Once the API exists, a geo plugin would be a good real-world test of it.

[jo-chemla]

Thanks for your comment, understood for the fact LabelMe will stay a generic 2D image annotator.
Tiled rendering indeed I'm also all-in for it, not only massive tiled geospatial formats like COGs, but also DZI, SZI (for single file tiled pyramids) or tile directories (often several 100k tile files).

https:// sometimes can be used for eg a reference image, to do change detection on a recent satellite imagery vs reference google earth for example. But this remote source mechanism might indeed be deferred to later. Understood for the fact also drawn feature coords stay in pixel units.

And great that you have modularity as a goal at some point. You spotted all geo-specifics input metadata, input filetypes, task-specific tags or view-syncing. I'll track that generic plugin thread indeed, and thanks for all the great work you put behind labelme!