Ideas from The Water Institute exploration

[cronosnull]

Hi @vickysharma-prog!

First of all, amazing work! It's been really interesting to follow your progress. I think you've uncovered several of the same challenges we ran into while recovering annotations from the historical imagery, so I wanted to share a few things that significantly improved our workflow. Hopefully some of these ideas are useful for your project as well.

1. Use the high-resolution images

One of the biggest challenges we found with the screenshots is that the annotation symbology varies considerably across years. Color-based segmentation becomes difficult because tuning the thresholds for one year tends to overfit, while broadening the ranges improves recall at the expense of precision.

Fortunately, every annotated screenshot has a corresponding high-resolution image without the annotations.

After a conversation with @bw4sz, we started experimenting with aligning the two images (using SIFT feature matching followed by a homography transformation) and then subtracting the high-resolution image from the annotated screenshot. This leaves a much cleaner representation of the annotations than relying on color masks alone.

As an added benefit, the alignment also gives us a reliable way to generate masks that exclude title bars, scroll bars, and other UI elements before the detection stage.

2. Try the LAB color space

We initially used HSV color thresholds with reasonable success. However, switching to the LAB color space gave us noticeably better separation between the annotations and the background.

This made the image subtraction step much more robust and reduced the amount of manual threshold tuning required across different image sets.

3. Start from the annotations, not the colors

This ended up being the biggest improvement in our pipeline.

Instead of asking "Which pixels have the annotation colors?", we began asking "Which pixels are different between the annotated image and the corresponding clean image?"

Once the subtraction isolates the annotations, color becomes a secondary cue rather than the primary signal. This approach has generalized much better across years, different annotation styles, and varying image quality.

We haven't fully validated it across the entire dataset yet, but the results so far have been very encouraging.

Thanks again for making this work public! It's great to see other groups tackling the same problem, and I'd be happy to compare notes if it would be helpful.

[vickysharma-prog]

Hi @cronosnull! 👋

First off sorry for the late reply, I was in the middle of the matching rework and wanted to come back with something concrete. This is genuinely one of the most useful notes I've gotten, so thank you for taking the time. It's reassuring (and a little validating) to see another group hit the exact same walls.

A quick update on where things are, since a lot has changed:

What's working now: The pipeline reads each image's legend/dialog directly and builds a per-image marker→class map (shape + colour), so I'm no longer relying on a global colour convention that convention actually changes across images, which was biting me early on. On the classification side I just finished a rework and the numbers moved in the right direction: within-colour separability (same-colour, different-class markers, the hard case) went from 55→76% on one colony image and 56→83% on another, and total count error against the legend ground truth dropped on all four test images (e.g. one went 835→351). Dense colonies were undercounting because overlapping dots merge into one blob, so I added distance-transform cluster splitting to recover them, plus Tesseract OCR to read the class names off the dialog.

Funny enough LAB. I independently landed on the same conclusion you did. I switched colour anchoring to LAB space (per-image, calibrated from the dialog's own palette) precisely because HSV thresholds were overfitting per year and light-red vs dark-red markers wouldn't separate on hue alone LAB's value/a·b channels split them cleanly. So a strong +1 from my side on that suggestion.

The module I'm on right now is the aerial classification stage (src/classify.py) colour anchoring → colour-masked glyph → NCC shape matching → count-guided filtering. The two open problems I'm chewing on: (1) within-colour shape separation on very dense colonies, and (2) count-OCR reliability on ~10px digits (~60–65%).

And this is where your subtraction idea is huge. "Start from what's different between the annotated screenshot and the clean high-res, not from the colours" that reframes my hardest stage completely. Right now I'm removing background by colour-masking the glyph, which is fragile; aligning the clean image via SIFT+homography and subtracting would give me a genuinely clean marker with the background gone, and as a bonus a reliable UI mask for title/scroll bars. I already have SIFT/homography machinery in the notebook for the later coordinate-mapping stage, so I can repurpose it earlier. That's my next experiment I'll prototype it on one image and report back with a before/after.

Honestly, I'm confident I can get this to a solid, generalizable state, and I'm already planning to write it up as a research paper once the pipeline is complete recovering ML-ready annotations from baked-in point counts across ~18k historical screenshots feels like a genuinely reusable contribution, and notes like yours are exactly the kind of cross-validation that makes it stronger.

Would definitely love to keep comparing notes thanks again for sharing all of this so openly. 🙏