[Feature] Search by selected image crop / region

[johnny-salz]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

I would like Immich to support searching by a selected region of an image, conceptually similar to how text query works but with image part.

Use case: I open a photo, draw a rectangle around some object or detail, and search for visually similar assets based only on that crop. This is useful when the full image has too much unrelated context, but one object in it is the thing I want to find elsewhere.

I made a local PoC and it seems fairly easy to wire into the existing smart search flow:

In the photo viewer, add a “Search crop” action.
Let the user draw a rectangular crop over the displayed image.
Send normalized crop coordinates plus assetId to the server.
Server reads an existing preview/thumbnail for the asset, extracts the crop with sharp, and sends that temporary image to the existing machine learning clip.visual endpoint.
Use the returned embedding with the normal smart search vector query.
Return the usual search results.

I have a working local PoC if maintainers want to see the patch or discuss the API shape.

search:

crop results:

vs full asset results:

Platform

• Server
• Web
• Mobile

[timonrieger]

cool idea. just wondering: did the "view similar photos" feature not satisfy your result expectations? I suppose specific details in a "bird-eye" shot are hard to find matches on, right?

[johnny-salz]

First of all, "view similar photos" is awesome and I probably choose a bad example (just took a screenshot of what was already opened at the moment and it turned out to be a hassle to redo, sorry). I initially started to want that feature when I thought "hmm, I wonder if I had more of that thing in photos" with that thing being a car or something on a very busy photo, similar images search returns matches by similarity for everything on the photo and desired object may be long down the list.

It can get confusing to a model when you crop too small, yeah, but otherwise it finds selected things on other busy photos decently well -- images containing the object are still closer than those without it even if not that close in absolute sense.

[johnny-salz]

I didn't want to include a kitchen sink in a single feature request, but the general idea is to make something close to Google Lens in browser but for my own photo library and I already somewhat got used to drawing a crop and getting results there. Also, pasting an image as a query would be great next thing to do.

[johnny-salz]

Made another comparison:

text search: IKEA. results show everything ikea'ish

show similar: image with IKEA banner. results show images of the roads, with ikea'ish banners

search crop of IKEA banner. results show decent selection of IKEA banners from all angles

[timonrieger]

thanks for the followup. Confirms my assumption. Great example, too 😄 . I'd suggest proposing this feature shortly in #contributing on Discord.