[Feature] Native LLM-based contextual understanding for media

[chamolivishesh]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

Currently, Immich can identify people, objects, locations and other metadata independently. However, it cannot understand the overall context of the scene.

Example: video.mp4

Recognised people:

• Troy
• James

Detected objects:

• Park
• Swing
• Slide

Today, these are separate metadata points.

What I am requesting is something like:

Troy and James are playing in the park and discussing why swings are better than slides.

Proposed Solution:

Introduce an optional LLM-powered media understanding that generates natural language description such as the one mentioned above (for both preferably, but at least photos).

It should have

• People awareness (eg. Troy is on the swing while James is on the slide)
• Overall awareness: The park is next to the hospital.

Note: Could be implemented in Immich-ML (local lightweight model most likely).

Benefit - this would help searches in the Immich app. For example: "Troy James park" pulls up this example media.

In the future (if required), searches can also be routed through the model for better results.

Platform

• Server
• Web
• Mobile

[bo0tzz]

#7264