[Feature/Question] Add rich text annotations to media files, searchable with native search

[Enelar]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

While immich NN does great job for basic semantic search with NN and OCR, it have known limitations: for videos only first frame is recognized, and it does not perform voice recognition. I do not believe that immich should be bloated and do everything, but since it has API it should be easy to write another service that enriching resources.
In my file storage I already have it: when new video added, whisper automatically transcribes it. But when I tried to connect it to immich I could not find which api call / field I should use. I've tried to push this text to description, but it seems that search unable to find media unless there is an "description" mode selected and words specified there exactly (no typos, exact order, exact match).

I would really prefer if it's possible to extend video vectors with text that I've supplied, but I'm failing to understand how to do it.
Same with using more than one frame for video, while it's technically easy with ffmpeg and NN, I do not understand how to add more vectors for context search.

Please help me find documentation on how to add plugins to immich?

Platform

• Server
• Web
• Mobile

[bo0tzz]

There is no support for plugins or way to "extend video vectors" or such

[Enelar]

Thanks for the quick reply!

How would you approach this?
Maybe I should make a proxy in front of machine learning and redirect search requests to my elastic search/vector search?