[Feature] Add OCR

[chriexpe]

Edit: yay thank you @mertalev and especially @flipped-1121!

The feature

OCR is the only thing that keeps bringing me back to Google Photos, as I have A TON of screenshots, memes and documents so it's really handy to find exactly what I want.

I've seen somewhere here someone using this repo for OCR PaddlePaddle/PaddleOCR, hopefully this might help you integrate into Immich!

Thank you guys for this amazing project!

Platform

• Server
• Web
• Mobile

[klaus1k]

I really like this feature on GPhotos!
It seems to me that it already works with CLIP? If I search for text in my pictures in Immich it shows the pictures with this text in them.

[chriexpe]

I think that might be just a coincidence, in my case most of the time the results are random

[NicholasFlamy]

I think that might be just a coincidence, in my case most of the time the results are random

It's CLIP which has quasi OCR capabilities. It can sometimes match text when you search. It works better with better models but even then it's still not true OCR.

[Fireflaker]

In my experience - CLIP OCR barely work at all. Screenshots with arbitrary text do not return in smart search.

[NicholasFlamy]

@Fireflaker I said sometimes. Also, try a better CLIP model.

[a45s67]

I agree with @chriexpe.

I have tons of meme pictures which are full of text. After trying the immich search engine for days, I found it is not easy to find out the image that has specific text contents.

For instance, although CLIP learns a capable OCR system, when evaluated on handwritten digits from the MNIST dataset, zero-shot CLIP only achieves 88% accuracy, well below the 99.75% of humans on the dataset.

According to the description in CLIP, this model is not good at detecting the text in images. So I think if the search engine integrate with some OCR models, it might fill the lack of CLIP, and have a much perfect search results. There is a closed PR which could be a good implementation reference (#1200).

For example, here are some images I can't find when given corresponding text:

	Images has corresponding text	Search result
Text: My Confidence
Text: XD

[NicholasFlamy]

Try a better model. Obviously this isn't a solution, but it will give better results.

[eagle470]

This is also a feature I use heavily. It would be supremely useful.

[eagle470]

I'll add clarity, I use it on my iPhone with apple photos. I have moved all images with text to immich similar to what was mentioned above.

I would do everything but that is currently not recommended by the developers.

[aisbergde]

For me OCR is major feature and part of my document archive workflow. My scanned documents are in G drive. In G drive, not in G Fotos. But screenshots are in G Photos and are scanned. So maybe I should even better think about how to store screenshots not in G Photos but in G drive.

Then OCR result in Photos would be much more clean. Anyway OCR is important.

[2001Y]

私もこの意見に非常に同意します。
書類はファイルストレージにあるが、スクリーンショットは画像管理アプリにあるし、お店の看板や車のナンバーも写真管理アプリにあります。

OCRは機械学習による画像認識より軽量だと思うので是非実装してほしいです。

[sgofferj]

For that use case, maybe paperless-ngx would be a better alternative? You would have all "documents" in one app. It also comes with a mobile app which supports images being shared to.

[aviv926]

At least until it is officially added here is something you can get right now with CLIP model:

the CLIP model that exists in immich supports OCR in a built-in way, but not at a level of height accuracy

Some models are trained on more data and some on less with different levels of accuracy.
CLIP does not set itself the primary goal of recognizing text in an image, but it is a type of option that exists, but it is certainly not something that can be relied on as OCR.

If you still want OCR (although I wouldn't recommend it for CLIP)
I would look for a balance between performance and results in the SST2 database

A model like laion2b_s32b_b79k gets good performance in Average perf. on 38 datasets and also in SST2 so it can suit you.

For comparison the basic model that Immich is set to work with (ViT-B-32__openai) got only 0.5865 in SST2 compared to laion2b_s32b_b79k who got 0.6392 in SST2.
Although again, I wouldn't expect a high level of OCR from a CLIP model.

Important note: different models require different levels of computing power (FLOPs) you should check if this is something your system is capable of before choosing to change models.

Another note is that it is necessary to check whether the model exists in the Immich database
You can check it here:
https://huggingface.co/collections/immich-app/clip-654eaefb077425890874cd07

Sources regarding tests in SST2 can be found here:
https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv#L84

[aisbergde]

A lot of resources are used to apply models and to get the results. Switching models currently means loosing all this effort.

Another idea would be to not delete the existing results but to store them "on hold" for later reusing. Nothing needs to be changed in the existing implementation, but only a save and restore part could be added. This can be done on the database level:

• additional backup tables should be created
• procedures to backup and restore the tagging results.

These are simple database operations.

[NicholasFlamy]

You could just take regular backups. Like right now with the current immich you could take regular backups to accomplish some of these goals by restoring the backups to compare.

[aisbergde]

For me it would be much easier to add some tables and procedures to the existing database, because databases are my daily business. But I don't know what happens, when I add my own database objects (tables, views, procedures) in case of updates. Would my additional objects sustain? Normally they should.

The question: is it worth the effort, when these requirements are not really requested? If it is not supported via the GUI, the usage requires database knowledge. Currently the one existing result set is OK for me. I am happy with the face tagging and I don't use other tagging. For me much more important will be the possibility for manually tagging.

[AngelaDMerkel]

If you want a tool for comparing AI models, that is not what immich is for.

I'm not skilled in this field at all, so please forgive me if I've explained poorly. Why would it not be possible to use an OCR tool like Tesseract and write any visible words to a little sidecar or somewhere in the database so that certain text strings can be searched and compared against transcribed content?

Would attempting to do so interfere with the ability to properly use the facial recognition models currently in use?

[NicholasFlamy]

Sorry @AngelaDMerkel, I should've @ the person I was talking to. You do realize you're asking for real OCR but @aisbergde seems to be asking for the ability to run multiple CLIP models at once so that you can switch while searching at least partly for the purpose of comparing models. I was attempting to address that comparing models can be done outside of immich and that the point of immich is a photo manager that has cool features like search using CLIP, not specifically a tool for comparing CLIP models.

[ZtereoHYPE]

Voyage AI just released a very promising model for multimodal (images and document screenshots) embeddings search.
Perhaps this could be good enough to not require OCR to be implemented separately from the current CLIP search?

[blublub]

OH YES PLZ! - any kind of detection of writing on screens and signs would be awesome

[mertalev]

FYI that model isn't open source.

A model that can effectively understand text in images is nice, but I think the copy/paste-ability of the text is important besides the ability to search for it. Embedding models can somewhat do the latter and will get better at it over time as new models are released, but the former is a gap that can only be filled through a different kind of model.

[ctrlshiftm129]

FYI that model isn't open source.

A model that can effectively understand text in images is nice, but I think the copy/paste-ability of the text is important besides the ability to search for it. Embedding models can somewhat do the latter and will get better at it over time as new models are released, but the former is a gap that can only be filled through a different kind of model.

Is OCR on the roadmap or will it be far away?

[mertalev]

I can say that I don't intend to work on it within the next 2 months. It's possible for it to be added before the stable release depending on timing, but more likely that it will come after.

[NicholasFlamy]

what's wrong with tesseract, other than not being the shiny new thing?

I don't think anyone said there was anything wrong with it. You could phrase your question as: "Is tesseract a good option?"

[Aevaris]

I look forward to this feature. I tried implementing paperless NGX recently and was frankly disappointed. My main use cases are images (memes, pictures of whiteboards, pictures of writing/plaques from museums, etc) rather than actual PDFs which paperless didnt do a good job with. If/when Immich adds this, it will be huge.

[mertalev]

unfortunately, i doubt adding OCR to immich would do much better, as paperless uses Tesseract, which is to my knowledge the best OCR engine.

This is definitely not true. There are plenty of models that outperform Tesseract.

[gcs8]

Also, getting something in a place nerds want to play with it will con them into putting in extra work to make it do the thing they want.

[Aevaris]

unfortunately, i doubt adding OCR to immich would do much better, as paperless uses Tesseract, which is to my knowledge the best OCR engine.

This is definitely not true. There are plenty of models that outperform Tesseract.

And even at a worst case, if it is as 'good' as Paperless, it would meet my expectations as I'd have everything under 1 roof (dont need to maintain 2 apps, have users go to 2 apps, avoids duplication of source files). The UI and general experience of Immich far exceeds that of paperless (for me).

If the model performs better than Paperless, then that is a bonus and it would exceed my expectations.

I expect with time (e.g. if i think 5 years from now), all models will improve so I'm not terribly concerned about the current accuracy.

[Fireflaker]

TLDR: Yes I want OCR too

I am not looking for high accuracy OCR on pages full of text, but OCR will indefinitely come in handy for screenshots - and I believe it will significantly improve search accuracy. Usually on google photo- when smart search can't get specific enough, I would search for some random text that I remembered in that photo- and it usually pulls the right photo right up.

[Martin22]

Just a small idea - whether it might be better to implement VLM directly instead of OCR, but I understand that it's more computationally demanding than combining OCR and CLIP.

[rekcodocker]

While it would be nice I would be against it.

Two reasons.

1. OCR has value on its own. Even just transcribing text in images will make them more useful and easier to find.

2. This is a big scope-creep. Textual description of images is new technology compared to OCR.
   There is possibly no free model that is stable enough and also easy to integrate.
   This requires more research, more work, not to mention large amounts of data and huge requirements for processing.
   OCR on the other hand, is mature technology and available in open source.

I am not saying it is impossible, or that it is a bad idea. I am saying that scope creep has a proven track record of killing projects.

[Martin22]

I have tried many open-source OCR solutions myself, but unfortunately, the accuracy heavily depends on the chosen language model. I can send an image where Google correctly recognizes the text and makes it searchable, but with open-source OCR, I always end up with a garbled result. And yet, it's such a simple text, no diacritics, all uppercase letters.

Just to mention a few, I tried Tesseract, EasyOCR, Kraken OCR, MMOCR, and maybe some others I don't remember. I haven’t tested VLM locally, though. There are many open-source VLM models, but the smaller ones that can run efficiently on a CPU and provide fast results are likely not as accurate.

[Martin22]

I actually haven't tried PaddleOCR yet, which is mentioned right at the beginning, but it managed to recognize the text from my image flawlessly, well, if I don't take the issue with spacing into account. Unfortunately, Tesseract distorted the text from the image.

[NicholasFlamy]

Just a small idea - whether it might be better to implement VLM directly instead of OCR, but I understand that it's more computationally demanding than combining OCR and CLIP.

VLM likely wouldn't accomplish the things most people would want from OCR. CLIP already gives you images based on words/descriptions, OCR is intended for extracting text from images (often documents) and there's been talks of further enhancing search with text from OCR.

I think VLM should have its own feature request. It might end up being closed as not planned but that's not my call.

Also, I think someone discussed the possibility of generating words and tagging images if the words match to a threshold with the CLIP embeddings.

[nyakang]

https://github.com/flipped-1121/immich/tree/ocr

I was eager to try OCR in Immich, so I built a local integration rather than wait for the official roadmap. The initial prototype supported both PaddleOCR and Tesseract, but Tesseract’s accuracy on non-English text was disappointing and it added extra maintenance overhead, so I’ve dropped it. PaddleOCR—consistently the stronger performer in my tests—is now the sole engine.

Because Immich is a sizable, sophisticated codebase, maintaining this feature alone long-term will be tough. If the community is interested, I’d welcome testers, feedback, and code contributions. I’ll keep the branch alive as long as there’s momentum, and once things are stable I’ll open a PR upstream. Keen to collaborate with anyone who’d like to help!

PixPin_2025-05-13_13-07-29.mp4

PixPin_2025-05-13_13-29-25.mp4

[ghost]

Awesome!

Have few questions:

1. I'm on weak machine with Immich and the ML is done by the Remote ML.
   Is this change compatible with it? How much RAM each model consume?
2. for install the build of docker image is needed locally and with that done just changing docker-compose for local build?
3. in case of need to switch back to main - what will happen to the files outside of containers (the state of OCR ML in database)?

[si9ma]

I tried it out and it feels pretty good. Overall it's usable!

[elmocito]

I tried it out and it feels pretty good. Overall it's usable!

How? 😩

[nyakang]

https://github.com/flipped-1121?tab=packages&repo_name=immich

[dodysw3]

With Deepseek OCR model released, which seems to be the most powerful OCR to date, maybe consider adding support to this model?

[WuSiYu]

+1, I think add the OCR function is meaningful. It's not conflict with CLIP. And apps like Apple Photo has OCR and fuzzy search at same time too. It's extremely useful when searching photos about text material or sign.

[Waffled-II]

Sad this hasn’t been really implemented yet. This is one of the few big features google photos has over this.

[reynold-lariza]

Yeah, I just discovered this just now, after a full night of migrating all my iCloud photos to it... XD

[m41denx]

I tried merging pr with latest release today and I have bad news. A LOT has changed in immich codebase since last commit in this pull request, especially search apis and some things in mobile app too. So I don't think we'll get OCR anytime soon

[elmocito]

Nooooooo 😭😭😭😭😭

[nyakang]

Thanks for the continued interest. Current status: this PR remains a draft with several items left to complete.
Due to my schedule, I can’t commit to a steady cadence right now, but I will continue progressing on it as time permits.

[elmocito]

If there is anything (non-DEV) I can help you with. Please tell me.

[nyakang]

The OCR feature has been merged into the main branch, and a release can be expected.

[mertalev]

We can look into possibly supporting other variants. It seems feasible to expose it as a setting from a quick look.

[elmocito]

The German results look pretty promising. A language multi selector would be pretty nice.

[Qhilm]

Where is the language setting? I must be blind.

[chriexpe]

There is no language setting for now, the model being used only supports Chinese and English, but if your language has Latin alphabet it does work too.

[NicholasFlamy]

If y'all want a language setting, please go ahead and make a new feature request and stop messaging in this thread.