AI: API to connect to external services for Image tagging / object recognition

[ScuttleSE]

I have been playing around with a few different APIs for image-tagging, and it would be neat if PhotoPrism had an option to use them directly in the application.

I have been testing AWS Rekognition, Imagga and Googles Vision AI.
They all cost money, with Imagga being the most expensive. AWS and Google charge per image, and it costs about $1 per 1000 images. Imagga charges $79 per month for 70.000 api calls/images.

All three has a free tier which allows for 1000 pictures per month.

Even I, with my horrible coding skills was able to write simple python-scripts that went through my collection and got tags for my pictures.

---

Note that we have started working on a microservice that will allow us to use advanced computer vision models through a REST API (also includes support for additional AI/ML hardware):

• https://github.com/photoprism/photoprism-vision

In the next step, we are going to integrate these models into PhotoPrism. We can either use this issue for that - or add a new issue if this feature request is specifically aimed at using cloud models, e.g. hosted by Amazon/AWS (for other models and/or lower hardware requirements) instead of a self-hosted service (for enhanced privacy).

Related Issues:

• Labels: Load and use a custom TensorFlow model #127 (comment)
• AI: Generate captions with Clip Interrogator #3438
• AI: Integrate Florence 2 Vision AI for Auto Caption #4356
• AI: Integrate a model for Optical Character Recognition (OCR) #907
• Config: Add support for AI inference with Coral Edge TPU USB devices #536
• AI: Upgrade TensorFlow from v1.15.2 to v2.18.0 #222

[lastzero]

Well, our idea is not to upload pictures to Google or Amazon. So while it can be done, it's not really our business. Might take a look at this later when we have time for experiments.

[streiman]

Even I, with my horrible coding skills was able to write simple python-scripts that went through my collection and got tags for my pictures.

I am thinking about tagging my picture collection myself outside of PhotoPrism. Would you share your scripts, so I and others can have a look and try myself without starting from scratch?

[jasontitus]

This does seem like it could open up a bunch of possibilities to try out different tagging mechanisms rather than just the built-in ones. Realistically, there will be ones that work better for various users (bird species tagging, car type tagging, etc that would never make sense to build for the main app).

[digitalferrari]

I've been experimenting with this idea as well. I'm classifying a million images taken by staff photographers over 20 years. We can't effectively use our old photos, because we can't search for anything. There's little use in that many photos if you can't search by context.

The current plan is to send an image to either Google Cloud Vision API or Amazon Rekognition API and then save the data into the IPTC or EXIF data on the original image file. Either singles or batched.

For non-personal, non-private projects/images, running photos through an image classifier is necessary if you want to search a million photos and find what you're looking for.

Running one image per request at a time during processing seems fine for daily workflows with limited images.
For batching an entire image set, I would put the images in the cloud, run them through the API, save the new classifications to the original file, then download the updated originals and import them into PhotoPrism. Then continue run one at a time during indexing for daily workflows going forward.

For development, I think building the single image API call during processing would be really helpful. Is there a plugin architecture available or planned? Contributors could write a plugin for these classifiers.

[streiman]

I process the images in a folder before moving them to photoprism using Amazon Rekognition. They are labeled, the labeled translated to German, and if a person is on the picture, Amazon Rekognition is called again to recognize faces (might be not necessary any more since Photoprism since I made this code added face recognition, but I like the names in my descriptions).

You can find the python source code here: https://github.com/streiman/tag-my-picture

Maybe you can adopt it for your usecase.

[ShakataGaNai]

I'm a really new user to this project but AI tagging is probably the headline feature these days. Part of the Google Photo (and similar) draw, is the ability to search for photos based on content. Not just searching for a thing like "dog", or text (since they attempt to OCR), but also searching a concept like "Cooking" or "Dancing" and returning mostly good results? Magic. Clearly PhotoPrism has already taken a swing at that, using local-first, which makes a ton of sense. But for those of us whom are OK with the price (both dollars and privacy) with the use of Cloud AI, they add a HUGE amount of value. The fact that Google Photo's can match a photo of my nephew from 3 years old to 16 years old, is so amazingly useful. Rekognition has very good facial technologies on the other hand.

My Google Photos takeout was just shy of 2TB, spanning 19 years of digital photography (Starting with the Nikon D70). While it would be not an insignificant cost, I would love the option to run everything through Google Cloud Vision API and/or Rekognition at least once. It would make sense for users to have the option for a one-time scan (of some, or all), re-scans (the AI's get better over time), or scan-upon-upload.

[grinco]

To add to this thread, the biggest benefit of supporting an external API would be the ability to use self-hosted AI tools like CodeProject.AI. It would also add additional benefits of having a distributed hardware set-up (as opposed to the all-in-one system).