Google Cloud Vision API enables developers to understand the content of an image by encapsulating powerful machine learning models in an easy to use REST API. It quickly classifies images into thousands of categories (e.g., "sailboat", "lion", "Eiffel Tower"), detects individual objects and faces within images, and finds and reads printed words contained within images. You can read more about Google Vision here - https://cloud.google.com/vision/

This Python script for Cantemo Portal Rules Engine 3 allows you to run Google Vision image analysis on images in Cantemo Portal.

Example Results

A picture is worth a thousand words, so here are some example results using this script. These show that Google's API is very good at generating relevant labels automatically from nothing more then the image data. Also text is recognized with high accuracy.

Photo by Asmo Soinio

Photo by Asmo Soinio

Photo by Asmo Soinio

Photo by Juho Särkilä

Photo by Genie Ranada-Babista

Prerequisites

• Cantemo Portal 2.3.0 or later, with license for Rules Engine
• Google Cloud Platform account with billing enabled, see https://cloud.google.com/
• Google Vision API key, see https://cloud.google.com/vision/docs/auth-template/cloud-api-auth

Google Vision API is free for up to 1000 requests/month, but a credit card is required when registering the project. See https://cloud.google.com/vision/docs/pricing

Installation

Create a Metadata Group for analysis output

This example uses an explicit metadata group to store the analysis results. You could also add these fields into an existing metadata group.

1. From Portal GUI, open Manage > Metadata Groups
2. Add a metadata group with the name GoogleVision
3. Add one Tags-field for Label Detection results
4. Add two Textarea-fields, one for Text Detection results and one for raw API JSON output
5. Set Max Length on the Textarea fields to 1000000 so the output fits in them

Feel free to rename the fields to your liking, and write descriptions. The resulting group should look like this:

After these steps in the Metadata Manager, permissions to access this metadata group must be added to the Admin user group, and any others that should be able to ingest images for analysis:

1. Open Admin > Users and Groups > All Groups
2. Select the group named Admin
3. Select tab Metadata Groups
4. Select GoogleVision from Available Metadata Groups and click arrow to add it to Chosen Metadata Groups
5. Save & continue

Get and configure the script

Download google_vision.py to your computer. Modify the local script file:

Add your own Google Vision API key on the following line, in brackets:

VISION_API_KEY = ''

For example:

VISION_API_KEY = 'AbCdEfGhIjKlMnOpQrStUvWxYz1234567890AA'

Replace the metadata Field Ids in the script with those from the GoogleVision group you created in Portal, modifying these rows:

TAGS_FIELD = 'portal_mf572883'
OCR_FIELD = 'portal_mf915296'
FULL_OUTPUT_FIELD = 'portal_mf700776'

The script is now ready to be used in your Portal system.

Create an automatic Rule to execute the script

1. Open Admin > Rules Engine 3 in Portal
2. Under Add a new script: select your local google_vision.py file and select Upload
3. Select Create rule for new items to have this applied an all ingested images (with GoogleVision metadata group)
4. Enter a descriptive name for the rule
5. Enable Run Shell Script and select google_vision.py from the Script file dropdown
6. Create rule

The rule can also be created as a Manual Rule, so it can be executed for example on search results page.

Executing the script

This uploads an image to Portal with the metadata group set. After the ingest is completed the rule will be executed and the item metadata contains the analysis results.

1. Select Ingest > Upload
2. Add some files to be uploaded
3. Select Add Metadata
4. From Show advanced > Change metadata form, select `GoogleVision
5. Select Start Upload

A Raw import Job should be visible in Admin > Jobs for the new file. After the job has finished, the rule will be automatically executed on the new item. This can be monitored from the Rules Engine 3 UI.

After the rule has finished, it should show 1 in Handled items. A link to the handled item is under Details. Open the item page to view the analysis results:

To really test the power of this API, upload some more interesting photos!

License

BSD 3-Clause License