services	platforms	author
functions	dotnet	lindydonna

Azure Functions Photo Mosaic Generator

Use Azure Functions and Microsoft Cognitive Services Custom Vision Service to generate a photo mosaic from an input image.

For example, you can train your model with Orlando landmarks, such as the Orlando Eye. Custom Vision will recognize an image of the Orlando Eye, and the function will create a photo mosaic composed of Bing image search results for "Orlando Eye." See example below.

Branches

• Use the master branch if you're on Windows
• Use the core branch if you're on a Mac.

Prerequisites

1. Visual Studio, either:

   • Visual Studio 2017 Update 3 with the Azure workload installed (Windows)
   • Visual Studio Code with the C# extension (Mac/Linux)

2. If running on a Mac/Linux, .NET Core 2.0

3. If running on a Mac/Linx, install azure-functions-core-tools from npm

4. Azure Storage Account

5. Azure CLI 2.0

1. Create API keys

1. Create a Bing Search API key:

   • In the Azure portal, click + New and search for Bing Search APIs.
   • Enter the required information in the Create blade. You may use the lowest service tier of S1 for this module.

2. (Optional) Create a Computer Vision API key. The function will fall back to the regular Computer Vision API if there isn't a match with images that have been trained in the Custom Vision Service. If you plan to test only with images that will match custom vision, you can skip this step.

   To create a Computer Vision API key:

   • In the Azure portal, click + New and search for Bing Search APIs.
   • Enter the required information in the Create blade. You may use the free tier F0 for this module.

2. Set up Custom Vision Service project

1. Go to https://www.customvision.ai/

2. Sign in with a Microsoft account

3. Create a new project and select the image type, such as "Landmarks"

4. Add images and tag them. You can use a Chrome extension such as Bulk Image Downloader to download Google or Bing images of landmarks.

5. Once you have added several landmarks, click the Train button on the upper right. Make sure you have at least 2 tags and 5 images for each tag.

6. (Optional) Test image recognition using the Test tab.

7. Click on the Performance tab. If you have more than one iteration, choose the latest iteration and click Make default.

3. Configure the photo mosaic project

Run the install script

There's a Python setup script setup.py that will set up your storage account keys. It uses the Azure CLI 2.0 to automate the storage account setup. Run the following commands:

az login
python setup.py storage-account resource-group

If you get python errors, make sure you've installed Python 3 and run the command python3 instead (see Installing Python 3 on Mac OS X).

This will modify the file local.settings.json.

Alternatively, you can run the script from the Azure Cloud Shell in the Azure Portal. Just run python and paste the script. The script prints out settings values that you can use to manually modify local.settings.json.

Edit local.settings.json

1. If using Visual Studio, open MosaicMaker.sln. On a Mac, open the photo-mosaic folder in VS Code.

2. Open the file MosaicMaker/local.settings.json

3. In the Custom Vision portal, get the URL for your prediction service. Select Prediction URL and copy the second URL in the dialog box, under the section "If you have an image file". It will have the form https://southcentralus.api.cognitive.microsoft.com/customvision/v1.0/Prediction/<guid>/image. Paste this value for the key PredictionApiUrl in local.settings.json.

4. In the Custom Vision portal, select the settings gear in the upper right. Copy the value of Prediction Key for the key PredictionApiKey in local.settings.json.

   

5. In the Azure portal, select your Bing Search APIs instance. Select the Keys menu item and copy the value of KEY 1. Paste the value for the key SearchAPIKeyin local.settings.json.

6. (Optional) Photo mosaic will fall back to the regular vision service if there is not a match with custom vision. Paste your key for your Cognitive Services Vision Service as the value for MicrosoftVisionApiKey in local.settings.json.

Summary of App Settings

Key	Description
AzureWebJobsStorage	Storage account connection string.
SearchAPIKey	Key for Bing Search API.
MicrosoftVisionApiKey	Key for Computer Vision Service.
PredictionApiUrl	Endpoint for Cognitive Services Custom Vision Service. It should end with "image".
PredictionApiKey	Prediction key for Cognitive Services Custom Vision Service.
generate-mosaic	Name of Storage queue for to trigger mosaic generation. Default value is "generate-mosaic".
input-container	Name of Storage container for input images. Default value is "input-images".
output-container	Name of Storage container for output images. Default value is "mosaic-output".
tile-image-container	Name of Storage container for tile images. Default value is "tile-images".
SITEURL	Set to http://localhost:7072 locally. Not required on Azure.
STORAGE_URL	URL of storage account, in the form https://accountname.blob.core.windows.net/
CONTAINER_SAS	SAS token for uploading to input-container. Include the "?" prefix.
APPINSIGHTS_INSTRUMENTATIONKEY	(optional) Application Insights instrumentation key.
MosaicTileWidth	Default width of each mosaic tile.
MosaicTileHeight	Default height of each mosaic tile.

If you want to set these values in Azure, you can set them in local.settings.json and use the Azure Functions Core Tools to publish to Azure.

func azure functionapp publish function-app-name --publish-app-settings

4. Load Tile Images

When the function app creates a mosaic, it needs source images to compose the mosaic. The tile-image-container referred to in App Settings (defaulted to tile-images) is the container that the function will look for images to use. Running the setup.py script above generated the containers for you, but you'll need to load images that container. Using the portal, Azure Storage Explorer, or any other method of uploading blobs, upload images into the tile-image-container inside of your storage account. You can reuse the landmark images you downloaded earlier if desired.

5. Run the project

1. Compile and run:

   • If using Visual Studio, just press F5 to compile and run PhotoMosaic.sln.

   • If using VS Code on a Mac, the build task will run dotnet build. Then, navigate to the output folder and run the Functions core tools:

     cd photo-mosaic/MosaicMaker/bin/Debug/netstandard2.0/osx
     func host start
     

   You should see output similar to the following:

   Http Functions:
   
           RequestMosaic: http://localhost:7072/api/RequestMosaic
   
           Settings: http://localhost:7072/api/Settings
   
   [10/4/2017 10:24:20 PM] Host lock lease acquired by instance ID '000000000000000000000000C9A597BE'.
   [10/4/2017 10:24:20 PM] Found the following functions:
   [10/4/2017 10:24:20 PM] MosaicMaker.MosaicBuilder.RequestImageProcessing
   [10/4/2017 10:24:20 PM] MosaicMaker.MosaicBuilder.Settings
   [10/4/2017 10:24:20 PM] MosaicMaker.MosaicBuilder.CreateMosaicAsync
   [10/4/2017 10:24:20 PM]
   [10/4/2017 10:24:20 PM] Job host started
   Debugger listening on [::]:5858
   

2. To test that the host is up and running, navigate to http://localhost:7072/api/Settings.

Run

• To run using the provided SPA, open in a command prompt and navigate to the Client directory.

  • Run npm install
  • Run npm start. This will launch a webpage at http://127.0.0.1:8080/.

• To run manually, send an HTTP request using Postman or CURL:

  POST http://localhost:7072/api/RequestMosaic

  Body:

  {
  "InputImageUrl": "http://url.of.your.image",
  "ImageContentString": "optional keyword for mosaic tiles",
  "TilePixels": 20 
  }