Use Cases

otu la edited this page Dec 15, 2015 · 20 revisions

Table of Contents

1 Introduction
2 Authentication
  2.1 Client Authentication
  2.2 Back-end Authentication
3 Use cases
  3.1 Content-Based Analysis
  3.2 Content-Based Search
  3.3 Facebook Profile Summarization
  3.4 Facebook Profile Summarization Feedback
  3.5 File Upload and Analysis
  3.6 Id-Based Search
  3.7 Keyword-Based Search
  3.8 Similarity Feedback
  3.9 Twitter Profile Summarization
  3.10 User Feedback
  3.11 User Feedback - Face Recognition

1 Introduction

This page is contains use cases for the Content Analysis (CA) API. The purpose of the documentation is to illustrate the use of APIs needed to implement basic use cases required for content analysis ecosystem: uploading content; executing content analysis on the service back-ends; and retrieving content – or content metadata – based on search terms provided by a client (or user). In the scope of this documentation content is limited to photographs (image-based content).

2 Authentication

2.1 Client Authentication

In the current implementation, the client authentication process uses the default front-end built-in authentication scheme (username + password) when needed. The supported authentication methods are:

  • HTTP Basic Authentication.
  • OAuth/OAuth2 using one of the supported 3rd party services (Google, Facebook, Twitter).
  • Use of TLS/HTTPS if so desired.

All 3rd party storage solutions (e.g. Picasa, Facebook) are authenticated by the front-end for back-ends by using URL redirection.

2.2 Back-end Authentication

In the current implementation the back-ends are assumed to be on trusted zone, and thus, are not authenticated in any other way than using a simple id check or IP-based authentication. The back-ends can be authenticated by utilizing one the methods available to clients if required.

3 Use cases

The use cases are illustrated by sequence diagrams, and further explained by REST method calls and example responses. The uses cases omit user authentication, but assume that authentication has been performed when required by the methods used in the use cases.

3.1 Content-Based Analysis

This use case is for scheduling analysis tasks on the analysis back-ends. The tasks are generated automatically when user connects an external photo account (e.g. Picasa), or when content of a previously connected account are synchronized with the front-end.

3.1.1 Diagrams

Content-based Analysis

3.1.2 Examples

This example assumes that connecting an account has already been made by using the front-end. The connecting process is not explained as it is performed by the front-end’s internal functionality, and thus, it is out of scope of this documentation.

addTask (Front-end -> Analysis Back-end):

Example Query:

POST /rest/ca/addTask
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskDetails>
 <taskId>1</taskId>
 <taskType>ANALYSIS</taskType>
 <backendId>0</backendId>
 <userId>1</userId>
 <userConfidence>1</userConfidence>
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri>
 <mediaList>
  <media>
   <UID>1</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url>
   <credits>Creative Commons</credits>
   <name>Photo</name>
   <description>cute photo</description>
   <visibility>PRIVATE</visibility>
   <mediaType>PHOTO</mediaType>
   <objectList>
    <object>
     <objectId>123</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>cute</value>
     <confidence>0.5</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskDetails>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addTask" service="ca">
 <status>ok</status>
</response>

Retrieve photo (redirect) (Analysis Back-end -> Front-end):

Example Query:

GET /rest/ca/r?uid=1&service_id=0

Example Result:

REDIRECT/PHOTO CONTENT

taskFinished (Analysis Back-end -> Front-end):

Example Query:

POST /rest/ca/taskFinished
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskResults>
 <taskId>1</taskId>
 <taskType>ANALYSIS</taskType>
 <backendId>0</backendId>
 <status>COMPLETED</status>
 <message>Analysis completed successfully.</message>
 <mediaList>
  <media>
   <UID>1</UID>
   <backendStatusList>
    <backendStatus>
     <backendId>0</backendId>
     <status>COMPLETED</status>
     <message>Completed successfully.</message>
    </backendStatus>
   </backendStatusList>
   <objectList>
    <object>
     <objectId>123</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>cute</value>
     <confidence>0.5</confidence>
     <rank>0</rank>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskResults>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="taskFinished" service="ca">
 <status>ok</status>
</response>

3.2 Content-Based Search

By using this method, the user can use a photo from external source to search similar photos from his/her photo collection, or from photos that other users have defined public.

3.2.1 Diagrams

Content-Based Search

3.2.2 Examples

similarPhotosByContent (Client -> Front-end):

Example Query:

POST /rest/ca/similarPhotosByContent?url=http%3A%2F%2Fexample.org%2Fimage.jpg&data_groups=all

similarPhotosByContent (Request; Front-end -> Analysis Back-end):

Example Query:

POST /rest/ca/similarPhotosByContent?url=http%3A%2F%2Fexample.org%2Fphoto.jpg&data_groups=all

Retrieve photo (direct access from the given URL):

Example Query:

GET /photo.jpg

Example Result:

PHOTO CONTENT

Final Result (Response; Analysis Back-end -> Front-end):

Final Search Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="similarPhotosByContent" service="ca">
 <status>ok</status>
 <mediaList>
  <media>
   <UID>1</UID>
 <objectList>
  <object>
   <objectId>123</objectId>
   <objectType>KEYWORD</objectType>
   <status>CANDIDATE</status>
   <backendId>0</backendId>
   <value>cute</value>
   <confidence>0.5</confidence>
   <rank>0</rank>
   <visualObjectId>1</visualObjectId>
   <mediaType>PHOTO</mediaType>
  </object>
 </objectList>
</media>

Final Result (Front-end -> Client):

The results may include a combined result set received from multiple analysis back-ends. 
In this example case only the results provided by a single back-end is shown.


Final Search Result to Client:

<?xml version="1.0" encoding="UTF-8"?>
<response method="similarPhotosByContent" service="ca">
 <status>ok</status>
 <mediaList>
  <media>
   <UID>1</UID>
   <objectList>
    <object>
     <objectId>123</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>cute</value>
     <confidence>0.5</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</response>

3.3 Facebook Profile Summarization

This use case is for scheduling summarization tasks on the analysis back-ends. The tasks are generated automatically when user connects an external account with profile content, or when content of a previously connected account are synchronized with the Front-end.

3.3.1 Diagrams

Facebook Profile Summarization

3.3.2 Examples

This example assumes that account connection has already been made using the Front-end. The connecting process is not explained as it is performed by The Front-end’s internal functionality, and thus, is out of scope of this documentation.

addTask (Front-end -> Summarizer):

Example Query: 
 
POST /rest/fbj/addTask 
Content-Type: text/xml; charset=UTF-8 
 
[HTTP BODY STARTS] 
<?xml version="1.0" encoding="UTF-8"?> 
<taskDetails> 
 <taskId>1</taskId> 
 <taskType>FACEBOOK_PROFILE_SUMMARIZATION</taskType> 
 <backendId>0</backendId> 
 <userId>1</userId> 
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri> 
 <profile> 
  <groupList> 
   <group> 
    <name weight=”5”>Super Secret Society of Super Secrecy</name> 
    <description weight=”3”>Waaaaay too secret for public description</description> 
    <owner>Pohtii Block</owner> 
    <updatedTimestamp>2013-04-24T15:20:13+0300</updatedTimestamp> 
    <groupPrivacy>SECRET</groupPrivacy> 
    <location> 
     <name>San Francisco, California</name> 
     <country>USA</country> 
     <state>California</state> 
     <city>San Francisco</city> 
    </location> 
   </group> 
  </groupList> 
  <likeList> 
   <like> 
    <category>Product/service</category> 
    <name>Facebook Developers</name> 
   </like> 
  </likeList> 
  <statusMessageList> 
   <statusMessage> 
    <commentList> 
     <comment> 
      <createdTimestamp>2013-04-11T16:02:21+0300</createdTimestamp> 
      <from>Pohtii Block</from> 
      <message weight=”1”>ooo, a real facebook developer :O</message> 
      <likeCount>0</likeCount> 
     </comment> 
    </commentList> 
    <from>Mary Peettersson</from> 
    <message weight=”2”>uuu... I am a real developer now!</message> 
    <updatedTimestamp>2013-04-11T12:02:21+0300</updatedTimestamp> 
    <likeCount>0</likeCount> 
   </statusMessage> 
  </statusMessageList> 
  <photoDescriptionList> 
   <photoDescription> 
    <photoUID>0</photoUID> 
    <commentList> 
     <comment> 
      <createdTimestamp>2013-05-07T17:36:12+0300</createdTimestamp> 
      <from>Mary Peettersson</from> 
      <likeCount>1</likeCount> 
      <message weight=1”>testing facebook comments</message> 
     </comment> 
    </commentList> 
    <createdTimestamp>2013-04-11T16:07:28+0300</createdTimestamp> 
    <description weight=2”>Except for this picture, which also has Mona Lisa.</description> 
    <from>Mary Peettersson</from> 
    <likeCount>0</likeCount> 
    <location> 
    <city>Paris</city> 
     <country>France</country> 
     <name>Musée du Louvre</name> 
    </location> 
    <updatedTimestamp>2013-05-07T17:04:21+0300</updatedTimestamp> 
    <photoTagList> 
     <photoTag> 
      <value>Museum</value> 
     </photoTag> 
    </photoTagList> 
   </photoDescription> 
  </photoDescriptionList> 
  <videoDescriptionList> 
   <videoDescription> 
    <createdTimestamp>2013-04-11T16:07:28+0300</createdTimestamp> 
    <description weight=”2”>An example of a video description.</description> 
    <from>Mary Peettersson</from> 
    <updatedTimestamp>2013-05-07T17:04:21+0300</updatedTimestamp> 
   </videoDescription> 
  </videoDescriptionList> 
  <eventList> 
   <event> 
    <RSVPStatus>ATTENDING</RSVPStatus> 
    <location> 
     <name>The Moon</name> 
    </location> 
    <owner>Pohtii Block</owner> <!-- owner of this event --> 
    <description weight=”3”>The coolest party on this wo... outside this world</description> 
    <name weight=”5”>Väärä's BDay Party</name> 
    <startTimestamp>2013-06-08T00:00:00+0300</startTimestamp> 
    <endTimestamp>2013-06-09T00:05:00+0300</endTimestamp> 
    <updatedTimestamp>2013-05-07T00:05:00+0300</updatedTimestamp> 
    <groupPrivacy>SECRET</groupPrivacy> 
   </event> 
  </eventList> 
  <userDetails> 
   <bio>I'm a fun loving girl. I like sports, camping and kids. Here ends the 
first chapter of my bio. Here begins the next chapter of my bio. Just typing 
here some stuff about me. Pretty cool, eh?</bio> 
   <birthday>1992-02-07T00:00:00+0200</birthday> 
   <gender>FEMALE</gender> 
   <hometown>Jakarta, Indonesia</hometown> 
   <location> 
    <name>London, United Kingdom</name> 
   </location> 
   <name>Peettersson, Mary</name> 
   <political>Republican Party</political> 
   <relationshipList> 
    <relationship> 
     <relationshipType>Married</relationshipType> 
     <with>Peter Peettersson</with> 
    </relationship> 
   </relationshipList> 
   <religion>Catholic</religion> 
   <updatedTimestamp>2013-05-07T17:04:21+0300</updatedTimestamp> 
   <userId>3</userId> <!-- Front-end UserId for this FB User --> 
   <facebookId>0</facebookId> 
  </userDetails>
 </profile> 
</taskDetails> 
[HTTP BODY ENDS] 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="addTask" service="fbj"> 
 <status>ok</status> 
</response>

taskFinished (Summarizer -> Front-end):

Example Query: 
 
POST /rest/fbj/taskFinished 
Content-Type: text/xml; charset=UTF-8 
 
[HTTP BODY STARTS] 
<?xml version="1.0" encoding="UTF-8"?> 
<taskResults> 
 <taskId>1</taskId> 
 <taskType>FACEBOOK_PROFILE_SUMMARIZATION</taskType> 
 <backendId>0</backendId> 
 <status>COMPLETED</status> 
 <message>Summarization completed successfully.</message> 
 <objectList> 
  <object> 
   <objectId>123</objectId> 
   <objectType>KEYWORD</objectType> 
   <status>CANDIDATE</status> 
   <backendId>0</backendId> 
   <value>cute</value> 
   <confidence>9.8058747237</confidence> 
   <rank>0</rank> 
  </object> 
 </objectList> 
 <mediaList> 
  <media> 
   <UID>0</UID> 
   <objectList> 
    <object> 
     <objectId>124</objectId> 
     <objectType>KEYWORD</objectType> 
     <status>CANDIDATE</status> 
     <backendId>0</backendId> 
     <value>biking</value> 
     <confidence>4.58747237</confidence> 
     <rank>0</rank> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> 
  </media> 
 </mediaList> 
</taskResults> 
[HTTP BODY ENDS]

Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="taskFinished" service="fbj"> 
 <status>ok</status> 
</response>

3.4 Facebook Profile Summarization Feedback

This use case is for scheduling summarization feedback tasks on the analysis back- ends. The tasks are generated automatically when gives feedback (i.e. ranks the tags) on the previously generated tags.

3.4.1 Diagrams

Facebook Summarization Feedback

3.4.2 Examples

setRank (Client -> Front-end):

Example Query: 
 
POST /rest/fbj/setRank?rank=1:100,2:-10 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response service="fbj" method="setRank"> 
 <status>ok</status> 
</response> 

addTask (Front-End -> Summarizer):

Example Query: 
 
POST /rest/fbj/addTask 
Content-Type: text/xml; charset=UTF-8 
 
[HTTP BODY STARTS] 
<?xml version="1.0" encoding="UTF-8"?> 
<taskDetails> 
 <taskId>1</taskId> 
 <taskType>FACEBOOK_PROFILE_SUMMARIZATION</taskType> 
 <backendId>0</backendId> 
 <userId>1</userId> 
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri> 
 <objectList> 
  <object> 
   <objectId>123</objectId> 
   <objectType>KEYWORD</objectType> 
   <status>CANDIDATE</status> 
   <backendId>0</backendId> 
   <value>cute</value> 
   <confidence>9.8058747237</confidence> 
   <rank>0</rank> 
   <visualObjectId>1</visualObjectId> 
  </object> 
 </objectList> 
</taskDetails> 
[HTTP BODY ENDS] 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="addTask" service="fbj"> 
 <status>ok</status> 
</response>

Note that the Facebook Summarization Feedback tasks may not produce any results, and thus, no further communication is needed. Responding with taskFinished is not required. Responding with a proper status code without results is accepted and will simply update the analysis status on the front-end.

3.5 File Upload and Analysis

This use case is for uploading a file to the service and starting a photo analysis task for the file. The example assumes that the uploaded file is an image file of a supported format. This example use case utilizes the default back-ends provided by the system, and omits the parts of the process involving back-ends for simplification.

3.5.1 Diagrams

File Upload and Analysis

3.5.2 Examples

This example omits the analysis task creation and delivery to the analysis back-ends.

addFile (Client -> Front-end):

Example Query: 
 
POST /rest/fs/addFile 
 
[HTTP BODY STARTS] 
 
[File Contents] 
 
[HTTP BODY ENDS] 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="addFile" service="fs"> 
 <fileList> 
  <file> 
   <fileId>0</fileId> 
   <url>http://fe.example.org/00000000-abcd-efgh-ijkl-000000000000.jpg</url> 
  </file> 
 </fileList> 
 <status>ok</status> 
</response>

Add received URL to analysis (Client -> Front-end):

Example Query:

POST /rest/cos/addUrl?url=http%3A%2F%2Ffe.example.org%2F00000000-abcd-efgh-ijkl-000000000000.jpg

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addUrl" service="cos">
 <status>ok</status>
</response>

3.6 Id-Based Search

This method requires the photo UID, which is automatically assigned to each uploaded photo by the front-end. Thus, the method can only use previously uploaded photos as reference. If the user is not authenticated, only UIDs of public photos can be given as reference and the search will only return photos that are defined public by the users.

3.6.1 Diagrams

Similarity Search by ID

3.6.2 Examples

similarPhotosById (Client -> Front-end):

Example Query: 
 
POST /rest/ca/similarPhotosById?uid=1&data_groups=all 

similarPhotosById (Request; Front-end -> Analysis Back-end):

Example Query: 
 
POST /rest/ca/similarPhotosById?uid=1&data_groups=all 

Final Result (Response; Analysis Back-end -> Front-end):

Final Search Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="similarPhotosById" service="ca"> 
 <status>ok</status> 
 <mediaList> 
  <media> 
   <UID>1</UID> 
   <objectList> 
    <object> 
     <objectId>123</objectId> 
     <objectType>KEYWORD</objectType> 
     <status>CANDIDATE</status> 
     <backendId>0</backendId> 
     <value>cute</value> 
     <confidence>0.5</confidence> 
     <rank>0</rank> 
     <visualObjectId>1</visualObjectId> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> 
  </media> 
 </mediaList> 
</response>

Final Result (Front-end -> Client):

The results may include a combined result set received from multiple analysis 
back-ends. In this example case only the results provided by a single back-end 
is shown. 
 
 
Final Search Result to Client: 
 
<?xml version="1.0" encoding="UTF-8"?> 
<response method="similarPhotosById" service="ca"> 
 <status>ok</status> 
 <mediaList> 
  <media> 
   <UID>1</UID> 
   <objectList> 
    <object> 
     <objectId>123</objectId> 
     <objectType>KEYWORD</objectType> 
     <status>CANDIDATE</status> 
     <backendId>0</backendId> 
     <value>cute</value> 
     <confidence>0.5</confidence> 
     <rank>0</rank> 
     <visualObjectId>1</visualObjectId> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> Tampere University of Technology    24 (45) 
  </media> 
 </mediaList> 
</response>

3.7 Keyword-Based Search

By using the keyword based search, the user can search photos which are related to each other by keywords.

3.7.1 Diagrams

Keyword-Based Search.

3.7.2 Examples

In this example, the optional result_info data group is used to request 
additional information about the search results. 
 
Example Query: 
 
GET /rest/ca/similarPhotosByKeyword?keywords=cute&data_groups=all,result_info 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?> 
 <response method="similarPhotosByKeyword" service="ca"> 
 <status>ok</status> 
 <mediaList> 
  <resultInfo> 
   <resultCount>1</resultCount> 
  </resultInfo> 
  <media> 
   <UID>1</UID> 
   <serviceId>0</serviceId> 
   <userId>1</userId> 
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url> 
   <credits>Creative Commons</credits> 
   <name>Photo</name> 
   <description>cute photo</description> 
   <visibility>PRIVATE</visibility> 
   <mediaType>PHOTO</mediaType> 
   <objectList> 
    <object> 
     <objectId>123</objectId> 
     <objectType>KEYWORD</objectType> Tampere University of Technology    26 (45) 
     <status>CANDIDATE</status> 
     <backendId>0</backendId> 
     <value>cute</value> 
     <confidence>0.5</confidence> 
     <rank>0</rank> 
     <visualObjectId>1</visualObjectId> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> 
  </media> 
 </mediaList> 
</response>

3.8 Similarity Feedback

User can submit feedback on the similarities of photos. The feedback does not have to relate to an existing search query or task, though in general the feedback is provided for a search query. This use case uses similarity search (ID-Based Search) as an example, but omits the front-end <-> back-end phase (with responses) of the similarity search.
The feedback is sent using the similarityFeedback method, which in turn, will trigger the generation of feedback task, which is sent to analysis back-ends. As with the regular Feedback Task ((User Feedback](#310-user-feedback)) the back-end could reply with additional taskFinished call to update metadata on the front-end.

3.8.1 Diagrams

Similarity Search Feedback

3.8.2 Examples

similarPhotosById (Client -> Front-end):

Example Query: 
 
POST /rest/ca/similarPhotosById?uid=1&data_groups=all 
 
Example Result: 
 
<?xml version="1.0" encoding="UTF-8"?>
 <response method="similarPhotosById" service="ca"> 
 <status>ok</status> 
 <mediaList> 
  <media> 
   <UID>1</UID> 
   <serviceId>0</serviceId> 
   <userId>1</userId> 
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url> 
   <credits>Creative Commons</credits> 
   <name>Photo</name> 
   <description>cute photo</description> 
   <visibility>PRIVATE</visibility> 
   <mediaType>PHOTO</mediaType> 
   <objectList> 
    <object> 
     <objectId>123</objectId> 
     <objectType>KEYWORD</objectType> 
     <status>CANDIDATE</status> 
     <backendId>0</backendId> 
     <value>cute</value> 
     <confidence>0.5</confidence> 
     <rank>0</rank> 
     <visualObjectId>1</visualObjectId> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> 
  </media> 
  <media> 
   <UID>2</UID> 
   <serviceId>0</serviceId> 
   <userId>1</userId> 
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>  
   <credits>Creative Commons</credits> 
   <name>Photo</name> 
   <description>cute photo</description> 
   <visibility>PRIVATE</visibility> 
   <mediaType>PHOTO</mediaType> 
   <objectList> 
    <object> 
     <objectId>124</objectId> 
     <objectType>KEYWORD</objectType> 
     <status>USER_CONFIRMED</status> 
     <backendId>0</backendId> 
     <value>cute</value> 
     <confidence>0.7</confidence> 
     <rank>0</rank> 
     <visualObjectId>1</visualObjectId> 
     <mediaType>PHOTO</mediaType> 
    </object> 
   </objectList> 
  </media> 
 </mediaList> 
</response> 
 
In practice, the query is first send to applicable back-ends, and based on the 
back-end responses the client response is generated. This phase is omitted for 
simplicity. See the example in id-based search for full listing (section 3.6).

similarityFeedback (Client -> Front-end):

Example Query:

POST /rest/ca/similarityFeedback
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<feedbackList>
 <mediaList>
  <media>
   <UID>1</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </mediaList>
 <similarMediaList>
  <media>
   <UID>2</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
   <UID>3</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=3&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </similarMediaList>
 <dissimilarMediaList>
  <media>
   <UID>4</UID>
 <serviceId>0</serviceId>
 <userId>2</userId>
   <url>http://fe.example.org/rest/ca/r?uid=4&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </dissimilarMediaList>
</feedbackList>
[HTTP BODY ENDS]

Note that the feedback items are not limited to the content of the original query. In this example the client provides similarity feedback containing photos with UID 3 and 4 even though those photos are not part of the original search results. The only limitation is that the client (or user) has the appropriate permissions to access the content provided in the feedback.

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="similarityFeedback" service="ca">
 <status>ok</status>
</response>

addTask (Front-end -> Analysis Back-end):

Example Query:

POST /rest/ca/addTask
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskDetails>
 <taskId>1</taskId>
 <taskType>FEEDBACK</taskType>
 <backendId>0</backendId>
 <userId>1</userId>
 <userConfidence>1</userConfidence>
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri>
 <mediaList>
  <media>
   <UID>1</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </mediaList>
 <similarMediaList>
  <media>
   <UID>2</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
   <UID>3</UID>
 <serviceId>0</serviceId>
 <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=3&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </similarMediaList>
 <dissimilarMediaList>
  <media>
   <UID>4</UID>
 <serviceId>0</serviceId>
 <userId>2</userId>
   <url>http://fe.example.org/rest/ca/r?uid=4&amp;service_id=0</url>
 <visibility>PRIVATE</visibility>
 <mediaType>PHOTO</mediaType>
  </media>
 </dissimilarMediaList>
</taskDetails>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addTask" service="ca">
 <status>ok</status>
</response>

3.9 Twitter Profile Summarization

This use case is for scheduling summarization tasks on the analysis back-ends. The tasks are generated automatically when user connects an external account with profile content, or when content of a previously connected account are synchronized with the front-end.

3.9.1 Diagrams

Twitter Profile Summarization

3.9.2 Examples

This example assumes that account connection has already been made using the Front-end. The connecting process is not explained as it is performed by The Front-end’s internal functionality, and thus, is out of scope of this documentation.

addTask (Front-end -> Summarizer):

Example Query:

POST /rest/ca/addTask
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskDetails>
 <taskId>1</taskId>
 <taskType>TWITTER_PROFILE_SUMMARIZATION</taskType>
 <backendId>0</backendId>
 <userId>1</userId>
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri>
 <profile>
  <userDetails>
    <bio>czhXX</bio>
    <favoritesCount>1992245036</favoritesCount>
    <followersCount>993252504</followersCount>
    <friendsCount>296699425</friendsCount>
    <twitterId>1</twitterId>
    <name>kfZmBq</name>
    <screenName>bBsc13UxU</screenName>
    <location>
      <name>wlRMa</name>
    </location>
    <userId>8367131745868141232</userId>
  </userDetails>
  <statusMessageList>
    <statusMessage>
      <from>S9QNEwFnR</from>
      <message>#BQtQp #KqT #rFA22 cg55 #su99 LIA 7PtLs #MAZv #4t9qBc6 zCMWL #AC1lyE #L29 XocJ9 9b6UzWWd0 #vUmUdItP aSZp5NuK KKnIfo soR MJVBXJC5 CRFvw0S</message>
      <updatedTimestamp>1991-04-06T09:58:19+0300</updatedTimestamp>
    </statusMessage>
  </statusMessageList>
  <photoDescriptionList>
    <photoDescription>
      <photoUID>55ba2f66-4f20-4843-9fba-0382c85f9ab1</photoUID>
      <createdTimestamp>2014-03-30T09:53:13+0300</createdTimestamp>
      <description>#83tGRK #AFT0 sI1Xi hWRNsR #PHT qWxF 3LB ZVP1n81P TIIM HUpp4Tp Mfxpw86eh WittXhz #WQ5MTP #bxabX #Uykk2RP #2iDgkKP xfyMR #uYfLyvP qJhZugRb3</description>
      <from>gf6UaPve</from>
      <source>vrYi</source>
      <location>
        <latitude>-22.0</latitude>
        <longitude>132.0</longitude>
      </location>
    </photoDescription>
  </photoDescriptionList>
  <videoDescriptionList>
    <videoDescription>
      <createdTimestamp>1980-03-06T23:28:45.293+02:00</createdTimestamp>
      <from>LLNb2Yq</from>
      <description>oCKh7Gz #tgJN0M7qm ez4s2 h4Zeo GiY #olfT8l pk9IuioPd #k3H7A gA6rPka 74pZSVy1t #DDefKiP0a #nz0 #eyxNl4I GVJgBmRL P53oo Y9Jz3L BNqEpkCwi</description>
      <location>
        <latitude>50.0</latitude>
        <longitude>88.0</longitude>
      </location>
    </videoDescription>
  </videoDescriptionList>
 </profile>
</taskDetails>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addTask" service="ca">
 <status>ok</status>
</response>

taskFinished (Summarizer -> Front-end):

Example Query:

POST /rest/ca/taskFinished
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskResults>
 <taskId>1</taskId>
 <taskType>TWITTER_PROFILE_SUMMARIZATION</taskType>
 <backendId>0</backendId>
 <status>COMPLETED</status>
 <message>Summarization completed successfully.</message>
 <objectList>
  <object>
   <objectId>123</objectId>
   <objectType>KEYWORD</objectType>
   <status>CANDIDATE</status>
   <backendId>0</backendId>
   <value>cute</value>
   <confidence>9.8058747237</confidence>
   <rank>0</rank>
  </object>
 </objectList>
 <mediaList>
  <media>
   <UID>0</UID>
   <objectList>
    <object>
     <objectId>124</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>biking</value>
     <confidence>4.58747237</confidence>
     <rank>0</rank>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskResults>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="taskFinished" service="ca">
 <status>ok</status>
</response>

3.10 User Feedback

When user updates the details of his or her photos, a feedback task may be generated by the front-end. The analysis back-ends may or may not respond to the given feedback. If the feedback affected photos that were not included in the original task, these can be added by the analysis back-end to the results (taskFinished method). If the analysis back-end does not have anything to respond, it does not need to call taskFinished method. Responding with a proper status code without results is accepted and will simply update the task status on the front-end.

3.10.1 Diagrams

User Feedback

3.10.2 Examples

updatePhotos (Client -> Front-end):

The feedback process is initialized by the user (or client) by calling updatePhotos on the front-end, which will trigger the generation of applicable Feedback Tasks, if needed. In this example the user has both rejected and confirmed a keyword. Note that the second object has name element, which can be used to declare clear text description of a keyword. This description can be added by the user, or by the system.

Example Query:

POST /rest/ca/updatePhotos
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<mediaList>
 <media>
  <UID>1</UID>
  <serviceId>0</serviceId>
  <userId>1</userId>
  <url>http://example.org/photo.jpg</url>
  <credits>Creative Commons</credits>
  <name>Photo</name>
  <description>ugly photo</description>
  <visibility>PRIVATE</visibility>
  <mediaType>PHOTO</mediaType>
  <objectList>
   <object>
    <objectId>123</objectId>
    <objectType>KEYWORD</objectType>
    <status>USER_REJECTED</status>
    <backendId>0</backendId>
    <value>cute</value>
    <confidence>1</confidence>
    <rank>0</rank>
    <visualObjectId>1</visualObjectId>
    <mediaType>PHOTO</mediaType>
   </object>
   <object>
    <objectId>124</objectId>
    <objectType>KEYWORD</objectType>
    <status>USER_CONFIRMED</status>
    <backendId>0</backendId>
    <value>ugly</value>
    <name>hideous</name>
    <confidence>1</confidence>
    <rank>0</rank>
    <visualObjectId>1</visualObjectId>
    <mediaType>PHOTO</mediaType>
   </object>
  </objectList>
 </media>
</mediaList>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="updatePhotos" service="ca">
 <status>ok</status>
</response>

addTask (Front-end -> Analysis Back-end):

Example Query:

POST /rest/ca/addTask
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskDetails>
 <taskId>1</taskId>
 <taskType>FEEDBACK</taskType>
 <backendId>0</backendId>
 <userId>1</userId>
 <userConfidence>1</userConfidence>
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri>
 <mediaList>
  <media>
   <UID>1</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=1&amp;service_id=0</url>
   <credits>Creative Commons</credits>
   <name>Photo</name>
   <description>ugly photo</description>
   <visibility>PRIVATE</visibility>
   <mediaType>PHOTO</mediaType>
   <objectList>
    <object>
     <objectId>123</objectId>
     <objectType>KEYWORD</objectType>
     <status>USER_REJECTED</status>
     <backendId>0</backendId>
     <value>cute</value>
     <confidence>1</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
    <object>
     <objectId>124</objectId>
     <objectType>KEYWORD</objectType>
     <status>USER_CONFIRMED</status>
     <backendId>0</backendId>
     <value>ugly</value>
     <name>hideous</name>
     <confidence>1</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskDetails>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addTask" service="ca">
 <status>ok</status>
</response>

taskFinished (Analysis Back-end -> Front-end):

Example Query:

POST /rest/ca/taskFinished
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskResults>
 <taskId>1</taskId>
 <taskType>FEEDBACK</taskType>
 <backendId>0</backendId>
 <status>COMPLETED</status>
 <message>Feedback affected other photos.</message>
 <mediaList>
  <media>
   <UID>2</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>
   <backendStatusList>
    <backendStatus>
     <backendId>0</backendId>
     <status>COMPLETED</status>
     <message>Keyword updated based on user feedback.</message>
    </backendStatus>
   </backendStatusList>
   <objectList>
    <object>
     <objectId>125</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>ugly</value>
     <confidence>0.75</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskResults>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="taskFinished" service="ca">
 <status>ok</status>
</response>

3.11 User Feedback - Face Recognition

Content-Based Analysis (Content-Based Analysis) combined with User Feedback (User Feedback). The only difference is in the transferred visual object metadata. As with the regular feedback, the back-end may (if it wants to update previously submitted metadata, or submit new metadata) or may not respond to the feedback provided by the user by calling taskFinished on the front-end. Responding with a proper status code without results is accepted and will simply update the task status on the front-end.

3.11.1 Diagrams

User Feedback Face

3.11.2 Examples

taskFinished (Analysis Back-end -> Front-end):

Note that the user feedback is not initiated by the taskFinished call from the analysis back-end. In the example case the first taskFinished call simply illustrates the fact that the original metadata have been (in this use case) been provided by a finished analysis task. The feedback task is generated based on the updated metadata provided by the user in the updatePhotos call. In this case, the initial metadata provided by the back-end is an unknown face, as shown in the example code below (the object element could also include an empty value element).

Example Query:

POST /rest/ca/taskFinished
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskResults>
 <taskId>0</taskId>
 <taskType>ANALYSIS</taskType>
 <backendId>0</backendId>
 <status>COMPLETED</status>
 <message>Analysis completed successfully.</message>
 <mediaList>
  <media>
   <UID>2</UID>
   <backendStatusList>
    <backendStatus>
     <backendId>0</backendId>
     <status>COMPLETED</status>
     <message>Recognized an unknown face.</message>
    </backendStatus>
   </backendStatusList>
   <objectList>
    <object>
     <objectId>124</objectId>
     <objectType>FACE</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <confidence>0.75</confidence>
     <rank>0</rank>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskResults>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="taskFinished" service="ca">
 <status>ok</status>
</response>

getPhotos (Client -> Front-end):

Note that the client could have received the metadata of the photo in question by any number of ways (such as a result of a search query), and getPhotos is used here only for an example.

Example Query:

GET /rest/ca/getPhotos?uid=2&datagroups=all

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="getPhotos" service="ca">
 <status>ok</status>
 <mediaList>
  <media>
   <UID>2</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>
   <visibility>PRIVATE</visibility>
   <mediaType>PHOTO</mediaType>
   <objectList>
    <object>
     <objectId>124</objectId>
     <objectType>FACE</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <confidence>0.75</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</response

updatePhotos (Client -> Front-end):

If the user wants to update the recognized face with a proper value (name, “Jack”, in the example), he can do so by using the updatePhotos method, as seen in the example below. User can also return other status information, such as “USER_REJECTED”, if he/she deems that the information provided by the back-end was incorrect.

Example Query:

POST /rest/ca/updatePhotos
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<mediaList>
 <media>
  <UID>2</UID>
  <serviceId>0</serviceId>
  <userId>1</userId>
  <visibility>PRIVATE</visibility>
  <mediaType>PHOTO</mediaType>
  <objectList>
   <object>
    <objectId>124</objectId>
    <objectType>FACE</objectType>
    <status>USER_CONFIMED</status>
    <backendId>0</backendId>
    <value>Jack</value>
    <confidence>1</confidence>
    <rank>0</rank>
    <visualObjectId>1</visualObjectId>
    <mediaType>PHOTO</mediaType>
   </object>
  </objectList>
 </media>
</mediaList>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="updatePhotos" service="ca">
 <status>ok</status>
</response>

addTask (Front-end -> Analysis Back-end):

Example Query:

POST /rest/ca/addTask
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskDetails>
 <taskId>1</taskId>
 <taskType>FEEDBACK</taskType>
 <backendId>0</backendId>
 <userId>1</userId>
 <userConfidence>1</userConfidence>
 <callbackUri>http://fe.example.org/rest/ca/taskFinished</callbackUri>
 <mediaList>
  <media>
   <UID>2</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=2&amp;service_id=0</url>
   <visibility>PRIVATE</visibility>
   <mediaType>PHOTO</mediaType>
   <objectList>
    <object>
     <objectId>124</objectId>
     <objectType>FACE</objectType>
     <status>USER_CONFIMED</status>
     <backendId>0</backendId>
     <value>Jack</value>
     <confidence>1</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
     <mediaType>PHOTO</mediaType>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskDetails>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="addTask" service="ca">
 <status>ok</status>
</response>

taskFinished (Analysis Back-end -> Front-end):

Based on the feedback, the analysis back-end may update the visual objects it has previously submitted by calling taskFinished. In the example, if visual object with objectId 125 already exists, it is updated, if not, new object is associated with the photo (with objectId=125). In the example, another face with value “Jack” has been found in another photo.

Example Query:

POST /rest/ca/taskFinished
Content-Type: text/xml; charset=UTF-8

[HTTP BODY STARTS]
<?xml version="1.0" encoding="UTF-8"?>
<taskResults>
 <taskId>1</taskId>
 <taskType>FEEDBACK</taskType>
 <backendId>0</backendId>
 <status>COMPLETED</status>
 <message>Feedback affected other photos.</message>
 <mediaList>
  <media>
   <UID>3</UID>
   <serviceId>0</serviceId>
   <userId>1</userId>
   <url>http://fe.example.org/rest/ca/r?uid=3&amp;service_id=0</url>
   <status>COMPLETED</status>
   <message>Face value updated based on user feedback.</message>
   <objectList>
    <object>
     <objectId>125</objectId>
     <objectType>KEYWORD</objectType>
     <status>CANDIDATE</status>
     <backendId>0</backendId>
     <value>Jack</value>
     <confidence>0.75</confidence>
     <rank>0</rank>
     <visualObjectId>1</visualObjectId>
    </object>
   </objectList>
  </media>
 </mediaList>
</taskResults>
[HTTP BODY ENDS]

Example Result:

<?xml version="1.0" encoding="UTF-8"?>
<response method="taskFinished" service="ca">
 <status>ok</status>
</response>
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.