Skylink is a sample application that connects drones to the cloud with near realtime image analysis leveraging IBM Cloudant, OpenWhisk, IBM Watson, and IBM Bluemix.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
client/skylink
github-assets Revert "added swift backend imlpementation" Aug 23, 2016
processing Revert "added swift backend imlpementation" Aug 23, 2016
training Revert "added swift backend imlpementation" Aug 23, 2016
web Update PORT env var Jan 6, 2017
.gitignore
LICENSE Initial commit Apr 28, 2016
README.md updated for 3.2 sdk Aug 11, 2016

README.md

skylink

Skylink is a sample application that lets you connect a DJI drone aircraft to the IBM Cloud with near realtime image analysis leveraging IBM Cloudant, OpenWhisk, IBM Watson, and Alchemy Vision.

This is a proof of concept use case that demonstrates how low cost/commodity off-the-shelf consumer drones can be integrated into enterprise workflows immediately, leveraging existing technology and infrastructure.

You can read more about the context and usage of this application in this blog post on the Bluemix blog, or by viewing this free webinar that walks through the entire solution.

Check out the video below for a quick overview how the application works.

video poster image

https://youtu.be/BHmo8oWENJo

General Architecture

The application connects a DJI drone aircraft to IBM Bluemix by using an iPad to bridge the connection from aircraft to the external network & cloud services. The aircraft remote control connects directly to the controller using a USB connection. This allows the aircraft to send a live video stream, captured media, and telemtry data directly to an app running on the iPad. This also allows the iPad to send control instructions to the aircraft, enabling the app to control what the aircraft is doing. All communication back and forth between the aircraft and app on the iPad is handled using DJI's developer toolkit.

screenshot of skylink app running on iPad

The app captures aircraft telemtry data and images (either full resolution images or video frame grabs) and stores them locally on the iPad using Cloudant Sync - This prevents data loss if you are flying in an area without any network connectivity. When there is data connectivity, the data is automatically replicated up to the Cloudant service. Saving data into Cloudant automatically triggers OpenWhisk actions to process the images and data using Watson Visual Recognition and Alchmey Vision services. Once all the data has been processed, it is available through a web interface powered by Node.js running on IBM Bluemix.

The Aircraft

All development was done using a DJI Phantom 3 Advanced aircraft, though this should also work with DJI Inspire series aircraft. This aircraft is relatively low cost (under $1000), is very easy to fly by leveraging GPS assisted & stabilized flight, delivers high quality stabilized HD video and high resolution still images, has a range over 1 mile, flight times over 20 minutes per battery, and is immediately ready to fly "out of the box".

DJI Phantom


Application Requirements

You might notice references to 'overwatch' throughout this project. That was the original project name, but let's face it, 'overwatch' makes you think of a creepy big-brother concept, so I changed it. Unfortunately, there are still references to the old name throughout the code.

Bluemix Services

Create a new Node.js app on Bluemix, and configure it to use the following services:

screenshot of bluemix app configuration

Cloudant Database

Once you've configured your app on Bluemix to use the Cloudant NoSQL database, you'll next need to configure it.

  1. First, create a new database called overwatch
  2. Next, create a view on the database with the index name overwatch.images, and use the following map function:
function (doc) {
    if ( doc.type == 'overwatch.image' ) {
      emit(doc);
    }
} 

screenshot of cloudant view

iOS Client Project

The iOS application is the bridge from aircraft to the cloud. The iPad connects directly to the aircraft controller over USB, and can communicate to the cloud using wifi or 3G/4G cellular networks. It provides a first-person video view, lets you view telemetry data, and has two options for capturing images from the aircraft:

  • Capture a full resolution (4000x3000) image using the camera funcionality (slow and requires interruption to live stream)
  • Capture a screen grab from the live video stream (720p MAX, low quality) (very fast, but not as high quality)

screenshot of skylink app running on iPad

The iOS Client project leverages DJI's developer SDK to communicate with the aircraft. In particular, this app was developed using version 3.2.

Note: It is very important to make sure that the aircraft firmware is in synch and using the same version of the SDK, otherwise aircraft features may not operate as expected.

To build the iOS Client, download the source for this project and navigate to the client folder. It is required to copy resources from the DJI SDK before the iOS project will compiler properly. You will also need to create an "app" on DJI's portal before you will be able to communicate with the aircraft.

The following resource need to be copied from the SDK download into the client project before you will be able to compile:

File SDK Location iOS Project Location
DJIBaseViewController.swift /Sample Code/SwiftSampleCode/DJISDKSwiftDemo/Missions /client/skylink/DJI-SDK/DJI-Swift
DJISDK.framework /Sample Code/ /client/skylink/DJI-SDK
Video Previewer.xcodeproj /Sample Code/VideoPreviewer/ /client/skylink/DJI-SDK
Video Previewer (directory) /Sample Code/VideoPreviewer/ /client/skylink/DJI-SDK

For more detail about compiling/building with the DJI SDK, check out the docs.

You will need to run the Cocoapods install process to pull down required dependencies:

pod install

Important parts of the iOS Client Project:

File Details
Overwatch/DataManager.swift Handles data persistence, writing data to the Cloudant Sync local store, and replication to the Cloudant service.
ViewController.swift Handles general UI interactivity and displaying data from the aircraft SDK

OpenWhisk Analysis

The OpenWhisk actions that are used to process these images are forked from the Dark Vision app created by my colleage Frederick Lavigne. If you have not yet seen this applicaiton, you really should check it out... it's quite impressive.

All code for the OpenWhisk actions are contained within the processing directory.

The OpenWhisk actions are automatically triggered by changes to the Cloudant database. The general flow is:

  • Data change invokes analysis actions
  • If the data is new & not analyzed, it will follow this sequence:
    • Retrieve the complete document from Cloudant
    • Retrieve the image attachment binary file
    • Generate a thumbnail image
    • Analyze the image (in parallel):
      • Alchemy Facial Recognition
      • Alchemy Image Keywords
      • Watson Visual Recognition
    • Write analysis results back to Cloudant
    • Save the image thumbnail back to Cloudant
    • Generate images for each detected face and save to Cloudant

You must have the OpenWhisk CLI already installed.

To install the OpenWhisk actions, first edit the local.env file to contain authentication credentials for your Bluemix services.

Next run the following command to create the trigger for the Cloudant database, and the actions required to complete the image analysis:

./skylink.sh --install 

There are also methods in the shell script to uninstall or reinstall these actions, if desired.

Node.js Web Client

You can run the web applicaiton locally, or push it to your Node.js instance(s) on Bluemix. All code for the Node.js app is contained within the web directory.

screenshot of skylink web data in the browser

Running Node.js Locally

To run the application locally, you first need to download dependencies by running:

npm install

You only need to do this next step if you are running locally, you do not need to configure credentials if running on Bluemix. Next, add authentication credentials to the configuration.json file. You can get these by clicking on the "Show Credentials" link for each service in your app on Bluemix. In the old Bluemix interface these are on the main app landince screen. In the new Bluemix interface, these are avilable under the "Connections" tab.

Launch the app:

node app.js

Running Node.js on Bluemix

To run the Node.js application on Bluemix, you only need to push the code to the Bluemix applicaiton instance. From inside the web directory, run the following command to push the app up to Bluemix:

cf push

This will use the app name configued within manifest.yml

Contribute

Please create a pull request with your desired changes.

Troubleshooting

OpenWhisk

Polling activations is good start to debug the OpenWhisk action execution b/c it let's you see console.log() output for most actions. Just run the following command in a terminal window (with the OpenWhisk CLI installed):

wsk activation poll

When data is saved into Cloduant, the OpenWhisk actions will automatically be invoked, and you will be able to view debug output for the processing steps.

Web application

Either use the Bluemix web user intefact to view debug logs, or use the Cloud Foundry CLI to look view live logs for the Node.js application running on Bluemix:

cf logs <appname>

License

See License.txt for license information.

Privacy Notice

The web application includes code to track deployments to IBM Bluemix and other Cloud Foundry platforms. The following information is sent to a Deployment Tracker service on each deployment:

  • Application Name (application_name)
  • Space ID (space_id)
  • Application Version (application_version)
  • Application URIs (application_uris)

This data is collected from the VCAP_APPLICATION environment variable in IBM Bluemix and other Cloud Foundry platforms. This data is used by IBM to track metrics around deployments of sample applications to IBM Bluemix to measure the usefulness of our examples, so that we can continuously improve the content we offer to you. Only deployments of sample applications that include code to ping the Deployment Tracker service will be tracked.

Disabling Deployment Tracking

Deployment tracking can be disabled by removing require("cf-deployment-tracker-client").track(); from the beginning of the web/app.js file.