RunwayML library for Processing

A library to easily use RunwayML with Processing.

Prerequisites

• RunwayML: Download the latest release of RunwayML and sign up for an account. Visit our installation guide for more details. If you encounter any issues installing RunwayML, feel free to contact us through the support page.

• Processing: Processing version 3.0 or greater is required.

Installation

1. Download RunwayML.zip
2. Unzip into Documents > Processing > libraries
3. Restart Processing (if it was already running)

📽 Watch How to Install and Use the RunwayML Processing Library

Testing installation

Before using the library please make sure RunwayML is started and is running one model.

You can test by opening your browser and navigating to http://localhost:8000/info to view info about your model.

If you see a JSON string with inputs and outputs information use the Basic Example below !

Feel free to test the Basic Example below.

Basic Example

For this example run the im2txt model.

This example will print information from RunwayML (e.g. running im2txt)

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  println(runwayData);
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Notice the text running displayed in console.

What if we want to display an image from RunwayML ?

Let's stop im2txt and use CygleGAN. Notice on info event details about the model: the output in particular is an image with the key "image". (You can see the information within RunwayML on the top right Network area of the Model).

Images to/from RunwayML are Base64 encoded and the library's ModelUtils helps convert between this format and the familiar PImage format.

With the CycleGAN model running we can tweak the above example to display an image received from RunwayML:

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;
// reference to image received
PImage resultImage;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
	// if there's an image received:
	if(resultImage != null){
		// display the image
		image(resultImage,0,0);
	}  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  // try to access "image" key and convert the received that to a PImage
  try{
  	String base64Image = runwayData.getString("image");
  	resultImage = ModelUtils.fromBase64(base64Image);
  }catch(Exception e){
  	// print an error message if the above fails
  	println("error parsing received data: " + runwayData);
  	e.printStackTrace();
  }
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Existing Examples

You can access the following examples via Processing > Examples > Contributed Libraries > RunwayML

• AdaIN-Style-Transfer (content image: "Still Life with fruit dish" by Paul Cézanne, style image: "Fruit Dish" by Georges Braques)

• Adaptive-Style-Transfer (content image: "Fruit Dish" by Georges Braques, inference using Cézanne (landscapes) style)

• Arbitrary-Image-Stylization (content image: "Girl with a Mandolin" by Pablo Picasso, style image: "Man with a guitar" by Georges Braques)

• AttnGAN:

• BigGAN:

• cnOCR:

• COCO-SSD: (content image: "The Card Players" by Paul Cézanne)

• CycleGAN: (content image: "Fruit Dish" by Georges Braques)

• DeepLab: (content image: "The Card Players" by Paul Cézanne)

• DeepLabV3: (content image: "The Card Players" by Paul Cézanne)

• DeepPrivacy: (image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)

• DenseCap: (content image: "The Card Players" by Paul Cézanne)

• DenseDepth: (content image: "The Card Players" by Paul Cézanne)

• DensePose: (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

• DeOldify: (content image: Grace Hopper, image credits: Courtesy of the Computer History Museum, source: https://news.yale.edu/2017/02/10/grace-murray-hopper-1906-1992-legacy-innovation-and-service)

• DeRaindrop: (content image: "Sudden shower over Shin Ōhashi bridge and Atake" by Hiroshige)

• EdgeConnect: (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

• ESRGAN: (image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)

• Face-Landmarks: (content image: "Celestina" by Pablo Picasso)

• Face-Parser: (content image: "Celestina" by Pablo Picasso)

• Face-Recognition: (image from 12 Angry Men (1957) directed by Sidney Lumet)

• Fast-Photo-Style (content image: "Still Life with fruit dish" by Paul Cézanne, style image: "Fruit Dish" by Georges Braques)

• Fast-Style-Transfer: (content image: "Nude Descending a Staircase, No. 2" by Marcel Duchamp)

• GLOW: (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

• GPT-2:

• im2txt: (content image: "The Card Players" by Paul Cézanne)

• Image-Inpainting-GMCNN: (content image: "Study after Velázquez's Portrait of Pope Innocent X" by Francis Bacon)

• Image-Super-Resolution: (image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)

• MaskRCNN: (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

• MobileNet: (content image: "Still Life with fruit dish" by Paul Cézanne)

• OpenGPT-2:

• OpenPifPaf-Pose: (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

• Photo-Sketch:

• Pix2Pix: (content image: "Broadway Boogie Woogie" by Piet Mondrian)

• Places365: (image credits: "London: The Thames from Somerset House Terrace towards the City" by Canaletto)

• PoseNet: (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

• SPADE-COCO: (uses segmentation based on "The Card Players" by Paul Cézanne)

• SPADE-Landscapes: (uses segmentation based on "The Card Players" by Paul Cézanne)

• Style2Paints: (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

• StyleGAN: (image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)

• YOLACT: (content image: "The Card Players" by Paul Cézanne)

• UGATIT: (content image: "Still Life with fruit dish" by Paul Cézanne)

Extra platforms

Additionally you can use RunwayML on a machine on the local network and connect from another device capable of running Processing.

Raspberry PI

As long as you have a Raspberry PI running Processing, all you need to do is specify the IP (and optionally port) of the computer running the RunwayML app: that's it! (e.g. if your computer's LAN IP is 192.168.0.12 you would initialize Runway like so: runway = new RunwayHTTP(this,"192.168.0.12",8000);)

To setup Processing on a Raspberry Pi please follow the official Get Started tutorial

Android

1. please install Android Mode for Processing (if it's not already installed on your system).
2. allow INTERNET permissions in Processing > Android > Sketch Permissions
3. run the code (e.g. attnGANAndroid)

The one major caveat at the moment is the Base64 <-> PImage conversion is handled by a separate class: ModelUtilsAndroid.

This means on Android

PImage result = ModelUtils.fromBase64(base64ImageString);

becomes:

PImage result = ModelUtilsAndroid.fromBase64(base64ImageString);

(Additionally, it's even possible to write RunwayML Processing sketchs on the Android device using APDE)

Dependencies

This Processing Library manages the OSC connection to Runway relying on the oscP5 library.

Please install oscP5 before using the OSC connection with this library via Sketch > Import Library... > Add Library... > Contribution Manager Filter > oscP5

Contributing

This is still a work in progress. Contributions are welcomed!

Credits

Special thanks for mentoring and support from Cris Valenzuela, Anastasis Germanidis and Daniel Shiffman

Additional thanks to Jen Sykes for updating the DenseCap and GPT2 examples

Main library development by George Profenza

Library examples are based partially on Runway Processing Examples by:

• Cris Valenzuela
• Anastasis Germanidis
• Gene Kogan
• maybay21
• JP Yepez
• Joel Matthys
• Alejandro Matamala Ortiz