Skip to content
Java CSS HTML
Branch: master
Clone or download
cvalenzuela Merge pull request #9 from orgicus/develop
library properties and README updates
Latest commit ce80ec9 Nov 13, 2019

README.md

RunwayML library for Processing

A library to easily use RunwayML with Processing.

Prerequisites

  • RunwayML: Download the latest release of RunwayML and sign up for an account. Visit our installation guide for more details. If you encounter any issues installing RunwayML, feel free to contact us through the support page.

  • Processing: Processing version 3.0 or greater is required.

Installation

  1. Download RunwayML.zip
  2. Unzip into Documents > Processing > libraries
  3. Restart Processing (if it was already running)

📽 Watch How to Install and Use the RunwayML Processing Library

Testing installation

Before using the library please make sure RunwayML is started and is running one model.

You can test by opening your browser and navigating to http://localhost:8000/info to view info about your model.

If you see a JSON string with inputs and outputs information use the Basic Example bellow !

Feel free test the Basic Example bellow.

Basic Example

For this example run the im2txt model.

This example will print information from RunwayML (e.g. running im2txt)

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  println(runwayData);
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Notice the text running displayed in console.

What if we want to display an image from RunwayML ?

Let's stop im2txt and use CygleGAN. Notice on info event details about the model: the output in particular is an image with the key "image". (You can see the information within RunwayML on the top right Network area of the Model).

Images to/from RunwayML are Base64 encoded and the library's ModelUtils helps convert between this format and the familiar PImage format.

With the CycleGAN model running we can tweak the above example to display an image received from RunwayML:

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;
// reference to image received
PImage resultImage;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
	// if there's an image received:
	if(resultImage != null){
		// display the image
		image(resultImage,0,0);
	}  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  // try to access "image" key and convert the received that to a PImage
  try{
  	String base64Image = runwayData.getString("image");
  	resultImage = ModelUtils.fromBase64(base64Image);
  }catch(Exception e){
  	// print an error message if the above fails
  	println("error parsing received data: " + runwayData);
  	e.printStackTrace();
  }
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Existing Examples

You can access the following examples via Processing > Examples > Contributed Libraries > RunwayML

  • AdaIN-Style-Transfer preview (content image: "Still Life with fruit dish" by Paul Cézanne, style image: "Fruit Dish" by Georges Braques)

  • Adaptive-Style-Transfer preview (content image: "Fruit Dish" by Georges Braques, inference using Cézanne (landscapes) style)

  • Arbitrary-Image-Stylization preview (content image: "Girl with a Mandolin" by Pablo Picasso, style image: "Man with a guitar" by Georges Braques)

  • AttnGAN:preview

  • BigGAN: preview

  • cnOCR: preview

  • COCO-SSD: preview (content image: "The Card Players" by Paul Cézanne)

  • CycleGAN: preview (content image: "Fruit Dish" by Georges Braques)

  • DeepFill: preview (content image: "Study after Velázquez's Portrait of Pope Innocent X" by Francis Bacon)

  • DeepLab: preview (content image: "The Card Players" by Paul Cézanne)

  • DeepLabV3: preview (content image: "The Card Players" by Paul Cézanne)

  • DeepPrivacy: preview (image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)

  • DenseCap: preview (content image: "The Card Players" by Paul Cézanne)

  • DenseDepth: preview (content image: "The Card Players" by Paul Cézanne)

  • DensePose: preview (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

  • DeOldify: preview (content image: Grace Hopper, image credits: Courtesy of the Computer History Museum, source: https://news.yale.edu/2017/02/10/grace-murray-hopper-1906-1992-legacy-innovation-and-service)

  • DeRaindrop: preview (content image: "Sudden shower over Shin Ōhashi bridge and Atake" by Hiroshige)

  • EdgeConnect: preview (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

  • ESRGAN: preview (image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)

  • Face-Landmarks: preview (content image: "Celestina" by Pablo Picasso)

  • Face-Parser: preview (content image: "Celestina" by Pablo Picasso)

  • Face-Recognition: preview (image from 12 Angry Men (1957) directed by Sidney Lumet)

  • Fast-Photo-Style preview (content image: "Still Life with fruit dish" by Paul Cézanne, style image: "Fruit Dish" by Georges Braques)

  • Fast-Style-Transfer: preview (content image: "Nude Descending a Staircase, No. 2" by Marcel Duchamp)

  • GLOW: preview (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

  • GPT-2: preview

  • im2txt: preview (content image: "The Card Players" by Paul Cézanne)

  • Image-Inpainting-GMCNN: preview (content image: "Study after Velázquez's Portrait of Pope Innocent X" by Francis Bacon)

  • Image-Super-Resolution: preview (image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)

  • MaskRCNN: preview (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

  • MobileNet: preview (content image: "Still Life with fruit dish" by Paul Cézanne)

  • Model3DDFA: preview (image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)

  • OpenGPT-2: preview

  • OpenPifPaf-Pose: preview (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

  • Photo-Sketch: preview

  • Pix2Pix: preview (content image: "Broadway Boogie Woogie" by Piet Mondrian)

  • Pix2Pix Facemarks2Portrait: preview

  • Places365: preview (image credits: "London: The Thames from Somerset House Terrace towards the City" by Canaletto)

  • PoseNet: preview (image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)

  • SPADE-COCO: preview (uses segmenation based on "The Card Players" by Paul Cézanne)

  • SPADE-Landscapes: preview (uses segmenation based on "The Card Players" by Paul Cézanne)

  • Style2Paints: preview (content image: "Self-Portrait with Striped Shirt" by Egon Schiele)

  • StyleGAN: preview

  • VisualImportance: preview (image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)

  • YOLACT: preview (content image: "The Card Players" by Paul Cézanne)

  • UGATIT: preview (content image: "Still Life with fruit dish" by Paul Cézanne)

Dependencies

This Processing Library manages the OSC connection to Runway relying on the oscP5 library.

Please install oscP5 before using the OSC connection with this library via Sketch > Import Library... > Add Library... > Contribution Manager Filter > oscP5

Contributing

This is still a work in progress. Contributions are welcomed!

Credits

Special thanks for mentoring and support from Cris Valenzuela, Anastasis Germanidis and Daniel Shiffman

Main library developement by George Profenza

Library examples are based partially on Runway Processing Examples by:

You can’t perform that action at this time.