Skywriter SpaceBrew Unity Tutorial: (The allegory of the Virtual Cave with real rubber duck)

RinaShu edited this page Jul 2, 2018 · 21 revisions

Concept

The concept is to create the experience of acknowledging the virtual world in the real world without looking into screens. There is a virtual maze and a real robot Duck. So, you need to figure out how to get out of the virtual maze that you can't see but you can feel it as a real Duck will stop when it hits the walls of the maze. For the demo, we built only one wall of the maze to demonstrate the idea and show how to work with software and hardware.

Materials

  • 1x Skywriter HAT - Pimoroni Detect position data and gesture information (flicks, taps) and the motion of your hand with X, Y and Z positions. Skywriter HAT Python library https://github.com/pimoroni/skywriter-hat
  • 2x Raspberry Pi 3 (Model B)
  • 1x Adafruit 16-Channel PWM/Servo HAT & Bonnet for Raspberry Pi
  • 2x Continuous Rotation Servos

Skywriter

The Skywriter HAT can sense position and a few gestures using electrical near-field 3D. The outputs you can expect to receive out of the box are (a) position of a object in (x,y,z), flicks (up, down, left, right), ‘air wheels’, taps, touches and double taps. To access these easily we downloaded the pre-made examples to our RPI using “git clone https://github.com/pimoroni/skywriter-hat”, the read me details all the python libraries and files we needed and how to load them on RPI terminal.

  • NOTE: We encountered known issues with “sudo pip install autopy” that we have yet to resolve. However the basic test.py functions without autopy and can be repurposed for your project *

Off the bat we noticed that the resolution of the (x,y,z) position coordinates that the skywriter returned were very good but the range in 3D space above the hat is very narrow. For us air wheels, taps, touches and double taps were fairly unreliable but flicks gave us robust signals reliably. Knowing we wanted a responsive control input for the duck and that SpaceBrew does not like too many inputs at once, we opted to use the flicks as the controller.

Skywriter/RPI(1) to SpaceBrew

We pulled the test.py from the Skywriter examples and refitted it with a publish script from the native SpaceBrew examples. The python script below is what we used to publish our flicks to Spacebrew.

#!/usr/bin/env python

import signal
import skywriter

import sys
import time
import subprocess
from pySpacebrew.spacebrew import Spacebrew

brew = Spacebrew("Bird & Worm", description="publish flicks",  server="sandbox.spacebrew.cc", port=9000)
brew.addPublisher("move", "string")
connected = False

CHECK_FREQ = 2 # check mail every 60 seconds
CURR_INDEX = 0 # Create a queue of items

print("sweet mercy, someone find me out here in cyberspace")

try:
    brew.start()
    while True:
        if (connected == True):
            @skywriter.flick()
            def flick(start,finish):
                print('Got a flick!', start, finish)
                if ((start == "north") and (finish == "south")):
                    brew.publish("move", "down")
                if ((start == "south") and (finish == "north")):
                    brew.publish("move", "up")
                if ((start == "west") and (finish == "east")):
                    brew.publish("move", "right")
                if ((start == "east") and (finish == "west")):
                    brew.publish("move", "left")
        connected = True
        time.sleep(CHECK_FREQ)
    signal.pause()

finally:
    brew.stop()

SpaceBrew to Unity

Once we built our unity file with the proper “maze” (or in our case just a wall) with all the proper physics and collision events it was time to subscribe to the flicks of the skywriter to move the virtual object. The following c# script contains the code to listen to Spacebrew – this will involve a new script (that we call littleguy.cs) and an edit to SpacebrewEvents.cs. It is important to create a SpacebrewClient component in your space brew objects with all the proper naming and structure – the following photo shows how we filled that out.

**littleguy.cs **

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class littleguy : MonoBehaviour {

    public float moveSpeed;
    public string virtualstep = "none"; 
	GameObject remoteBrew;
	SpacebrewEvents brewbrewSpace;
    bool shouldMove = true; 

	// Use this for initialization
	void Start () {
        print("yo I'm your guy"); 
        moveSpeed = 100f;
        // GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); // the name of your client object
	    // SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
	}
	
	// Update is called once per frame
	void Update () {
        if (virtualstep == "left") {
            GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); 
            SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
            brewbrewSpace.moveL(); 
            virtualstep = "none"; 
        }
        if (virtualstep == "right") {
            GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); 
            SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
            brewbrewSpace.moveR(); 
            virtualstep = "none"; 
        }
        if (virtualstep == "forward") {
            if (shouldMove) {
                GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); 
                SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
                brewbrewSpace.moveFore(); 
                virtualstep = "none"; 
            }
        }
            if (virtualstep == "back") {
            GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); 
            SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> (); 
            brewbrewSpace.moveBackward(); 
            virtualstep = "none"; 
        }
	}

    public void moveLeft() {
        print("moveLEFT");
        transform.Translate(Vector3.left * moveSpeed * Time.deltaTime);
        virtualstep = "left"; 
    }

    public void moveRight()
    {
        print("moveRIGHT");
        transform.Translate(Vector3.right * moveSpeed * Time.deltaTime);
        virtualstep = "right"; 
    }
    public void moveForward()
    {
        print("moveFORWARD");
        transform.Translate(Vector3.forward * moveSpeed * Time.deltaTime);
        virtualstep = "forward"; 
    }
    public void moveBack()
    {
        print("moveBACK");
        transform.Translate(Vector3.back * moveSpeed * Time.deltaTime);
        virtualstep = "back"; 
    }
    void OnCollisionEnter(Collision collision) {
        if (collision.gameObject.name == "theWall") {            
            Debug.Log ("Hit a wall");
            shouldMove = false; 
            //virtualstep = "back"; 
            }
        }
}

**littleguy.cs **

using UnityEngine;
using System.Collections;

public class SpacebrewEvents : MonoBehaviour {

	SpacebrewClient sbClient;
	bool lightState = false;

	// Use this for initialization
	void Start () {
		GameObject go = GameObject.Find ("SpacebrewObject"); // the name of your client object
		sbClient = go.GetComponent <SpacebrewClient> ();

		sbClient.addEventListener (this.gameObject, "move");
	}

	// Update is called once per frame
	void Update () {
		if (Input.GetKeyDown ("space")) {
			print ("Sending Spacebrew Message");
			// name, type, value
			// COMMON GOTCHA: THIS MUST MATCH THE NAME VALUE YOU TYPED IN THE EDITOR!!
			sbClient.sendMessage("buttonPress", "boolean", "true");
		}
		foreach (char c in Input.inputString) {
			print("Just pressed: "+c.ToString());
			if (c >= 'a' && c <= 'z') {
				sbClient.sendMessage("letters", "string", c.ToString());
				GameObject go = GameObject.Find ("MatrixContainer"); // the name of your client object
				MatrixMaker grid = go.GetComponent <MatrixMaker> ();
				grid.ParseIncomingLetter(c.ToString());
				grid.delayLayer();
				//grid.CreateLayer(true);
				
			}
		}

		if (Input.touchCount > 0 && Input.touches[0].phase == TouchPhase.Began) {
			sbClient.sendMessage("buttonPress", "boolean", "true");
		}
	}

	public void moveFore() {
		sbClient.sendMessage("move", "string", "forward");
	}
	public void moveBackward() {
		sbClient.sendMessage("move", "string", "back");
	}
	public void moveL() {
		sbClient.sendMessage("move", "string", "left");
	}
	public void moveR() {
		sbClient.sendMessage("move", "string", "right");
	}
	
	public void OnSpacebrewEvent(SpacebrewClient.SpacebrewMessage _msg) {

		// print ("Received Spacebrew Message");
		// print("message name" + _msg.name); 
		// print("value" + _msg.value); 

        // skywriter flicks test
        if (_msg.name == "move") {
            if (_msg.value == "up") {
                //print("up detected");
                GameObject go = GameObject.Find("littleguy");
                littleguy s = go.GetComponent <littleguy> ();
                s.moveForward();
            }
            if (_msg.value == "down")
            {
                print("down detected");
                GameObject go = GameObject.Find("littleguy");
                littleguy s = go.GetComponent <littleguy> ();
                s.moveBack();
            }
            if (_msg.value == "left")
            {
                print("left detected");
                GameObject go = GameObject.Find("littleguy");
                littleguy s = go.GetComponent <littleguy> ();
                s.moveLeft();
            }
            if (_msg.value == "right")
            {
                print("right detected");
                GameObject go = GameObject.Find("littleguy");
                littleguy s = go.GetComponent <littleguy> ();
                s.moveRight();
            }
        }
    }
}
  • NOTE: Remember your noodles (in Spacebrew) *

Unity

In Unity, you need to create the maze (in our case only one wall) and a virtual avatar of robot Duck (Virtual Object), which can be just a sphere or any object. It is required to add Rigidbody to both objects. The mass of the wall should be much bigger than the mass of the avatar (100:1). Add gravity to both the avatar and the wall. Than add plane as a ground to place objects with gravity on it. Later you can disable plane (make it invisible) in Inspector.

To move the avatar, you will use data from Skywriter HAT, but just for testing, you can use KeyArrows. Code for it:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class MovingObjectCube : MonoBehaviour {
public float moveSpeed=100f;
public float moveTime = 0.1f; 
bool shouldMove = true;

void Start () {
        }
void Update () {
	if (Input.GetKey (KeyCode.DownArrow)&&shouldMove) {
	transform.Translate  (Vector3.back * moveSpeed * Time.deltaTime) 
	print ("move back"); 
	}
	if (Input.GetKey (KeyCode.UpArrow)&&shouldMove) {
        transform.Translate  (Vector3.forward * moveSpeed * Time.deltaTime) 
	print ("move forward"); 
	}
	if (Input.GetKey (KeyCode.LeftArrow)&&shouldMove) {
        transform.Translate  (Vector3.left * moveSpeed * Time.deltaTime);
         print ("move left");    
         }
    	if (Input.GetKey (KeyCode.RightArrow)&&shouldMove) {
            transform.Translate  (Vector3.right * moveSpeed * Time.deltaTime);
            print ("move right")
	 }
}

To stop the object (the avatar) when it hits another object (the wall), you need to use OnCollisionEnter script:

void OnCollisionEnter(Collision collision) {
    if (collision.gameObject.name == "Wall") {            
    Debug.Log ("Hit Wall");
    shouldMove = false; 
    }
}

Unity to SpaceBrew to RPI(2)/PMW Servo Shield

This is where things all come together, or fall apart. For much better references on how the Adafruit PMW Servo hat work see the servo examples in the wiki (i.e. hotdog not hotdog). However on the Adafruit PMW Servo hat GitHub you can download the examples by “git clone https://github.com/adafruit/Adafruit_Python_PCA9685.” Suffice to say we tested a few values that got our duck to move close to forward, backward, left and right without understanding the PMW logic. We wrote a script for the RPI to receive values from Spacebrew and translate them into servo moves.

  • NOTE: Beware the cold solder *
from __future__ import division

import sys
import time
from pySpacebrew.spacebrew import Spacebrew

import Adafruit_PCA9685
pwm = Adafruit_PCA9685.PCA9685()

# Configure min and max servo pulse lengths
servo_min = 0  # Min pulse length out of 4096
servo_max = 3000  # Max pulse length out of 4096

letter = "emptystring"
CHECK_FREQ = 0.2

brew = Spacebrew("duckfeed", description="duckweed",  server="sandbox.spacebrew.cc", port=9000)
brew.addSubscriber("thebeak", "string")
brew.addPublisher("eyeoftheduck", "string")

def handleString(value):
    global letter
    letter = value
    print(letter)
    print("this is a duck prayer")

brew.subscribe("thebeak", handleString)

def set_servo_pulse(channel, pulse):
    pulse_length = 1000000    # 1,000,000 us per second
    pulse_length //= 60       # 60 Hz
    print('{0}us per period'.format(pulse_length))
    pulse_length //= 4096     # 12 bits of resolution
    print('{0}us per bit'.format(pulse_length))
    pulse *= 1000
    pulse //= pulse_length
    pwm.set_pwm(channel, 0, pulse)

pwm.set_pwm_freq(60)

try:
    
    print("Press Ctrl-C to quit.")
    brew.start()
    
    while True:
        if (letter == "forward"):
            pwm.set_pwm(3, 0, 200)
            pwm.set_pwm(0, 0, 3000)
            time.sleep(1)
            pwm.set_pwm(3, 0, servo_min)
            pwm.set_pwm(0, 0, servo_min)
            letter = "null"
        if (letter == "back"):
            pwm.set_pwm(3, 0, 3000)
            pwm.set_pwm(0, 0, 3000)
            time.sleep(2)
            pwm.set_pwm(3, 0, 200)
            pwm.set_pwm(0, 0, 3000)
            time.sleep(1)
            pwm.set_pwm(3, 0, servo_min)
            pwm.set_pwm(0, 0, servo_min)
            letter = "null"
        if (letter == "left"):
            pwm.set_pwm(3, 0, 300)
            pwm.set_pwm(0, 0, 300)
            time.sleep(1)
            pwm.set_pwm(3, 0, 200)
            pwm.set_pwm(0, 0, 3000)
            time.sleep(1)
            pwm.set_pwm(3, 0, servo_min)
            pwm.set_pwm(0, 0, servo_min)
            letter = "null"
        if (letter == "right"):
            pwm.set_pwm(3, 0, 1000)
            pwm.set_pwm(0, 0, 1000)
            time.sleep(1)
            pwm.set_pwm(3, 0, 200)
            pwm.set_pwm(0, 0, 3000)
            time.sleep(1)
            pwm.set_pwm(3, 0, servo_min)
            pwm.set_pwm(0, 0, servo_min)
            letter = "null"
        if (letter == "null"):
            pwm.set_pwm(3, 0, servo_min)
            pwm.set_pwm(0, 0, servo_min)
        time.sleep(CHECK_FREQ)

finally:
    brew.stop()

Challenges, Traps & future directions.

**First and very foremost – unless you're comfortable with internet protocol, OSC/UDP/Websockets and processing – We wouldn't recommend trying to connect a MKR1000 to Spacebrew via WiFi. **

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.