Skip to content

Day 4

Jump to bottom
Adesign92 edited this page Jun 22, 2018 · 21 revisions

The plan for Thursday morning was clear - debugging the script and finalise the concept of code. We already knew that we will use Unity to create scenes > ArKit to create AR environment > XCode to compile apps on mobile devices > Spacebrew.cc to communicate between devices and create the shared AR experience.

Our biggest questions we solved today:

  1. Camera access/authorization

    During the week we faced several times a camera boot problem with custom unity scenes uploaded on the iPads. We discovered that the problem was due to a lack of description for the camera usage in Unity, this information is essential to iOS to boot the camera. So to specify the camera usage in Unity you have to go in Edit > Project Settings > Player and in the Other Settings tab add a description in the Camera Usage Description box.

  2. Serialize particle position and parse it back.

    In order to have a shared experience we had to send the position of every particle drawn from one iPad to the other iPad. To do that we first took the x y and z position from the Vector3 of last particle drawn from the drawing iPad and convert them into a String to be uploaded on Spacebrew, so we changed the ARFrameUpdated function in the ParticlePainter script like this.

    public void ARFrameUpdated(UnityARCamera camera)
 {
     Matrix4x4 matrix = new Matrix4x4();
     matrix.SetColumn(3, camera.worldTransform.column3);
   
     Vector3 currentPositon = UnityARMatrixOps.GetPosition(matrix) + (Camera.main.transform.forward * penDistance);
     if (Vector3.Distance (currentPositon, previousPosition) > minDistanceThreshold) {
         if (paintMode == 2) {
             currentPaintVertices.Add (currentPositon);
             StringBuilder sb = new StringBuilder(); //create a new stringBuilder object
             //create a new string appending the values and dividing them with a space
             sb.Append(currentPositon.x).Append(" ").Append(currentPositon.y).Append(" ").Append(currentPositon.z);
             GameObject go = GameObject.Find("SpacebrewObject");
             SpacebrewEvents client = go.GetComponent <SpacebrewEvents> ();
             client.sendParticle(sb.ToString()); //send the string value of the stringBuilder object to the spacebrew script to be uploaded
         }
         frameUpdated = true;
         previousPosition = currentPositon;
     }
}

    Then in Spacebrew script we added the function sendParticle to upload the string on spacebrew.

    public void sendParticle(string position) { //send to spacebrew the 3 values of Vector3 position as 1 string
   sbClient.sendMessage("particleUp","string", position);
}

    Once the message is received on the other iPad, the string is parsed back into a Vector3 and sent to the particle script into a new function similar to the original ARFrameUpdated but that not takes the values from the hardware of iPad but uses the values of the other iPad to generate the new particle.

    if (_msg.name == "particleDown") {
  string[] values = _msg.value.Split(' '); //split the string with the position of particle in 3 values x,y,z
  Vector3 result = new Vector3(float.Parse(values[0]), float.Parse(values[1]), float.Parse(values[2])); //create the new Vector3 particle
  GameObject go = GameObject.Find("ParticlePainter");
  ParticlePainter client2 = go.GetComponent <ParticlePainter> ();
  client2.otherARFrameUpdated(result); //send the position to the particle script to be added to to the array of particles created above
}
    public void otherARFrameUpdated(Vector3 val) //the function that receive the position value of the particle from the spacebrew script and add a new particle in that position
{
   Vector3 currentPosition = new Vector3(val.x,val.y,val.z);
   if(otherPaintMode == 2 && paintMode == 0) {
      currentPaintVertices.Add (currentPosition);
   }
   frameUpdated = true;
}

    Another thing to send from one iPad to the other through Spacebrew is the button state, meaning basically if the other iPad is drawing or not. We did this to separate the drawing from the viewing state. So when one iPad is drawing the other iPad should be in a not drawing mode.

    To do this we just took the value of the button state and convert into a String to be uploaded on Spacebrew.

     void OnGUI()
 {
     string modeString = paintMode == 0 ? "OFF" : (paintMode == 1 ? "PICK" : "PAINT");
     if (GUI.Button(new Rect(Screen.width -100.0f, 0.0f, 100.0f, 50.0f), modeString))
      {
         paintMode = (paintMode + 1) % 3;
         string button = paintMode.ToString(); //convert the button value into a string
         GameObject go = GameObject.Find("SpacebrewObject");
         SpacebrewEvents client = go.GetComponent <SpacebrewEvents> ();
         client.sendButton(button); //send to spacebrew script to be uploaded
         colorPicker.gameObject.SetActive (paintMode == 1);
         if (paintMode == 2)
             RestartPainting ();
      }
     
 }

    Then on Spacebrew script we sent the value.

    public void sendButton(string _button) { //send to spacebrew the values of the button
   sbClient.sendMessage("buttonUp","string", _button);
}

    To parse it back on the other iPad we added a listener, send the value the the Particle script and also checked if the other iPad goes into drawing mode then start a new Particle System.

    if(_msg.name == "buttonDown") {
   int otherButton = int.Parse(_msg.value); //transform the value of the button from string to int
   GameObject go3 = GameObject.Find("ParticlePainter");
   ParticlePainter client3 = go3.GetComponent <ParticlePainter> ();
   client3.transferValues(otherButton); //send the button value to the particle script
   if(otherButton != prevOtherButton) {
   //if the first iPad just start drawing then call the RestartPainting function
   //in the particle script in order to start a new array of particles
      prevOtherButton = otherButton;
      if(otherButton == 2) {
         GameObject go = GameObject.Find("ParticlePainter");
         ParticlePainter client = go.GetComponent <ParticlePainter> ();
         client.RestartPainting();
      }
}

    Then we added the receiving function on the Particle script

     public void transferValues(int _buttonDown) { //the function that receive the button value from the spacebrew script and put it in a global variable
     otherPaintMode = _buttonDown;
 }

    In the end we adde another instance of the drawing function into the Update function that only happen if this iPad is in viewing mode and the other iPad is in drawing mode.

    if(otherPaintMode == 2 && paintMode == 0 && frameUpdated) {
    if ( currentPaintVertices.Count > 0) {
        int numParticles = currentPaintVertices.Count;
        ParticleSystem.Particle[] particles = new ParticleSystem.Particle[numParticles];
        int index = 0;
        foreach (Vector3 currentPoint in currentPaintVertices) {     
            particles [index].position = currentPoint;
            particles [index].startColor = currentColor;
            particles [index].startSize = particleSize;
            index++;
        }
        currentPS.SetParticles (particles, numParticles);
    } else {
        ParticleSystem.Particle[] particles = new ParticleSystem.Particle[1];
        particles [0].startSize = 0.0f;
        currentPS.SetParticles (particles, 1);
    }
    frameUpdated = false;
}

  3. Event listeners in Unity

    To detect the change of color we had to add one Event listener on the Setup function of the ParticlePainter.cs at line 33. That's because the color of the particle systems gets updated in another script when onValueChanged event occurs.

    colorPicker.onValueChanged.AddListener(delegate{SendColor();});

    The solution we adopted is the most simple way to invoke a function as an event as seen here.

    For further info about Events and delegates look here and here

  4. Serialize color to spacebrew, and parse it back.

    The function called in the event listener convert the RGBA values of the color into a String, and pass it to the Spacebrew event to be published.

    void SendColor() {
     string r = currentColor.r.ToString();
     string g = currentColor.g.ToString();
     string b = currentColor.b.ToString();
     string a = currentColor.a.ToString();
     StringBuilder colorToSend = new StringBuilder();
     colorToSend.Append(r).Append(" ").Append(g).Append(" ").Append(b).Append(" ").Append(a);
     GameObject go = GameObject.Find ("SpacebrewObject");
     SpacebrewEvents se = go.GetComponent <SpacebrewEvents> ();
     se.SendColor(colorToSend.ToString());
 }

    When received on a Spacebrew subscriber, the string is parsed to a Vector4 inside the SpacebrewEvents script:

    if (_msg.name == "colorDown") {
     //change otherColor
     string[] values = _msg.value.Split(' '); //split the string with the position of particle in 4 values r,g,b,a
     Color otherColor = new Vector4(float.Parse(values[0]), float.Parse(values[1]), float.Parse(values[2]), 
     float.Parse(values[3])); //create the new Vector4 particle
     GameObject pp = GameObject.Find("ParticlePainter");
     ParticlePainter pps = pp.GetComponent <ParticlePainter> ();
     pps.ChangeOtherColor(otherColor);
}

    And eventually the Vector4 is assigned to the otherColor variable in ParticlePainter by calling the function ChangeOtherColor in the ParticlePainter script.

  5. Package everything together.

    Once we had a working scene, we wanted to put everything together in a handy package, with just the elements we needed. It turned out to be simple and effective!

Short description of how everything will work:

The prototype for Friday will be a shared AR scene where people will be able to draw in a 3D space by using a particle system and see live what others are drawing in the same physical space. The AR scene will be created in Unity using ARKitPlugin. In order to share the scene within a group of several devices, we will use Spacebrew. By using those tools we will be able to share the position(x,y,z) and colour(rgba) of recently created particles and see it with other devices.

A description of how the code works:

Spacebrew Events and Particle painter

After compiling everything to Xcode we used a JavaScript multiple string subscriber (get the code) for Spacebrew to log values we are sending.

Our further steps:

  • Test it with people in a natural environment (design questions, interactions)
  • Try to load the local Spacebrew server using Raspberry PI
  • GUI > improve button element
  • UI > I/O HSV Colour picker + Touchevent to draw
  • Occlusion
  • World Map
  • Lifetime of a particle system
  • Speed of drawing = pressure (size of a particle system)
  • Further concept

Clone this wiki locally