Skip to content

Latest commit

 

History

History
 
 

Lab 5

Observant Systems

For lab this week, we focus on creating interactive systems that can detect and respond to events or stimuli in the environment of the Pi, like the Boat Detector we mentioned in lecture. Your observant device could, for example, count items, find objects, recognize an event or continuously monitor a room.

This lab will help you think through the design of observant systems, particularly corner cases that the algorithms needs to be aware of.

Prep

  1. Pull the new Github Repo.
  2. Install VNC on your laptop if you have not yet done so. This lab will actually require you to run script on your Pi through VNC so that you can see the video stream. Please refer to the prep for Lab 2, we offered the instruction at the bottom.
  3. Read about OpenCV, MediaPipe, and TeachableMachines.
  4. Read Belloti, et al.'s Making Sense of Sensing Systems: Five Questions for Designers and Researchers.

For the lab, you will need:

  1. Raspberry Pi
  2. Webcam
  3. Microphone (if you want to have speech or sound input for your design)

Deliverables for this lab are:

  1. Show pictures, videos of the "sense-making" algorithms you tried.
  2. Show a video of how you embed one of these algorithms into your observant system.
  3. Test, characterize your interactive device. Show faults in the detection and how the system handled it.

Overview

Building upon the paper-airplane metaphor (we're understanding the material of machine learning for design), here are the four sections of the lab activity:

A) Play

B) Fold

C) Flight test

D) Reflect


Part A

Play with different sense-making algorithms.

OpenCV

A more traditional method to extract information out of images is provided with OpenCV. The RPI image provided to you comes with an optimized installation that can be accessed through python. We included 4 standard OpenCV examples: contour(blob) detection, face detection with the Haarcascade, flow detection (a type of keypoint tracking), and standard object detection with the Yolo darknet.

Most examples can be run with a screen (e.g. VNC or ssh -X or with an HDMI monitor), or with just the terminal. The examples are separated out into different folders. Each folder contains a HowToUse.md file, which explains how to run the python example.

Following is a nicer way you can run and see the flow of the openCV-examples we have included in your Pi. Instead of ls, the command we will be using here is tree. Tree is a recursive directory colored listing command that produces a depth indented listing of files. Install tree first and cd to the openCV-examples folder and run the command:

pi@ixe00:~ $ sudo apt install tree
...
pi@ixe00:~ $ cd openCV-examples
pi@ixe00:~/openCV-examples $ tree -l
.
├── contours-detection
│   ├── contours.py
│   └── HowToUse.md
├── data
│   ├── slow_traffic_small.mp4
│   └── test.jpg
├── face-detection
│   ├── face-detection.py
│   ├── faces_detected.jpg
│   ├── haarcascade_eye_tree_eyeglasses.xml
│   ├── haarcascade_eye.xml
│   ├── haarcascade_frontalface_alt.xml
│   ├── haarcascade_frontalface_default.xml
│   └── HowToUse.md
├── flow-detection
│   ├── flow.png
│   ├── HowToUse.md
│   └── optical_flow.py
└── object-detection
    ├── detected_out.jpg
    ├── detect.py
    ├── frozen_inference_graph.pb
    ├── HowToUse.md
    └── ssd_mobilenet_v2_coco_2018_03_29.pbtxt

The flow detection might seem random, but consider this recent research that uses optical flow to determine busy-ness in hospital settings to facilitate robot navigation. Note the velocity parameter on page 3 and the mentions of optical flow.

Now, connect your webcam to your Pi and use VNC to access to your Pi and open the terminal. Use the following command lines to try each of the examples we provided: (it will not work if you use ssh from your laptop)

pi@ixe00:~$ cd ~/openCV-examples/contours-detection
pi@ixe00:~/openCV-examples/contours-detection $ python contours.py
...
pi@ixe00:~$ cd ~/openCV-examples/face-detection
pi@ixe00:~/openCV-examples/face-detection $ python face-detection.py
...
pi@ixe00:~$ cd ~/openCV-examples/flow-detection
pi@ixe00:~/openCV-examples/flow-detection $ python optical_flow.py 0 window
...
pi@ixe00:~$ cd ~/openCV-examples/object-detection
pi@ixe00:~/openCV-examples/object-detection $ python detect.py

***Try each of the following four examples in the openCV-examples, include screenshots of your use and write about one design for each example that might work based on the individual benefits to each algorithm.***

  • contours-detection

Alt Text

Design: It can be used to make virtual background and add filters on users' faces.

  • face-detection

Alt Text

Design: Greeting doorbell. The doorbell will say greeting words when it detects a human face in front of it.

  • flow-detection

Alt Text

Design: It can be used in games. The play needs to follow the game instruction to dance or do certain movement, and the flow detection can ckeck the degree of completion the player takes.

  • object-detection

Alt Text

Design: It can be used to automatic goods counting. When we buy many things at the same time online, we can use object detection to do a pre counting of goods, so that to make sure there is no package left. This design can also be used in express distribution centers.

MediaPipe

A more recent open source and efficient method of extracting information from video streams comes out of Google's MediaPipe, which offers state of the art face, face mesh, hand pose, and body pose detection.

Alt Text

To get started, create a new virtual environment with special indication this time:

pi@ixe00:~ $ virtualenv mpipe --system-site-packages
pi@ixe00:~ $ source mpipe/bin/activate
(mpipe) pi@ixe00:~ $ 

and install the following.

...
(mpipe) pi@ixe00:~ $ sudo apt install ffmpeg python3-opencv
(mpipe) pi@ixe00:~ $ sudo apt install libxcb-shm0 libcdio-paranoia-dev libsdl2-2.0-0 libxv1  libtheora0 libva-drm2 libva-x11-2 libvdpau1 libharfbuzz0b libbluray2 libatlas-base-dev libhdf5-103 libgtk-3-0 libdc1394-22 libopenexr23
(mpipe) pi@ixe00:~ $ pip3 install mediapipe-rpi4 pyalsaaudio

Each of the installs will take a while, please be patient. After successfully installing mediapipe, connect your webcam to your Pi and use VNC to access to your Pi, open the terminal, and go to Lab 5 folder and run the hand pose detection script we provide: (it will not work if you use ssh from your laptop)

(mpipe) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 5
(mpipe) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python hand_pose.py

Try the two main features of this script: 1) pinching for percentage control, and 2) "Quiet Coyote" for instant percentage setting. Notice how this example uses hardcoded positions and relates those positions with a desired set of events, in hand_pose.py lines 48-53.

***Consider how you might use this position based approach to create an interaction, and write how you might use it on either face, hand or body pose tracking.***

(You might also consider how this notion of percentage control with hand tracking might be used in some of the physical UI you may have experimented with in the last lab, for instance in controlling a servo or rotary encoder.)

The position based approach can be used to detect complexed gestures, such as thumbs up, pointing, hands up, OK, and using fingers to indicate numbers.

  • Those signals can be used as the input instruction for Pi. For example, when people are cooking, or doing skin care, it's not convinient for them to press buttons, so that they can make gestures to control Pi.

  • Also, the direction of pointing can be used to point a certain object, combining with object detection. If the direction of finger is on the same direction of an object, it means the user is pointing that object.

Alt Text

Alt Text

Teachable Machines

Google's TeachableMachines might look very simple. However, its simplicity is very useful for experimenting with the capabilities of this technology.

Alt Text

To get started, create and activate a new virtual environment for this exercise with special indication:

pi@ixe00:~ $ virtualenv tmachine --system-site-packages
pi@ixe00:~ $ source tmachine/bin/activate
(tmachine) pi@ixe00:~ $ 

After activating the virtual environment, install the requisite TensorFlow libraries by running the following lines:

(tmachine) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 5
(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ sudo chmod +x ./teachable_machines.sh
(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ ./teachable_machines.sh

This might take a while to get fully installed. After installation, connect your webcam to your Pi and use VNC to access to your Pi, open the terminal, and go to Lab 5 folder and run the example script: (it will not work if you use ssh from your laptop)

(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python tm_ppe_detection.py

(Optionally: You can train your own model, too. First, visit TeachableMachines, select Image Project and Standard model. Second, use the webcam on your computer to train a model. For each class try to have over 50 samples, and consider adding a background class where you have nothing in view so the model is trained to know that this is the background. Then create classes based on what you want the model to classify. Lastly, preview and iterate, or export your model as a 'Tensorflow' model, and select 'Keras'. You will find an '.h5' file and a 'labels.txt' file. These are included in this labs 'teachable_machines' folder, to make the PPE model you used earlier. You can make your own folder or replace these to make your own classifier.)

***Whether you make your own model or not, include screenshots of your use of Teachable Machines, and write how you might use this to create your own classifier. Include what different affordances this method brings, compared to the OpenCV or MediaPipe options.***

Alt Text

Design:

  • It can be used in the smart doorbell with a camera. The doorbell will notify the house owner who is visiting, any of her friends, family, colleages or a stranger.
  • It can be used to detect if a person is wearing a mask properly, during the pandemic period. And the device can be put at the door, so that when a person comes in the house without a mask, the device will remind her to take one on.

Don't forget to run deactivate to end the Teachable Machines demo, and to reactivate with source tmachine/bin/activate when you want to use it again.

Filtering, FFTs, and Time Series data. (optional)

Additional filtering and analysis can be done on the sensors that were provided in the kit. For example, running a Fast Fourier Transform over the IMU data stream could create a simple activity classifier between walking, running, and standing.

Using the accelerometer, try the following:

1. Set up threshold detection Can you identify when a signal goes above certain fixed values?

2. Set up averaging Can you average your signal in N-sample blocks? N-sample running average?

3. Set up peak detection Can you identify when your signal reaches a peak and then goes down?

***Include links to your code here, and put the code for these in your repo--they will come in handy later.***

Part B

Construct a simple interaction.

Pick one of the models you have tried, pick a class of objects, and experiment with prototyping an interaction. This can be as simple as the boat detector earlier. Try out different interaction outputs and inputs.

***Describe and detail the interaction, as well as your experimentation here.***

I used TeachableMachines to detect people with masks or not.

mask_dtc.mp4

Part C

Test the interaction prototype

Now flight test your interactive prototype and note down your observations: For example:

  1. When does it what it is supposed to do?

It supposes to detect wether the person is wearing a mask properly.

  1. When does it fail?

When the user's face is partially hidden, it's hard to detect the mask even the user is wearing one.

  1. When it fails, why does it fail?

Because the classifier cannot see the mask.

  1. Based on the behavior you have seen, what other scenarios could cause problems?

The scenarios that a user is not wearing a mask properly, such as the mask does not cover his nose, or he puts the mask on his forehead for some reeasons, those will be detected as the user is wearing a mask, which should not be the case.

***Think about someone using the system. Describe how you think this will work.***

  1. Are they aware of the uncertainties in the system?

I think they won't aware of the uncertainties in the system.

  1. How bad would they be impacted by a miss classification?

The bad situation is a user not wearing a mask properly walks into a resturant and there will be a risk of COVID spreading.

  1. Are there optimizations you can try to do on your sense-making algorithm.

I will add more training images in the class of wearing masks inproperly, so that the system will learn how to classify more correctly.

Part D

Characterize your own Observant system

Now that you have experimented with one or more of these sense-making systems characterize their behavior. During the lecture, we mentioned questions to help characterize a material:

  • What can you use X for?
  • What is a good environment for X?
  • What is a bad environment for X?
  • When will X break?
  • When it breaks how will X break?
  • What are other properties/behaviors of X?
  • How does X feel?

***Include a short video demonstrating the answers to these questions.***

mask_dtc2.mp4

Part 2.

Following exploration and reflection from Part 1, finish building your interactive system, and demonstrate it in use with a video.

***Include a short video demonstrating the finished result.***

Pi acts as a mask detector in front of the door of a restauruant. So that it will alarm the customer if she is not wearing a mask before entering the restauruant.

I divided the class into three clusters, mask, no mask and background, so that the Pi wouldn't predict a "no mask" class when there is no one in front of the door.

It seems there is something wrong with my microphone, and it cannot make sounds. Still, I wrote the code of speaking in my repository. And I use my voice to demostrate the effect of microphone voice.

video2.mp4