Lab 6: Visual servoing and FSMs

In this lab you will integrate the visual perception capabilities you developed ion Lab 4 and 5 into the Cozmo robot's behavior.

---

Step 1: You will make Cozmo follow a ball. If you have not already done so in Lab 4, find a ball and a space with clean background, such as a box you can place the Cozmo into. You will use the ball recognition algorithm you developed in Lab 4 to find the ball in Cozmo’s camera image. If necessary, update your find_ball() function for the new setting you are working in.

You are provided the following file to assist with this part of the lab:

• goto_ball.py – this is the main file where you will enter your solution. When Cozmo runs, the code will open a viewer that will display the robot’s camera data. Two viewer annotators are provided, BatteryAnnotator() and BallAnnotator(), that display the battery and ball information, respectively. Feel free to add your own annotators if you find that helpful. The main functionality of the code resides in run() which you should extend to have the robot look around for and go to the ball. Make sure you can execute this Python script and make sure your ball detector is working.

You will complete the code to enable the robot to find, move to, and hit the ball. Implement the following behavior as a Finite State Machine:

• Search: When started, the robot executes a search behavior until the ball is in view
• Move-to: The robot moves toward the ball as long as it is in view, goes back to search if it looses the ball.
• Hit: When the ball is close enough, the robot moves its arm to hit the ball and goes back to search.

There are a variety of behaviors you can program for Search, including rotating in place or driving around randomly. The Move-to should be done with visual servoing, i.e. adjusting the relative speed of the two wheels to center the object in the robot's image. The Hit can be an existing Cozmo animation or lifting/lowering of the arm. Feel free to integrate animations, sounds and the display into your code. Make a video demonstrating the robot's ball-following behavior, making sure all states (Search, Move to, Hit) are shown.

---

Step 2: Next you will demonstrate the usage of your trained classifier (Lab 5) in real time by the Cozmo robot. Develop a simple robot program that starts in an "idle" state and responds to seeing three different cards by triggering three different robot responses. You can print your own cards from this pdf.

Choose three of the cards that your classifier is trained to recognize. In response to seeing each card, the robot should use speech to announce the label name of the detected card and perform a physical movement. You can use one of the built-in Cozmo animations or create one yourself. You are not given any starter code for this part of the lab but you can easily reuse the skeleton of code provided in earlier labs.

Make sure that your program is robust to occasional false alarms by looking at recognition results over a certain time window, rather than reacting to the detection at every time step.

To complete this part of the lab, make a video that demonstrates the robot's reaction to three different cards.