Tutorial 1

Into the Looking Glass

Preface

When we look around, we see a world full of rich detail—faces, text, colors, and shapes. For us, figuring out where we want to go or what to eat can be as simple as reading a sign or spotting a landmark. Our brains process all of that context instantly and effortlessly.

But for a robot, that is not the case. That same room is a nightmare. When a robot looks through its camera lens, all it sees are millions of changing colors flashing dozens of times a second. If it tried to recognize every chair, fruit, or face, it would need a computer and a battery the size of a watermelon. And that's quite a problem for a little robot that could fit inside a watermelon itself.

So for our little robot, instead of trying to read the world like humans do, we cheat. We use something called fiducial markers.

Now, you've probably encountered something like this before in the everyday QR code. But standard QR codes have one big drawback for a fast-moving robot: they are crammed with data, like long website links. Processing all that detail takes time—if a QR code is moving, it could take a whole half-second for a tiny processor to decode it. In the robotics world, half a second is a painfully long time to drive blind!

This is where a specialized fiducial marker, like an AprilTag, comes to the rescue.

Instead of storing tons of text, an AprilTag is deliberately kept simple. It's just a bold, high-contrast black-and-white grid framed by a thick, dark square border. Because it's so distinct, the robot can skip processing the messy room and perform two blazing-fast tricks:

• Edge Detection: The robot scans the image specifically for the extreme jump where dark black pixels meet bright white pixels. In an instant, it cuts through all the background clutter and finds the tag.

• Corner Detection: Next, it locks onto the four sharp corners of that outer square border.

Here is the clever part: the robot already knows that the physical tag is a flat, perfect square. So, if those four corners look stretched, tilted, or squished in the camera frame, the robot's software uses simple geometry to figure out why. In just a couple of milliseconds, it calculates the exact distance, angle, and 3D tilt of the tag.

No massive computer needed. With just four high-contrast corners and a little math, our tiny robot can start to explore the world.

The Little Robot's World

With the help of a little bit of graphics, we too can see what our little robot sees.

Go to the UI and connect to your robot. Make sure the Raspberry Pi and camera are working. Now place a cube in front of the camera. With enough moving around, you may be able to see the tag of one of the faces of the cube.

Diving Head First

While it is nice to see what our robot sees through the UI, its hard to do much more, especially if we wanted to make our robot a bit more clever. We will still have to rely on good old fashioned code to unlock more potential. To do this, copy & paste the following into a new file:

from soble import SO101Platform

if __name__ == "__main__":
  # Connect to the platform, using whichever name you wrote to the BLE Name
  platform = SO101Platform("Capybara")

  while True:
    detected = platform.detectApriltags(estimate_tag_pose=True)
    for tag_id, corners, R, tvec in detected:
      print(
        f"Tag ID: {tag_id},"
        f"Corners: {corners},"
        f"Rotation: {R},"
        f"Translation: {tvec}"
      )

Place an Apriltag cube in front of the camera. If the cube is placed correctly, you should be able to see something detected on the screen.

At first glance, a few pieces of this output make total sense. You can easily spot the tag ID—which is 29—and the four pixel coordinates marking its corners in the image: bottom-left, bottom-right, top-right, and top-left. Simple enough.

Then your eyes hit the rest of the printout, and things get weird.

Sitting right next to those clear numbers is a intimidating 3x3 matrix labeled "Rotation", followed by a 3x1 vector labeled "Translation". If your idea of rotation involves clean degrees or basic geometry, this block of nine floating-point numbers looks less like math and more like gobbley-dook. How on earth does a grid of nine random numbers tell anyone—let alone a robot—which way a tag is pointing?

To make sense of it, you will need to try your best to change your point of view.

How to Change Your Point of View

Imagine yourself hanging upside down, at least relative to a friend of yours. Your up (+z) would be the ground, whereas your friend's up (+z) would be the sky. And if you spun yourself rightside up then turned around to look at your friend, your left (-x) would be your friend's right (+x).

The Apriltag, we will soon discover, is this sort of strange. Let's choose +y to be forward. As it turns out the Apriltag's forward (+y) is actually down (-z). And since its +x is pointing right, its up (+z) is actually forward (+y).

Standard	Apriltag
+x	+x
+y	+z
+z	-y

And our cameras are like this too! +x would point to your right, +y would point down, and +z forward.

Say it Sideways

Let's now take a look back at that pesky Rotation matrix. As it turns out (and pay close attention)—the Rotation matrix of the tag also has +x +y and +z axes:

$$ \begin{bmatrix} x_0 & y_0 & z_0 \\ x_1 & y_1 & z_1 \\ x_2 & y_2 & z_2 \end{bmatrix} $$

You may now at this point have more questions than clarity. Why are there 3 numbers for a +x? What does this mean?

I will say it first, and then show you. A rotation matrix shows the relative axis position with respect to the parent axis. So the

$$ \begin{bmatrix} x_0 \\ x_1 \\ x_2 \end{bmatrix} $$

vector is actually the tag's +x axis relative to the camera's +x axis.

So, if the tag is perfectly aligned with the camera (no rotation), then its +x axis will be perfectly aligned with the camera's +x axis:

$$ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$

However, if the tag is rotated 90 degrees counterclockwise, then the +x should be aligned with the -y axis of the camera:

$$ \begin{bmatrix} 0 \\ -1 \\ 0 \end{bmatrix} $$

Or if the tag was rotated 90 degrees clockwise, then the +x axis of the tag would look like:

$$ \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} $$

Similarly, the translation is the position of the tag in the camera's Frame of Reference:

$$ \begin{bmatrix} x \\ y \\ z \end{bmatrix} $$

Meaning that if y=1, then the tag is 1 unit below the camera.

---