Repository navigation
Tutorial 1
When we look around, we see a world full of rich detail—faces, text, colors, and shapes. For us, figuring out where we want to go or what to eat can be as simple as reading a sign or spotting a landmark. Our brains process all of that context instantly and effortlessly.
But for a robot, that is not the case. That same room is a nightmare. When a robot looks through its camera lens, all it sees are millions of changing colors flashing dozens of times a second. If it tried to recognize every chair, fruit, or face, it would need a computer and a battery the size of a watermelon. And that's quite a problem for a little robot that could fit inside a watermelon itself.
So for our little robot, instead of trying to read the world like humans do, we cheat. We use something called fiducial markers.
Now, you've probably encountered something like this before in the everyday QR code. But standard QR codes have one big drawback for a fast-moving robot: they are crammed with data, like long website links. Processing all that detail takes time—if a QR code is moving, it could take a whole half-second for a tiny processor to decode it. In the robotics world, half a second is a painfully long time to drive blind!
This is where a specialized fiducial marker, like an AprilTag, comes to the rescue.
Instead of storing tons of text, an AprilTag is deliberately kept simple. It's just a bold, high-contrast black-and-white grid framed by a thick, dark square border. Because it's so distinct, the robot can skip processing the messy room and perform two blazing-fast tricks:
-
Edge Detection: The robot scans the image specifically for the extreme jump where dark black pixels meet bright white pixels. In an instant, it cuts through all the background clutter and finds the tag.
-
Corner Detection: Next, it locks onto the four sharp corners of that outer square border.
Here is the clever part: the robot already knows that the physical tag is a flat, perfect square. So, if those four corners look stretched, tilted, or squished in the camera frame, the robot's software uses simple geometry to figure out why. In just a couple of milliseconds, it calculates the exact distance, angle, and 3D tilt of the tag.
No massive computer needed. With just four high-contrast corners and a little math, our tiny robot instantly knows where it is and where it needs to go.
If we truly wish to look through the robot's lens, we can first connect through it on the UI.
Go to the UI and connect to your robot. Make sure the Raspberry Pi and Camera and working. Now place a cube in front of the camera. You should be able to see the tag that was detected.
Note: This section is Python-heavy. Proceed at your own risk.
Before we proceed, install the following:
pip install numpy opencv-pythonCopy & paste the following into a new file:
from soble import SO101Platform
if __name__ == "__main__":
# Connect to the platform, using whichever name you wrote to the BLE Name
platform = SO101Platform("Capybara")
while True:
detected = platform.detectApriltags(estimate_tag_pose=True)
for tag_id, corners, R, tvec in detected:
print(
f"Tag ID: {tag_id},"
f"Corners: {corners},"
f"Rotation: {R},"
f"Translation: {tvec}"
)Place an Apriltag cube in front of the camera. If the cube is placed correctly, you should be able to see something on the screen.
At first glance, a few pieces of this output make total sense. You can easily spot the tag ID—which is 29—and the four pixel coordinates marking its corners in the image: bottom-left, bottom-right, top-right, and top-left. Simple enough.
Then your eyes hit the rest of the printout, and things get weird.
Sitting right next to those clear numbers is a intimidating 3x3 matrix labeled "Rotation", followed by a 3x1 vector labeled "Translation". If your idea of rotation involves clean degrees or basic geometry, this block of nine floating-point numbers looks less like math and more like gobbley-dook. How on earth does a grid of nine random numbers tell anyone—let alone a robot—which way a tag is pointing?
To make sense of it, we have to start with a concept that sounds a bit mind-bending at first: the idea of axes existing in a whole different Frame of Reference.
A Frame of Reference is essentially a fancy term for point of view - if you were hanging upside down, your up (+z) would be the ground, whereas your friend's up (+z) would be the sky. And if you spun yourself rightside up then turned around to look at your friend, your left (-x) would be your friend's right (+x).
The Apriltag, we will soon discover, is also quite unusual - if you chose +y to be forward, then the Apriltag's up (+z) is actually forward (+y), and its forward (+y) is actually down (-z).
As it turns out, our cameras are the same type of unusual as the Apriltag—+x would point to your right, +y would point down, and +z forward.
You may wonder which came first—the camera or the tag. And while in other scenarios (such as that between the egg and the chicken) the answer may not be as clear, in this case its the camera.
Let's now take a look back at that pesky Rotation matrix. As it turns out (and pay close attention)—the Rotation matrix of the tag also has +x +y and +z axes:
You may now at this point have more questions than clarity. Why are there 3 numbers for a +x? What does this mean?
I will say it first, and then show you. A rotation matrix shows the relative axis position with respect to the parent axis. So the
vector is actually the tag's +x axis relative to the camera's +x axis.
So, if the tag is perfectly aligned with the camera (no rotation), then its +x axis will be perfectly aligned with the camera's +x axis:
However, if the tag is rotated 90 degrees counterclockwise, then the +x should be aligned with the -y axis of the camera:
Or if the tag was rotated 90 degrees clockwise, then the +x axis of the tag would look like:
Similarly, the translation is the position of the tag in the camera's Frame of Reference:
Meaning that if y=1, then the tag is 1 unit below the camera.