Automated Billiard Project

🔎 Demonstration

To see the details of our program and the demonstration video, you can click the web address blow !

👉 Our Yutube Channel: https://youtu.be/1msCYlrSwBo

⭐ Project Description

Automated Billiard Ball Machine Implemented with Manipulator and 3D Computer-Vision in High Accuracy:

Create a system combined with manipulator and depth-camera to play billiard ball automatically. The system includes Image-processing (Mask, Opening, Hough Circle Transformation) and AI (Simple Object Detection with Tensorflow) to apply object detection, Robot-Vision (Camera Calibration, Hand-Eye Calibration),Algorithm designed by ourselves to find the optimal solution to win the game, and Manipulator Kinematics to apply coordinate transformation.

💡 Note: To see the final demonstration without reading the description, you can goto our Youtube watching the demonstration video!

👉 https://youtu.be/1msCYlrSwBo

📙 Test execution

Eye-Hand Calibration and Camera Calibration with “ChAruco Board”

1. Execute camera_eye_hand.py

python /eye_hand/camera_eye_hand.py

Ball Detection and Algorithm

1. Execute main.py

python main.py

2. After seeing the prompt text "Continue: press c", press "c" to start the execute the program

3. The Manipulator will hit the ball automatically

4. Repeat the process by pressing "c" again; Stop the process by pressing the word except "c"

🌍 Program Flow

Eye-Hand Calibration and Camera Calibration with “ChAruco Board”

1. Open the HIWIN_Contest UI

Connect to the manipulator and turn on the "SMON"

2. Execute camera_eye_hand.py

Pose controlling Robot Manipulator to take pictures of “ChAruco Board”

3. Pose estimation to “ChAruco Board” and apply hand-eye calibration with OpenCV

Get the extrinstic matrix between the end-effector and the camera

4. Get intristic camera matrix with OpenCV

Get the intrinstic matrix and the distortion factor

Ball Detection and Algorithm

1. 1. Take color image (color.png) and depth image (config/depth.npy)

align_depth2color.take_pic()

💡 Note: the left is RGB image, the right is depth image

2. Undistort the color image and the depth image

test_undistort.undistort()

💡 Note: the obvious black lines around the picture are the consequence of the correction of distortion

3. Image noise reduction with applying Mask and Opening(morphology)

a. Apply green mask to the color image(bitwise_and)

b. Detect the edge of the table, using the picture above

c. Apply opening to reduce the image noise and save the image(mask.png)

4. Find the Holes of the table (config/hole.txt)

Applying Hough Circle Transformation to find the coordinates of the table holes from the mask.png Find Holes coordinates of the table with Hough Circle Transformation

findHole_new.findHole()

5. Simple Object Detection with Tensorflow to classify balls as Cue ball and others

6. Find Balls coordinates(config/white_pixel.txt, config/ball_pixel.txt)

hough_circle_test.findBall()

a. Apply bitwise_and between the color image and mask.png

b. Apply Hough Circle Transformation to find the balls' coordinates

7. Apply Billiard Ball Strategy Algorithm to find optimal solution

num_ball, isConvex = hit_ball_strategy.hit_ball_strategy()

The best solution to the case will be visualize on the image.

💡 Note: To see the rules of the strategy, you can turn to the Readme in the "algorithm/"

8. Coordinate Transformation from Image Coordinate to Robot Base Coordinate(World Coordinate) and Hit the Ball !

The coordinate of the cue ball(image coordinate) will be transformed into Robot-Base coordinate, and the manipulator will hit the ball with the Pneumatic cylinder.

coordinate_transformation.coordinate_transformation(num_ball, isConvex)