Bosse: Segmentation Automation

Effortlessly generating datasets without manual labelling, our innovative solution combines the SICK 3D camera, webcam, and ABB GoFa arm, making it ideal for use in production lines.

Team

Meet the game-changers who brought Bosse to life:

• Michael Khalfin

  • Organization: Rice University
  • Email: mlk15@rice.edu
  • LinkedIn: Profile

• Noah Meissner

  • Organization: University Regensburg
  • LinkedIn: Profile

• Joseph Wright

  • Organization: Drexel University
  • Email: jrw435@drexel.edu
  • LinkedIn: Profile

• Harsh Dhillon

  • Organization: Magna International
  • Email: harshpartap2001@gmail.com
  • LinkedIn: Profile

• Jingyan Yang

  • Organization: Technical University of Denmark
  • Email: jingyanyang94@gmail.com
  • LinkedIn: Profile

Abstract

As an logistics worker, your crucial responsibility is to track and ship a wide range of products. Traditionally, this involved labor-intensive tasks like manually labeling images by outlining objects, which could eat up hours of your day or necessitate additional staff.

But now, enter Bosse, the innovation of the future. With Bosse, we've automated this entire process. As products glide down the production line, a precision robotic arm gracefully rotates and captures them from multiple angles. Using cutting-edge technology like computer vision and a 3D camera, Bosse swiftly identifies object coordinates with precision. Your role as a logistics worker? Just answer a few quick questions about the type of object – Bosse's advanced technology handles the rest. It ensures objects are never scanned twice, and deep learning is automatically applied for future identifications.

Bosse: simplifying logistics, streamlining your work, and optimizing efficiency – all in a day's work.

How It Works

Video: Video

We ensure precision in our image processing by capturing objects from multiple angles. Our cutting-edge computer vision technology automates the task of outlining objects by identifying approximately 500 high-contrast points in each image. To enhance accuracy, we eliminate outlier points, retaining only those that contribute to forming lines or curves.

We input roughly 50 images per object, each of which features labeled regions of interest, into the Ultralytics YOLOv8 Nano model. This model utilizes a small batch size of 1 and is trained over 4 epochs, a careful approach to avoid overfitting the region of interest. This strategy optimizes key metrics such as precision-recall, confidence, and loss.

With this method, we ensure that we never segment the same image twice, and our deep learning model, securely stored locally on your device, is applied seamlessly during future encounters with the object. This meticulous process assures unparalleled accuracy, guaranteeing that your shipments meet the highest quality standards.

Technologies Used

Bosse was developed using a powerful combination of technologies, including:

• Python: Python was the main language we used for our frontend and backend.
• PyQt6: PyQt6 was our native GUI toolkit of choice.
• Robot Operating System (ROS) & Raspberry Pi: ROS and a Raspberry Pi Model 4B were used for connecting devices across a 5G network.
• 3D Camera & Webcam: The Visionary-T Mini camera, sponsored by SICK, was responsible for imaging the object every 4 seconds; then we could view the output using a webcam.
• ABB GoFa Arm: The arm, sponsored by ABB, rotated the object in front of the camera, completing the hardware stack.

Advantages

• Enhanced Efficiency: Our innovative approach streamlines the image capture and object outlining process, saving valuable time and reducing manual effort.
• Precision and Reliability: Our computer vision technology ensures meticulous identification of key points, resulting in highly accurate object segmentation and reduced errors.
• Optimized Model Training: By carefully managing batch sizes and epochs, our Ultralytics YOLOv8 Nano model achieves optimal performance, enhancing precision, recall, and overall accuracy.
• Elimination of Redundancy: Our approach guarantees that the same image is never segmented twice, preventing duplication and ensuring an efficient workflow.
• Local Deep Learning Integration: The locally stored deep learning model simplifies future object identifications, contributing to a seamless and efficient process.
• High-Quality Shipments: The result of our meticulous process is impeccable shipments that consistently meet the highest quality standards.
• Dynamically Sample Products: Painlessly generate new input for the model if it hasn't seen the object before.
• Transparent Analysis and Communication: We maintain a commitment to transparency by providing users with insights into our processes, ensuring you have a clear understanding of the technology and its applications.

Acknowledgments

We would like to thank the SICK Solution Hackathon team and all the corporate sponsors at this event. We had a blast spending the week at the SICK SIA Campus, AG Distribution Center, and Super8 Hotel. The Hackathon was a wonderful opportunity to work with people from around the world specializing in a variety of disciplines. Additionally, we would like to thank the coaches, who helped a lot, and our family, friends, and mentors who supported us through this journey.