SAM2 First → YOLO Classification for 46 Custom Props — Is This a Good Pipeline?

[saekut]

Hi, I’m building a real-time recognition system for 46 unique physical props.

I already have 200 rendered PNG views per prop (photo scanned 3d prop) from different angles, all with transparent backgrounds.

My planned workflow is:

Training:
46 props × 200 transparent views → train one YOLO classification model with 46 classes

Live camera:
Camera frame → SAM2 segments each physical prop first → extract the full RGB crop of each SAM2 mask → send each isolated crop into YOLO → YOLO predicts which of the 46 props it is.

So in my system, SAM2 is responsible only for separating objects, while YOLO is responsible only for identity/classification.

My main questions are:

1. Is this segment-first → classify-second architecture reasonable with Ultralytics YOLO11?
2. If two props overlap and only part of one prop remains visible, can YOLO11-CLS still identify it reliably, or should I generate additional partially-occluded training images?
3. Since my training renders have transparent backgrounds and SAM2 also removes the background at runtime, is this a good domain match, or should I still composite the renders onto real camera backgrounds during training?

I’d appreciate any advice on whether this pipeline is technically sound before I train the full 9,200-image dataset.

[UltralyticsAssistant]

👋 Hello @saekut, this segment-first → classify-second pipeline is technically reasonable for 46 known props: SAM2 can isolate instances, while YOLO11-CLS predicts the identity of each crop.

For overlapping props, YOLO11-CLS may work when enough distinctive geometry remains visible, but you should include partially occluded examples during training. Synthetic occlusion, varied crop sizes, rotations, lighting, blur, and camera-like noise can help, and validation images should reflect the real occlusion levels.

Transparent renders are a useful starting point, but training only on transparent backgrounds may create a domain gap. Composite the renders onto varied real or representative backgrounds and match the live preprocessing closely. Also test whether classification performs better with the masked crop plus some surrounding context rather than only the visible foreground pixels.

Before training all 9,200 images, a small pilot across visually similar props and representative overlap conditions would help validate the crop and background strategy. This is an automated response, and an Ultralytics engineer will assist you soon.

[pderrenger]

Yes, this is a reasonable pipeline, but SAM 2 is prompt-based, so you still need points/boxes or another proposal method to initialize each object; also benchmark its latency for real-time use. Train YOLO classification on crops that closely match inference: include partial occlusions, imperfect masks, varied lighting/scale, and both transparent/masked and real-background composites, since synthetic-only clean renders will likely create a domain gap. Keep a real-camera validation set, and consider a single custom YOLO segmentation model as a faster end-to-end baseline.

[saekut]

noted, thanks for answering

[qadeesanoor]

Your 200 renders per prop are probably clean, consistent framing. SAM2's live masks won't be box size/aspect ratio will vary based on angle and occlusion...

[pderrenger]

Yes, the pipeline is reasonable, but accuracy will depend on matching training data to actual SAM2 outputs: generate crops with variable mask quality, scale, padding, aspect ratio, partial occlusion, lighting, blur, and real background composites rather than relying only on clean transparent renders. Also keep enough padding around masks, since classification preprocessing uses crop-based transforms that may remove details from extreme aspect ratios. Before training all 9,200 images, validate a small model on real camera crops produced by your exact SAM2 pipeline.