F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

TL;DR: F2G-Pose estimates an object’s 6D pose, metric size, and completed 3D shape from a single segmented RGB-D observation in one forward pass.

Project page · arXiv · Paper PDF · Weights

Watch the 90-second film (MP4)

F2G-Pose lifts visual foundation features onto observed 3D points. Inputs are an RGB image, aligned depth, camera intrinsics, and a foreground mask; no category label or object-specific CAD model is needed.

• 🛠️ Installation
• ▶️ Demo & inference
• 🧪 Training
• 📊 Evaluation
• 🚀 Deployment

1. Lift appearance into 3D. A frozen RADIO backbone supplies dense image features, which are projected onto the observed point cloud.
2. Combine appearance and geometry. DGCNN builds local geometric tokens; a Mixture-of-Experts Transformer captures object-level structure.
3. Predict pose, size, and shape. Parallel heads estimate the object-to-camera transform, metric dimensions, and a completed point cloud. Use pose-only mode when you only need pose and size.

The model is trained on synthetic SOPE data and evaluated on both SOPE and real-world ROPE data.

Pose and shape on ROPE. Columns show RGB, ground truth, GenPose++, F2G-Pose, and the observed-to-completed point cloud. These examples are from the paper.

The paper also demonstrates pose-guided pick-and-place with an ARX robot, a parallel-jaw gripper, and an RGB-D camera. Completed shapes provide geometry for collision checking.

Follow Installation to set up the environment, RADIO, and model weights. Then launch the interactive demo:

f2g-pose demo --checkpoint weights/f2g-pose.pt --port 7860

Open http://127.0.0.1:7860, upload RGB, depth, and a mask, then enter the camera intrinsics and depth units. The demo displays pose axes, a 3D box, and the completed shape, with JSON and PLY downloads. Optional SAM prompting lets you select the object with clicks or a box.

For a ready-to-use real input, try the bundled ROPE teapot sample. It includes aligned RGB, metric depth, a target mask, camera intrinsics, and a one-object evaluation manifest.

For Python applications:

from f2g_pose import PoseEstimator

estimator = PoseEstimator("weights/f2g-pose.pt")
result = estimator.predict(rgb, depth, K, mask, depth_scale=0.001)

T_cam_obj = result["T_cam_obj"]       # 4×4 object-to-camera transform
size_m = result["size_m"]             # object dimensions in meters
points_cam = result["points_cam_m"]   # completed shape in camera coordinates

Here, rgb, depth, and mask are aligned NumPy arrays; K is a 3×3 matrix or [fx, fy, cx, cy]. Use depth_scale=0.001 for millimeter depth or 1.0 for meters. See input formats and coordinates for a complete example.

Weights: download f2g-pose.pt (SHA-256 bff06bd58a42fd805d8232f9ab7b4612b8eb833edf6b6cccd23e5040a8c9ca18). RADIO is downloaded separately.

Full-test results for the provided model:

AUC values are percentages. See Evaluation for VUS, evaluation settings, and reproduction commands.

On RTX 4090, batch-1 model inference runs at 36.6 FPS with shape completion and 37.2 FPS in pose-only mode. These FP32 measurements use 30 warmup and 200 synchronized iterations; preprocessing and SAM are excluded.

If you use F2G-Pose in your research, please cite:

@article{zhang2026f2gpose,
  title={F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation},
  author={Zhang, Jinyu and Lin, Haitao and Hou, Jiashu and Xue, Xiangyang and Fu, Yanwei},
  journal={arXiv preprint arXiv:2610.31623},
  year={2026},
  eprint={2610.31623},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.31623}
}

We thank the authors of RADIO, PoinTr / AdaPoinTr, PointNet++, Omni6DPose, GDR-Net, CenterNet, NOCS, SGPA, FS-Net, and Segment Anything for their code, datasets, and ideas.

F2G-Pose integration code uses Apache-2.0. Third-party components retain their own licenses; see THIRD_PARTY.md. RADIO is installed and downloaded separately under NVIDIA's terms.