Skip to content

Latest commit

 

History

History
87 lines (68 loc) · 7.08 KB

README.md

File metadata and controls

87 lines (68 loc) · 7.08 KB
Raw

About

This ar-affordances/robot directory has all the information and linked packages required to perform manipulations that were defined by the user wearing an AR-HMD.

Prerequisites

Hardware

  • Mobile Manipulator with SLAM Capabilities
  • RGB-D Sensor (minimum)
  • Local Wireless Network

Software

The following packages must be installed and configured on both the robot hardware and on an operator workstation machine. If using Vaultbot (Demo Robot) these packages are already installed and configured. If setting up a new robot, the list of software packages below is the minimum requirements for peforming this demo. You will still need basic packages on board to run sensors, manipulators, navigation, etc. Follow the links below to learn more about each package, duplicate, and leverage if needed.

Package Name Description
affordance_primitives This is the high-level repository for Affordance Primitives (APs).
ap_planning Code for pre-planning kinematic trajectories for screw-based Affordance Primitives.
compliance_controller Code to enable or disable compliance required for manipulation.
affordance templates (vbats) This repo is a VaultBot-specific AT code, partnered with the robot-agnostic affordance_primitives repository. New robots will need a similar repo setup.
temoto-framework TeMoto actions must be created for navigation, manipulation, etc. See drop-down list below.
Required TeMoto Actions:
TeMoto Action Description Input Parameters
ta_initialize_robot Brings up the robot and its main capabilities: Navigation, maniulation, and gripper features robot_name
ta_start_component TeMoto action used to load the cameras component
ta_move_manip_target_pose Moves the manipulator to a named target pose pre defined on the srdf robot_name
planning_group
target_pose
ta_move_base Action that sends a navigation goal for the mobile base. Use to approach to the object. robot_name
reference_frame
pose_2d(x,y,theta)
ta_move_gripper Controls the openning of the gripper robot_name
position(0_100%)
ta_screw_vector Validates the trajectory from multiple screws, and moves the arm to the start pose robot_name
planning_group
end_effector_name
screw_array
grab_pose
ta_state_ap TeMoto action used to perform the screws (approach, turn valve, and retreat motion) robot_name
ap_action_name
end_effector_name
screw_frame
is_pure_tranlation
screw_axis
screw_origin
screw_distance
screw_pitch
task_impedance_rotation
tansk_impedance_translation
thate_dot
ta_find_grasp Performs the exploration methods to find a valid grasp pose robot_name
gripper_name
clase_tolerance

Using Vaultbot (Dual Arm Robot Used in Demo)

  • Onboard: The packages listed in the above table should all be installed onboard Vaultbot. If there are issues, the vaultbot_onboard (Github) repo is the main repo used to install everything again. Follow instructions on the ReadMe to install the necessary software onboard.

    ❗ make sure you used git submodule update --init --recursive to get all submodules
    ❗ Use noetic branch (as of 8/17/2023)

  • Operator: If the operator machine is not already configured for controlling vaultbot. vaultbot_control is a combination of everything you need to get the packages up and running on your local machine.

AugRE Integration

  • Instructions below will get your robot integrated into the AugRE framework used for AR-based human-robot interactions. The following packages are required to communicate and localize with AR (Microsoft HoloLens 2) users. If using Vaultbot, this is already all installed.
  1. Follow UTNuclearRobotics/afc_demos installation and configuration instructions.1

Error Correction Method Setup

  • Eye-To-Hand Calibration (EHC)

    1. TODO: detail hololens visuals

  • Feel-in-the-Dark Exploration (FitD)

    1. Change into the robot_static repo and do stuff
    cd robot/robot_statics && #do stuff

  • k-Nearest Neighbor Regression (k-NN)

    1. At a properly sourced workspace on the robot, launch the k-NN grasp refinement server:
    rosrun vbats nn_pose_correction_server

    Ensure the /mega/calibration_mode parameter is set to 'john'.

    Now, invoke the ta_screw_vector action, and the k-NN grasp correction service will be used.

  • Point Cloud Segmentation (PC Seg)

    1. Change into the pc_seg repo and do stuff
    cd robot/pc_seg && #do stuff

Running Demo

Follow the instructions linked on the VB TeMoto Operator repo. Direct link to demo: Hololens Integration Demo

-NOTE: These instructions below are specific to Vaultbot, but if using a new robot, the steps will still be the same, just different naming conventions may be used.

Continue Setup

Once the robot is initialized and localized, we can command it to perform a task. Follow the Server README next to setup the required server for solving screw axis parameters and then follow instructions on ARHMD README to deploy and launch AugRE application with the ADAM module.

Details

The user, using the HMD (Hololens 2) performs a demonstration on an object to define the screw axis and the grasp point, which internally generates a UMRF graph that contains a sequence of TeMoto actions in a JSON format (string), that is sent to the robot through robofleet using the /broadcast_start_umrf_graph topic.

When TeMoto receives the UMRF graph, it validates if all of the TeMoto actions exist and starts the graph from the root node or the first TeMoto action.

Footnotes

  1. UTNuclearRobotics/afc_demos will provide details on how to install a RobofleetClient on the robot as well as all the necessary Microsoft Azure Spatial Anchor code necessary to communicate and localize with AR-HMD users. ↩