The paper "Multi Actor-Critic DDPG for Robot Action Space Decomposition: A Framework to Control Large 3D Deformation of Soft Linear Objects", written by Mélodie Daniel1, Aly Magassouba2, Miguel Aranda3, Laurent Lequièvre2, Juan Antonio Corrales Ramón4, Roberto Iglesias Rodriguez4, and Youcef Mezouar2, has been submitted for publication in RA-L.
Title : MultiAC6 framework
Robotic manipulation of deformable linear objects (DLO) has great potential for applications in diverse fields such as agriculture or industry. However, a major challenge lies in acquiring accurate deformation models that describe the relationship between robot motion and DLO deformations. Such models are difficult to calculate analytically and vary among DLOs. Consequently, manipulating DLOs poses significant challenges, particularly in achieving large deformations that require highly accurate global models. To address these challenges, this paper presents MultiAC6: a new multi Actor-Critic framework for robot action space decomposition to control large 3D deformations of DLOs. In our approach, two deep reinforcement learning (DRL) agents orient and position a robot gripper to deform a DLO into the desired shape. Unlike previous DRL-based studies, MultiAC6 is able to solve the sim-to-real gap, achieving large 3D deformations up to 40 cm in real-world settings. Experimental results also show that MultiAC6 has a 66% higher success rate than a single-agent approach. Further experimental studies demonstrate that MultiAC6 generalizes well, without retraining, to DLOs with different lengths or materials.
Title : Video summary with demos
Title : Additional demos
virtualenv is a tool to create lightweight “virtual environments” with their own site directories isolated from system site directories. Each "virtual environment" has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.
sudo apt install python3-pip
pip3 install virtualenv
sudo apt install python3-virtualenv
virtualenv ve_rl --python=python3
source ve_rl/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
gui : boolean to show or not the gui (False by default).
mode : string that enable the main file to be launched into a particular mode (train by default). Modes values are train, test, test_real, generate_db.
config_file : string that contains a path (directory) concatenate to the name of the config file (config.json).
example of config_file : './trains_tests/simulation/test/MultiAC6Star/reward_max/small_0_05/config.json'
The configuration folder must constain :
3 files at its root :
A file named 'db.txt' containing a database of deformations to be reached by the deformable object.
A file named 'frite.vtk' containing the 3D graphical structure of the deformable object.
A file named 'config.json' containing the parameters necessary for the 'Gym' environment.
2 subfolders :
'env_panda' containing actor/critic neural weights (of Agentp) necessary to achieve a deformation.
'env_rotation_gripper' containing actor/critic neural weights (of Agento) necessary to achieve initial gripper orientation.
Note that, the actor and critic files are named 'actor.pth' and 'critic.pth'.
By changing some parameters, you can define different behavior during training and testing mode.
The initial rotation gripper :
'env'/'panda_parameters'/'orientation_gripper'/'index' : 0 = 'from_initial' (set the initial orientation), 1 = 'from db' (set the orientation value contained in db file), 2 = 'from_agent' (set the orientation given by Agento).
The reward calculation method :
'env'/'reward_parameters'/'reward_index' : 0 = 'mean' (mean distance error), 1 = 'max' (max distance error), 2 = 'dtw' (sum distance error).
The action space :
'env'/'all_spaces'/'action_space'/'index' : 0 = '3d' (3 DOF movement of the gripper), 1 = '6d' (6 DOF movement of the gripper).
Set number of episodes and steps per episode
'env'/'env_test'/'n_episodes' or 'env'/'env_train'/'n_episodes'
'env'/'env_test'/'n_steps' or 'env'/'env_train'/'n_steps'
The name of the log file :
'env'/'log'/'name' : by default the name is 'log_main_json.txt'.
cd DDPG_GPU_MPI
python main_json.py --mode generate_db --config_file './trains_tests/simulation/generate_db/with_orientation/small/config.json'
The generation of the db takes into account the parameters included in 'config.json' (array specified in the key 'env'/'database'/'frite_parameters'/'parameters_array'). Each element of this array contains the pose and goal space dimensions as well as the number of deformations to generate. The database will be created in a file named 'generate_db.txt'. If you want to use this generated new db, you will have to copy this file into a config folder and rename it as 'db.txt'.
cd DDPG_GPU_MPI
mpirun -n 32 python main_rotation_gripper.py --mode train --config_file './trains_tests/simulation/train/Agent_o/config.json'
The database used is a file named 'db.txt' in the directory './trains_tests/simulation/train/Agent_o'
The neural network weights of the Agento will be saved in the directory './trains_tests/simulation/train/Agent_o/env_rotation_gripper'
cd DDPG_GPU_MPI
mpirun -n 32 python main_json.py --mode train --config_file './trains_tests/simulation/train/Agent_p/reward_max/config.json'
The database used is a file named 'db.txt' in the directory './trains_tests/simulation/train/Agent_p/reward_max'
The neural network weights will be saved in the directory './trains_tests/simulation/train/Agent_p/reward_max/env_panda'
The neural network weights of the Agento should be copied from the train folder (e.g., './trains_tests/simulation/train/Agent_o/env_rotation_gripper') and saved in the directory './trains_tests/simulation/train/Agent_p/reward_max/env_rotation_gripper'
cd DDPG_GPU_MPI
python main_json.py --mode test --config_file './trains_tests/simulation/test/MultiAC6/small_0_05/config.json' --gui true
The database used is a file named 'db.txt' in the directory './trains_tests/simulation/test/MultiAC6/small_0_05'
The neural network weights should be copied from the train folder (e.g., './trains_tests/simulation/train/Agent_p/reward_max/env_panda') and saved in the directory './trains_tests/simulation/test/MultiAC6/small_0_05/env_panda'
The neural network weights of the Agento should be copied from the train folder (e.g., './trains_tests/simulation/train/Agent_o/env_rotation_gripper') and saved in the directory './trains_tests/simulation/test/MultiAC6/small_0_05/env_rotation_gripper'
cd DDPG_GPU_MPI
<br>python main_json.py --mode test_real --config_file './trains_tests/real/MultiAC6/config.json' --gui true
The deformations to reach are contained in a file named "db_selected_save.txt".
Note that, these instructions assume that ROS Noetic is already installed as well as the Panda robot controller (e.g., https://github.com/lequievre/panda_controllers/tree/main).
1Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, F-33400 Talence, France.
2CNRS, Clermont Auvergne INP, Institut Pascal, Université Clermont Auvergne, Clermont-Ferrand, France.
3Instituto de Investigación en Ingeniería de Aragón, Universidad de Zaragoza, Zaragoza, Spain.
4Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, Spain.