[AAAI 2024] Representation-Based Robustness in Goal-Conditioned Reinforcement Learning

Xiangyu Yin¹, Sihao Wu¹, Jiaxu Liu¹, Meng Fang¹, Xingyu Zhao², Xiaowei Huang¹, Wenjie Ruan¹

¹Department of Computer Science, University of Liverpool, ²WMG, University of Warwick

This is a PyTorch implementation of our paper Representation-Based Robustness in Goal-Conditioned Reinforcement Learning; this code can be used to reproduce Section Experiments and Appendix of the paper.

We evaluate our method using four robot manipulation tasks, namely FetchPush, FetchReach, FetchSlide, and FetchPick as below.

FetchPush	FetchPickAndPlace	FetchReach	FetchSlide

SetUp

1. Create conda environment and activate it:

   conda env create -f environment.yml
   conda activate ReRoGCRL
   pip install torch==1.13.1 gym==0.15.7
   

2. Download the offline dataset and place /offline_data in the project root directory.

Experiments

We provide commands for reproducing the various attack methods (Table 1), the defensive performance (Table 3).

1. Train the 3 algorithms as our baselines for GCRL (DDPG, GCSL, GoFar) and save the models:

mpirun -np 1 python3 train.py --env $env --method $method --device $device --seed $seed

Flags and Parameters	Description
--env $ENV	tasks: FetchReach, FetchPush, FetchPick, FetchSlide
--method $METHOD	algorithms: ddpg, gcsl, gofar

2. Run various attack methods in GCRL for all the baselines to get Table 1.

sh scripts/adversarial_attack.sh $device $env $method $attack_obj

Flags and Parameters	Description
$env	tasks: FetchReach, FetchPush, FetchPick, FetchSlide
$method	algorithms: ddpg, gcsl, gofar
$attack_obj	attack objects: obs, goal, both

3. To run the defensive performance (Table 3), we first train the defensive methods in all environments and save the models.

sh scripts/trainning.sh $env $device $method

Flags and Parameters	Description
$env	tasks: FetchReach, FetchPush, FetchPick, FetchSlide
$method	algorithms: ddpg_simsr, ddpg_aug, ddpg_arts, gofar_simsr, gofar_aug, gofar_arts

4. Run various attack methods in GCRL for defensive methods to get Table 3.

sh scripts/adversarial_attack.sh $device $env $method $attack_obj

Flags and Parameters	Description
$env	tasks: FetchReach, FetchPush, FetchPick, FetchSlide
$method	algorithms: ddpg_simsr, ddpg_aug, ddpg_arts, gofar_simsr, gofar_aug, gofar_arts
$attack_obj	attack objects: obs, goal, both

Acknowledgement:

We borrowed some code from the following repositories:

• GoFAR
• SimSR