Towards Disturbance-Free Visual Mobile Manipulation (WACV 2023)

[Project Site] [arXiv]

Authors: Tianwei Ni (Mila, AI2), Kiana Ehsani (AI2), Luca Weihs† (AI2), Jordi Salvador† (AI2) († equal advising)

Install

Please git clone AllenAct repo with this submodule and change to a specific commit for reproducibility:

git clone --recurse-submodules https://github.com/allenai/allenact.git
git checkout f00445e4ae8724ccc001b3300af5c56a7f882614

We recommend using a virtual environment like conda to install the dependencies. Typically, our experiment needs a 8-GPU machine with around 60 CPU cores. Below are scripts for installation by conda:

conda create -n disturb
conda activate disturb
conda install python==3.6.12
pip install -r projects/disturb-free/requirements.txt

Then download the APND dataset to the allenact folder.

Stage I: Pre-Training

The main configuration is in projects/manipulathor_disturb_free/armpointnav_baselines/experiments/ithor/armpointnav_depth.py.

First keep all the configurations fixed to run pre-training experiments. By choosing different auxiliary task or commenting all the tasks in AUXILIARY_UUIDS in projects/manipulathor_disturb_free/armpointnav_baselines/experiments/armpointnav_mixin_ddppo.py, we can run experiment with CPC|A, Inverse Dynamics, Disturbance Prediction, or without auxiliary task, i.e.,

    AUXILIARY_UUIDS = [
        # InverseDynamicsLoss.UUID,
        # CPCA16Loss.UUID,
        # DisturbPredictionLoss.UUID,
    ]

Run pre-training by the following script:

python main.py projects/manipulathor_disturb_free/armpointnav_baselines/experiments/ithor/armpointnav_depth.py \
    -s 302 --save_dir_fmt NESTED \
    -o experiment_output/armpointnav/

Then we will see the experiment results in experiment_output/armpointnav/ folder. We can use tensorboard to visualize the results on training scenes.

Stage II: Fine-Tuning

Now we switch to the fine-tuning stage, where we use the pre-trained checkpoint saved in the experiments results folder */checkpoints/*.pt. Choose one checkpoint (e.g. at 20M steps) and set LOAD_PRETRAINED_WEIGHTS = True and DISTURB_PEN=15.0 in armpointnav_depth.py.

Then run fine-tuning using the script below:

PRETRAIN_CKPT=<ckpt_path>
python main.py projects/manipulathor_disturb_free/armpointnav_baselines/experiments/ithor/armpointnav_depth.py \
    -s 302 --save_dir_fmt NESTED \
    -o experiment_output/armpointnav/ \
    -r -c $PRETRAIN_CKPT 
# don't forget -r argument

Evaluate and Visualize the Best Checkpoints

Please download the best checkpoints of our method (Disturb_Pred), the compared methods (CPCA_16 and InvDyn) and baseline (no_aux) using the script:

sh projects/manipulathor_disturb_free/download_ckpts.sh armpointnav-disturb-free-2021

Evaluation

Run the following script to evaluate (take our disturbance prediction task as example) after setting the same configurations in TestScene.py:

TEST_CKPT=pretrained_model_ckpts/armpointnav-disturb-free-2021/best_models/gnresnet18-woNormAdv-wact-man_sel-polar_radian-finetune-disturb_pen15.0_all-Disturb_Pred-gamma2.0/checkpoints/exp_resnet18-woNormAdv-wact-man_sel-polar_radian-finetune-disturb_pen15.0_all-Disturb_Pred-gamma2.0__stage_00__steps_000025054444.pt

python main.py projects/manipulathor_disturb_free/armpointnav_baselines/experiments/eval/TestScene.py \
    -s 1 --save_dir_fmt NESTED --infer_output_dir --eval -i \
    -c $TEST_CKPT

Visualization

For visualization, please use the following configs in TestScene.py:

    VISUALIZERS = [
        lambda exp_name: ImageVisualizer(exp_name, 
            add_top_down_view=True,
            add_depth_map=True,
        ),
        lambda exp_name: TestMetricLogger(exp_name),
    ]
    CAMERA_WIDTH = (
        # 224
        224 * 2 # higher resolution for better visualization, won't change the agent obs shape
    )
    CAMERA_HEIGHT = (
        # 224
        224 * 2
    )

    NUM_TASK_PER_SCENE = (
        # None
        6 # set a small number for light storage
    )

Then run the following script (we take our best model as example):

# Currently we only support visualization on a single checkpoint
python main.py projects/manipulathor_disturb_free/armpointnav_baselines/experiments/eval/TestScene.py \
    -s 1 --save_dir_fmt NESTED --infer_output_dir --eval -i \
    -c $TEST_CKPT \
    --config_kwargs '{"test_ckpt": "'$TEST_CKPT'"}'

It will generate the gifs of both egocentric and top-down view with annotation of current disturbance distance in meter on the top-left corner. It will also generate the start and goal image in RGB and depth, and all the evaluation metrics in json.

Core Functions

• Disturbance distance sensor
• Disturbance distance penalty
• Disturbance distance prediction task

Contact

If you have any questions, please create an issue or contact Tianwei (tianwei.ni@mila.quebec)

Acknowledgement

This repository uses AllenAct as codebase and ManipulaTHOR as testbed. GroupNormResNet18 and auxiliary tasks are heavily based on Auxiliary Tasks Speed Up Learning PointGoal Navigation.

Citation

@article{ni2021towards,
  title={Towards Disturbance-Free Visual Mobile Manipulation},
  author={Ni, Tianwei and Ehsani, Kiana and Weihs, Luca and Salvador, Jordi},
  journal={arXiv preprint arXiv:2112.12612},
  year={2021}
}