Pytorch Implementation of Lattice Point Classifier (LPC)

Created by Kaidong Li, Ziming Zhang, Cuncong Zhong, Guanghui Wang

This repo is the official implementation for Robust Structured Declarative Classifiers for 3D Point Clouds: Defending Adversarial Attacks with Implicit Gradients. We study its classification and defense performances against some state-of-the-art adversarial attacks and compare the results with state-of-the-art defenses.

We also collect the implementations of these 3D point cloud attacks (FGSM, JGBA, Perturbation Add Cluster and Object Attacks, CTRI) and defenses (DUP-Net, IF-Defense and RPL) in this repository, in the hope of serving as a reference point for future research.

Environments

We run our experiments under:

Python 3.6.9
GCC 6.3.0
CUDA 10.1.105
torch 1.7.1
pytorch3d 0.4.0
open3d 0.9.0.0

And other packages might be needed.

Classification

Data Preparation

• ModelNet40

  Download alignment ModelNet40 here and save in data/modelnet40_normal_resampled/.

• ScanNet

  Please refer to here for downloading ScanNet data.

  Refer to here for converting the data to classification task and save in data/scannet/.

  • Once receiving ScanNet download script from official site, download the v2 "_vh_clean_2.ply", "_vh_clean_2.0.010000.segs.json", ".aggregation.json" and label map files
  • Once download completes, run extract_scannet_objs.py and prepare_scannet_cls_data.py in this repo to get classification data as suggested.
  • During extract, there are 5 mismatched class name pairs ("trash", "trash can"), ("bathtub", "tub"), ("shelf", "bookshelf"), ("keyboard", "computer keyboard") and ("tv", "monitor or tv"). We modify extract_scannet_objs.py to link these pairs and change to the former name.

Run Classification

# train LPC with EfficientNet-B5 as backbone and scaling factor 456 on ScanNet dataset
python train_cls.py --dataset ScanNetCls --num_cls 17 --model lattice_cls --backbone efficientnet-b5 --dim 456 --learning_rate 0.0001 --log_dir lpc_eff_456 --batch_size 4

# train PointNet on ModelNet40 dataset
python train_cls.py --model pointnet_cls --dataset ModelNet40 --num_cls 40 --log_dir pointnet_cls --batch_size 24

Refer to train_cls.py for detailed options. Best model from training will be saved in log/classification/***/checkpoints/best_model.pth.

To switch between binarized weights and barycentric weights, it is currently done manually. Find the following lines in models/lattice_cls.py. It is binarized weights. By commenting out all the lines, the LPC models use barycentric weights.

if not self.normal_channel:
    # next two lines are for cutoff weights
    # cutoff = filter_2d[filter_2d>0].mean() * 2
    # filter_2d[filter_2d>cutoff] = cutoff
    # next line is for binarized weights
    filter_2d[filter_2d>0] = 1.0

DUP-Net and IF-Defense have similar architecture of their specific module plus PointNet. We can use the previously trained PointNet model. Download DUP-Net specific pretrained model pu-in_1024-up_4.pth from here and save in dump/.

PointNet with RPL can be trained using our code. But we also recommend using this implementation for faster result and moving the trained model here.

Accuracy

Model	ModelNet40	ScanNet
PointNet	90.15	84.61
DUP-Net	89.30	83.62
IF-Defense	87.60	80.19
PointNet RPL	84.76	76.02
LPC w/ vgg16	88.65	--
LPC w/ resnet50	88.90	--
LPC w/ efficientnet-b5	89.51	83.16

Attack

Model Preparation

• FGSM, JGBA, Perturbation

  Copy model folder from classifcation result log/classification/ to log/perturbation/

• Add, Cluster, Object Attacks

  Copy model folder from classifcation result log/classification/ to log/attacks/

• CTRI

  Copy saved .pth model from classifcation result log/classification/***/checkpoints/best_model.pth to log_ctri_attack/

• DUP-Net

  Download pretrained model pu-in_1024-up_4.pth from here to dump/ and log_ctri_attack/

Run

Be careful and match model parameters with the setups during training process

• FGSM/JGBA attack (on PointNet, PointNet w/ RPL and LPC, except DUP-Net and IF-Defense):

# FGSM/JGBA attack on PointNet model on ModelNet40
python pert_JGBA_attack.py --attack [FGSM/JGBA] --model pointnet_cls --dataset ModelNet40 --num_cls 40 --log_dir [folder_name_under_perturbation]

• FGSM/JGBA attack (for both DUP-Net and IF-Defense):

# FGSM/JGBA attack on ScanNet
python pert_JGBA_attack_SOR.py --attack [FGSM/JGBA] --dataset ScanNetCls --num_cls 17 --log_dir [folder_name_under_perturbation]

• Perturbation, add, cluster and object attacks (on PointNet, PointNet w/ RPL and LPC, except DUP-Net and IF-Defense):

# Perturbation attack on PointNet on ModelNet40
python perturbation_attack.py --batch_size 6 --learning_rate 0.01 --target 0 --model pointnet_cls --log_dir [folder_name_under_perturbation]

# Add attack on RPL on ScanNet, adding 60 independent points
python independent_attack.py --batch_size 6 --learning_rate 0.01 --target 0 --model pointnet_ddn --add_num 60 --log_dir [folder_name_under_attacks] --dataset ScanNetCls --num_cls 17

# Cluster attack on LPC with vgg16 and scaling factor 512 on ModelNet40, adding 3 clusters of 32 points
python cluster_attack.py --batch_size 6 --learning_rate 0.01 --target 0 --model lattice_cls --backbone vgg16 --dim 512 --add_num 32 --num_cluster 3 --log_dir [folder_name_under_attacks] --eps 0.11 --mu 0.1 --init_pt_batch 8 --max_num 32

Download the object airplane.npy to be attached from here, and copy it to data/.

# Object attack on PointNet on ModelNet40, adding 3 objects of 64 points
python object_attack.py --batch_size 6 --learning_rate 0.01 --target 0 --model pointnet_cls --add_num 64 --num_cluster 3 --log_dir [folder_name_under_attacks] --eps 0.11 --mu 0.1 --init_pt_batch 8 --max_num 32

The above attack results (e.g. adversarial samples) will be saved in a folder attacked_[attack_name] under $log_dir. For DUP-Net and IF-Defense, perturbation, add, cluster and object attacks are done on clean PointNet. Then package the attack results under the attacked_[attack_name] folder using data_analysis/sum-[attack_name].py and move to here for defense performance analysis.

• CTRI attack (on PointNet, DUP-Net, PointNet w/ RPL and LPC, except IF-Defense):

# CTRI attack on DUP-Net on ModelNet40
python ctri_attack.py --data modelnet40 --model dupnet --model_path [pretrain_name_under_log_ctri_attack] --num_points 1024

Reference Implementations

laoreja/HPLFlowNet
yanx27/Pointnet_Pointnet2_pytorch
Wuziyi616/IF-Defense
xiangchong1/3d-adv-pc
yangyanli/PointCNN
pytorch/fgsm
machengcheng2016/JGBA-pointcloud-attack
anucvml/ddn
skywalker6174/3d-isometry-robust