Project Page | Video | arXiv | Paper | Data
This repository contains the official code release for Totems: Physical Objects for Verifying Visual Integrity. The code was based on this NeRF implementation here.
git clone https://github.com/jingweim/totems.git
cd totems
# Make a conda environment
conda create --name totems python=3.6
conda activate totems
# Install requirements
pip install -r requirements.txt
Pre-compiled data (link)
The pre-compiled dataset contains manipulated and unmanipulated photos that have gone through pre-processing (i.e. undistortion, computing initial totem pose) and are ready to run with the reconstruction and detection code. In the current release we only have one manipulation type: randomly added color patches. The other manipulation types are coming soon.
data-compiled/
calib.npy # Shared camera calibration
unmanipulated/
JT8A8283/
JT8A8283.JPG # image from camera
image.png # undistorted using calib.npy
initial_totem_pose.npy # totem poses used for initialization, estimated from totem masks
totem_masks/ # annotated+undistorted totem masks, 0-n = left-right
totem_000.png # white = totem, black = scene
totem_001.png
...
color-patch/ # Randomly added color patches
JT8A8283_000/
JT8A8283_000.png # Manipulated image
JT8A8283_000_mask.png # Manipulation mask
image.png # Maipulated image (undistorted)
manip_mask.png # Manipulation mask (undistorted)
initial_totem_pose.npy # totem poses used for initialization, estimated from totem masks
totem_masks/ # annotated+undistorted totem masks, 0-n = left-right
totem_000.png # white = totem, black = scene
totem_001.png
...
...
The raw dataset includes JPG files straight from a Canon EOS 5D Mark III camera, annotated totem masks, the checkerboard images used for camera calibration, and the computed calibration.
Scripts that preprocess the raw data into the compiled version, compute camera calibration, or generate manipulations.
# Fit NeRF to scene (jointly optimizing totem poses and NeRF)
python run.py --config configs/color-patch_JT8A8283_000_joint-pose.txt
# Fit NeRF to scene (fixed totem poses)
python run.py --config configs/color-patch_JT8A8283_000_initial-pose.txt
# Monitor reconstruction progress
tensorboard --logdir logs/summaries
# Check reconstruction results and save intermediate files for detection stage
python run.py --config configs/color-patch_JT8A8283_000_joint-pose.txt \
--render_only --render_cam --render_totem
python run.py --config configs/color-patch_JT8A8283_000_joint-pose.txt --detect_only
logs/
${EXP1_NAME}/ # Experiment folder
010000.tar # Model at iteration 10000
...
050000.tar # Model at iteration 50000
args.txt # All arguments and values
config.txt # A copy of the config file
loss.txt # Losses every 100 steps
render_050000/ # Camera and totem view renderings
camera_view.png
totem_views/
detect_050000/ # Detection intermediate files and results
intermediate/ # Intermediate files generated from dataset and trained model
image.png # Input image
recon.png # Reconstructed image
totem_mask.png # Merging individual totem masks
protect_mask.png # Generated protect mask
protect_mask/ # Protect mask intermediate files
results/ # Detection results
L1/
heatmap.npy # grid_size x grid_size numpy array
heatmap_vis.png # Resized to image size and color mapped with matplotlib 'jet'
heatmap_overlay.png # Overlayed on top of manipulated(?) image
metrics.npy # Dictionary storing image-wise and patch-wise metrics
LPIPS/ # Same as the above folder, using LPIPS metric instead
...
${EXP2_NAME}/
...
summaries/ # Tensorboard files
${EXP1_NAME}/
${EXP2_NAME}/
...
If you find our work useful, please consider citing:
@article{ma2022totems
author = {Ma, Jingwei
and Chai, Lucy
and Huh, Minyoung
and Wang, Tongzhou
and Lim, Ser-Nam
and Isola, Phillip
and Torralba, Antonio},
title = {Totems: Physical Objects for Verifying Visual Integrity},
journal = {ECCV},
year = {2022},
}