This repository contains the code for the paper Exploiting feature-rich image locations for adversarial attacks on image classifiers without network access. The code provided performs both training of networks using image-gradient pairs, as well as the capability to generate gradient maps using sets of surrogate models.
The README is divided into six sections:
- Introduction A brief overview of what Multi-Targeted Gradient Training (MTGT) is and why it matters.
- Installation A quick guide to installing this project.
- Configuration How to set up YAML configs for training (loss, epochs, models) and dataset creation (surrogate ensembles, output paths).
- Training with MTGT Documentation for using the provided code to train models.
- Creating Combined Gradient Dataset Step-by-step documentation for generating custom datasets with the code.
- Available Datasets Links to and descriptions of existing datasets that are ready to use.
Multi-Targeted Gradient Training (MTGT) is a method for generating saliency maps that enable adversarial attacks without direct access to the target network. An adversarial example is a small, carefully designed perturbation to input data that causes a neural network to misclassify it. For example, making an image of a cat be recognized as a dog after only subtle pixel-level changes. Many existing attacks rely on backpropagating loss gradients through the target model, which requires access to its weights. The resulting maps translate to adversarial attacks having direct knowledge of how to change pixel values such that they can increase loss via gradients (as opposed to traditional training which changes neuron parameters to reduce loss). The major logistical concern for adversarial attacks is the requirement to have live access to a target model to perform that backpropagation. MTGT bypasses this requirement by training an encoder-decoder network to directly output “feature-rich locations” for any input image, identifying regions that adversarial attacks can exploit.
This repository provides the official implementation of MTGT, which is built around the concept of combined gradients. Instead of relying on a single surrogate network, MTGT aggregates saliency maps from an ensemble of diverse models (e.g., ResNets, VGGs, RegNets). Each model contributes its own gradient map for the same input image, and these maps are aligned, optionally normalized or weighted, and then averaged together into a single combined gradient map. This captures feature-rich regions that are consistently emphasized across architectures, improving transferability to unseen networks. The repository includes tools to generate such combined gradient datasets, training utilities with custom loss functions for MTGT, public links to precomputed datasets, and modular configs to create custom datasets tailored to specific downstream attacks.
MTGT is intended for researchers and practitioners interested in adversarial robustness, transferability, and dataset-driven approaches to attack design. The codebase is modular and extensible, making it straightforward to reproduce results or adapt MTGT to new architectures and applications. Originally, MTGT was developed to support the Wavelength-Specific Map Attack (WSMA), a physically realized adversarial attack using laser-based holography.
Clone the repository and install in editable mode:
git clone https://github.com/scottgHDS/mtgt.git
cd mtgtStandard install (just use MTGT as a library) pip install .
Editable install(If you want to edit MTGT in your IDE) pip install -e .
MTGT uses YAML configuration files parsed with OmegaConf. Configs live in the 'configs' directory and fall into two categories:
- Training configs - control how networks are trained on image-gradient pairs.
- Dataset creation configs - control how combined gradient maps are generated from ensembles of surrogate models.
At runtime, you can override any parameter in a config file directly from the command line using key=value syntax. This allows quick experimentation without editing the base YAML.
All major workflows in this repo follow the same pattern:
- A YAML config file is loaded (from
configs/). - Any command-line overrides (
key=value) are applied. - The config is passed into the appropriate CLI entry point, which calls the core implementation.
For example:
- Training uses
mtgt/cli/train.py, which loads the config and invokesmtgt/training/loop.py. - Dataset creation uses
mtgt/cli/create_combined_gradients.py, which contains the functions that directly implement dataset creation (model management, gradient combination, and saving).
Hyperparameters and general settings are specified through a YAML file: configs/train.yaml
This file defines all parameters used in MTGT training, including:
- data: dataset paths, batch size, workers
- model: architecture type and encoder checkpoint
- loss: choice of loss function (either
order_topnormse) - optim: optimizer settings (SGD or AdamW)
- sched: learning-rate scheduler (plateau or cosine)
- train: training parameters (epochs, AMP, gradient clipping, validation frequency)
- ckpt: checkpoint directory and resume path
python -m mtgt.cli.train \
--config configs/train.yaml \
training.epochs=30 \
loss.name=mseAll dataset creation settings are specified through a YAML file: configs/create_dataset.yaml
This file defines all parameters used in dataset creation, including:
- data: raw image path
- output: path to save combined gradients
- dataset: details of the creation processes themselves
- logging: how often to log progress
python -m mtgt.cli.create_combined_gradients \
--config configs/create_gradients.yaml \
dataset.color=R
The mtgt/data/loaders.py module provides utilities for pairing ImageNet images
with precomputed gradient maps.
MTGTDataset: A PyTorchDatasetreturning(image_tensor, grad_tensor, image_id)tuples.build_datasets: A factory function to create train/val splits.
images_root/train//<img_id>.JPEG grads_root/train//<img_id>.npy
wnid is the WordNet ID for the ImageNet class (e.g., n01440764 which is for the tench class).
A checkpoint at the end of every training epoch will be saved as last.pth. Additionally, the model which performed strongest on the validation set will also be seperatly saved as best.pth.
All dataset and checkpoints in the YAML config can be specified as either:
- Relative paths (recommended): resolved relative to the configuration file Note an absolute path should not have a leading slash ( / ), or it will be read as an absolute path
- Absolute paths: respected as-is, useful for shared datasets such as ImageNet-1k
During training, this project automatically exapnds '~' (the home directory) and
environment variables (e.g., $DATA_ROOT) when resolving paths. The resolved config
is copied into the run directory for reproducibility.
The package mtgt/models provides MTGT-ready architectures and a factory for constructing them from the YAML config.
- "resnet_unet": ResNet-50 encoder with U-Net-style decoder (skip connections).
- "resnet_ae"/"resnet_autoencoder" - ResNet-50 encoder with a lightweight upsampling decoder
Both models output a single-channel map by default (out_channels: 1) and apply a positive normalization step.
Select the model in your OmegaConf config (configs/train.yaml):
model:
name: resnet_unet # {"resnet_unet", "resnet_ae"}
out_channels: 1 # default in this repo
encoder_ckpt: null # optional: path to encoder weights- Input: (Batch_size, 3, 224, 224) - ImageNet preprocessing (normalization) assumed in the loaders
- Output: (Batch_size, out_channels, 224, 224) - single-channel saliency map (default).
Build the model from config and move it to device:
from mtgt.models import build_model
from mtgt.utils.device import get_device
device = get_device()
model = build_model(cfg).to(device)Direct class usage (if you do not want to use the factory):
from mtgt.models.resnet_unet import ResNetUNet
model = ResNetUNet(out_channels=1)The module mtgt/training/losses.py provides the loss functions used in MTGT.
- MSE (baseline): Standard mean squared error between prediction and target maps. Not effective for MTGT.
- Order-based Top-N (default in the paper): A order-based loss that compares pairwise differences between the Top-N target pixels and all other pixels. See the paper for details.
Select the loss in the OmegaConf config (configs/train.yaml):
loss:
name: mseor
loss:
name: order_topn
top_n: 25 # Your choice of N
The module mtgt/evaluation/validation.py provides
a helper function for computing the mean validation loss over a dataloader
during training.
from mtgt.evaluation.validation import evaluate
val_loss = evaluate(model, val_loader, loss_fn, device)
print(f"Validation Loss: {val_loss:.4f}")The module mtgt/training/optim.py provides
a factory to build the model's optimizer, configured via the YAML file.
Supported:
optim.name: "sgd"- useslr,momentum,weight_decayoptim.name: "adamw"- useslr,weight_decay
from mtgt.training.optim import build_optimizer
optimizer = build_optimizer(model, cfg)The module mtgt/training/sched.py provides
a factory to build the optimizer's scheduler, configured via the YAML file.
Supported:
sched.name: "plateau"- usesfactor,patiencesched.name: "cosine"- usesT_max,eta_min
PyTorch's ReduceLROnPlateau reduces the LR after the specified patience
number of epochs plus one without improvement. For example, if you pass
patience=2, the LR is reduced after 3 stagnant epochs.
This repo corrects that by subtracting 1 internally, so the YAML value matches plain language:
Meaning
sched:
name: plateau # {"plateau", "cosine"}
factor: 0.1 # only use this pair when name == "plateau"
patience: 3 #will require 3 epochs of non-improving validation to reduce the learning rate.
from mtgt.training.sched import build_sched
sched = build_scheduler(optimizer, cfg)mtgt/utils/checkpoint.py provides simple save/load helpers
for the model being trained.
Note: Loading a pretrained encoder is handled separately in
mtgt/models/build_model.py
from mtgt.utils.checkpoint import save_ckpt, load_ckpt
save_ckpt("runs/exp1/epoch_10.pt", epoch, model, optimizer, scheduler)
state = load_ckpt("runs/exp1/epoch_10.pt", model, optimizer, scheduler)
start_epoch = state.get("epoch", 0) + 1mtgt/utils/logging.py provides a simple logger
- Logs include timestamps and levels (
INFO,WARNING,ERROR). - Can optionally write to a file in the run directory.
from mtgt.utils.logging import get_logger
logger = get_logger(log_file="runs/exp1/train.log")
logger.info("Training started")mtgt/utils/seed.py provides set_seed(seed, deterministic=True) to make runs reproducible
using a common seed.
from mtgt.utils.seed import set_seed
set_seed(42)mtgt/utils/device.py provides helpers to neatly discover which computational
device is available and to easily transfer tensors to the GPU if available.
from mtgt.utils.device import get_device, to_device
device = get_device()
x, y = to_device(x, y, device=device)Three precomputed gradient target datasets are available for download if you do not wish to generate your own.
Each dataset contains combined gradient maps derived from multiple source networks in the PyTorch Model Library.
The gradients were optimized for a cyan (488 nm) Wavelength-Specific Map Attack (WSMA) by weighting and summing across RGB channels.
- Based on ImageNet-1K (training and validation splits).
- Each sample corresponds to one original ImageNet image.
- Stored as single-channel maps of shape (1 x 224 x 224).
- Computed by:
- Averaging the cross-entropy gradient from each source network w.r.t. the true class.
- Multiplying channel-wise by the RGB translation of a 488 nm cyan laser.
- Summing across channels.
- Preprocessing: input images resized so the shortest side = 256 px (preserving aspect ratio), then center-cropped to 224 x 224.
-
Combined Cross-Entropy Gradients From Diverse Sources for MTGT With Cyan WSMA
-Source networks: AlexNet, DenseNet-161, EfficientNet-B0, GoogLeNet, MNASNet (depth multiplier 0.5), MobileNetV2, MobileNetV3-Small, RegNetY-800MF, ResNet-50, ResNeXt-50 (32 X 4d), ShuffleNetV2-x0.5, SqueezeNet1.1, VGG-16, and Wide ResNet-50 v2-Purpose: Approximates gradient behavior across a diverse range of network architectures and scales.
-
Combined Cross-Entropy Gradients From Non-Residual Sources for MTGT With Cyan WSMA
-Source networks: SqueezeNet 1.0, SqueezeNet 1.1, GoogLeNet, AlexNet, VGG-11, VGG-11 with batch normalization, VGG-13, VGG-16, and VGG-16 with batch-Purpose: Approximates gradient behavior across networks without residual connections.
-
Combined Cross-Entropy Gradients From Residual Sources for MTGT With Cyan WSMA
-Source networks: RegNetX-400MF, RegNetX-800MF, RegNetX-1.6GF, RegNetY-400MF, RegNetY-800MF, RegNetY-1.6GF, RegNetY-3.2GF, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, ResNeXt-50 (32 X 4d), ResNeXt-101 (32 X 8d), Wide ResNet-50-2, and Wide ResNet-101-2-Purpose: Approximates gradient behavior across networks with residual connections.
This project allows for users to create their own combined gradient datasets. Dataset creation uses an ensemble of classifiers to compute gradient-based saliency maps for each training/validation image. Each map is optionally weighted by wavelength (for Wavelength-Specific Map Attack experiments), collapsed channel-wise, normalized, and then averaged across the ensemble to produce a single combined gradient map.
flowchart TD
A[Input image] --> B{Models};
B -->|resnet50| C1[Saliency map 3-ch];
B -->|vgg16_bn| C2[Saliency map 3-ch];
B -->|regnet_y_800mf| C3[Saliency map 3-ch];
C1 --> W1["Color weighting<br>(R,G,B * w(color)"];
W1 --> S1["Channel collapse<br>(sum RGB to 1-ch)"];
S1 --> N1[Positive normalize];
C2 --> W2["Color weighting<br>(R,G,B * w(color))"];
W2 --> S2["Channel collapse<br>(sum RGB to 1-ch)"];
S2 --> N2[Positive normalize];
C3 --> W3["Color weighting<br>(R,G,B * w(color))"];
W3 --> S3["Channel collapse<br>(sum RGB to 1-ch)"];
S3 --> N3[Positive normalize];
N1 --> J[Average across models];
N2 --> J;
N3 --> J;
J --> H["Final combined map (1-ch)"];
Direct creation of combined gradient datasets can be directly accomplished using ModelManagerConfig and GradientModelsManager.
import torch
from mtgt.gradient_models_manager import ModelManagerConfig, GradientModelsManager
from mtgt.data.gradient_combiner import combine_gradients
# Configure which models and saliency type to use
cfg = ModelManagerConfig(
model_names=["resnet50", "vgg16"],
saliency="jacobian",
normalize_output=True,
)
manager = ModelManager(cfg, device="cuda")
# Example inputs
images = torch.randn(4, 3, 224, 224, device="cuda") # batch of 4 images
labels = torch.tensor([0, 1, 2, 3], device="cuda") # class indices
# Compute per-model saliency (M,B,3,H,W)
grads_mbchw = manager.gradients_for_batch(images, labels)
# Combine across models into (B,H,W)
combined = combine_gradients(grads_mbchw, color="G", normalize=True)mtgt/data/gradient_model_manager.py contains the model manager class which handles how to generate the saliency
maps from the model ensemble that are eventually used to create the combined gradient maps.
The models chosen for dataset creation can be individually chosen using the YAML:
dataset:
models:
names: [alexnet, vgg16, squeezenet1_1] # list specific model names (overrides present if non-empty)
All models loaded are Pytorch pre-trained models, using the v1 edition of their weights. A list of all available models can be found on the pytorch pytorch model webpage. Refer to the PyTorch model zoo documentation to identify the correct builder name, and list it in your YAML file.
In the original paper four subsets of gradient source ensembles were tested. Three of them are available for download in Available Datasets. All four are able to be easily selected using the yaml as follows:
dataset:
models:
preset: diverse_16 # {diverse_16, resid_4, resid_16, non_resid_8}Each subset and their respective source networks:
- diverse_16 AlexNet, DenseNet-161, EfficientNet-B0, GoogLeNet, MNASNet (depth multiplier 0.5), MobileNetV2, MobileNetV3-Small, RegNetY-800MF, ResNet-50, ResNeXt-50 (32 X 4d), ShuffleNetV2-x0.5, SqueezeNet1.1, VGG-16, and Wide ResNet-50 v2
- resid_4 RegNetY-800MF, ResNet-50, ResNeXt-50 (32 X 4d), Wide ResNet-50-2
- resid_16 RegNetX-400MF, RegNetX-800MF, RegNetX-1.6GF, RegNetY-400MF, RegNetY-800MF, RegNetY-1.6GF, RegNetY-3.2GF, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, ResNeXt-50 (32 X 4d), ResNeXt-101 (32 X 8d), Wide ResNet-50-2, and Wide ResNet-101-2
- non_resid_8 SqueezeNet 1.0, SqueezeNet 1.1, GoogLeNet, AlexNet, VGG-11, VGG-11 with batch normalization, VGG-13, VGG-16, and VGG-16 with batch
The preset "diverse_16" includes two models (ConvNeXt and MaxVit) that used different preprocessing during their training compared to the rest of the models. They were fine-tuned before dataset correction for consistency. The fine-tuned weights can be downloaded with:
python scripts/download_checkpoints.pyThey will be saved to the folder "weights/backbones" at project root and will be automatically loaded during dataset generation if indicated in dataset.models.checkpoints as shown in the default create_dataset.yaml.
The mtgt/data/raw_loaders.py module provides utilities for loading raw images
and their true class labels.
ImageFolderwithIDS: A PyTorchDatasetreturning(image_tensor, target, image_id)tuples.build_raw_image_datasets: A factory function to create train/val splits with ImageNet-style preprocessing.
The module mtgt/data/gradient_combiner.py
converts raw per-model saliency maps into a single combined gradient map per image.
This step is required when constructing the combined gradient dataset from the gradient source ensemble.
The pipeline is:
-
Color weighting - Apply an RGB weight corresponding to a laser/display color. Supported codes:
"R"(red ~650 nm)"O"(orange ~593 nm)"Y"(yellow ~589 nm)"G"(green ~532 nm)"C"(cyan ~488 nm)"B"(blue ~450 nm)"V"(violet ~405 nm)"N"(no weighting, RGB channels treated equally)
-
Channel collapse - Sum across RGB channels to produce a single-channel map per model.
-
Model averaging - Average across models to produce one map per sample.
-
Normalization (optional) - Shift each map to [0, 1] for consistent scaling. This is recommended because the outputs of the trained MTGT encoder-decoder networks are positive normalized.
This snippet one normalized map per input image in the batch, ready for saving in the combined gradient dataset.
from mtgt.data.gradient_combiner import combine_gradients
# grads_mbchw: (M,B,3,H,W) tensor from ModelManager
combined = combine_gradients(grads_mbchw, color="G", normalize=True) # (B,H,W)Combined gradients are saved as '.npy' files (float32, shape 'H x W'), per one image. For example: (output.dir)/train/(image_id).npy
Helpers in mtgt/utils/io.py provide programattic access:
- 'save_combined_gradients_batch(out_dir, split, image_ids, maps_bhw)`
- 'save_combined_gradient(out_dir, split, image_id, map_hw)`
- 'load_combined_gradient(root, split, image_id, as_tensor=False)`
If you use MTGT in your research, please cite:
S. G. Hodes, Exploiting feature-rich image locations for adversarial attacks on image classifiers without network access, SPIE Conference on Defense + Commercial Sensing, 2025. Link
This project is licensed under the MIT License – see the LICENSE file for details.