WeedSwin hierarchical vision transformer with SAM-2 for multi-stage weed detection and classification

Taminul Islam¹, Toqi Tahamid Sarker¹,Khaled R Ahmed¹, Cristiana Bernardi Rankrape² Karla Gage³,

¹ School of Computing, ² School of Agricultural Sciences, ³ School of Biological Sciences

Southern Illinois University Carbondale

• Abstract
• Getting Started
  • WeedSwin Installation

Abstract

Weed detection and classification using computer vision and deep learning techniques have emerged as crucial tools for precision agriculture, offering automated solutions for sustainable farming practices. This study presents a comprehensive approach to weed identification across multiple growth stages, addressing the challenges of detecting and classifying diverse weed species throughout their developmental cycles. We introduce two extensive datasets: the Alpha Weed Dataset (AWD) with 203,567 images and the Beta Weed Dataset (BWD) with 120,341 images, collectively documenting 16 prevalent weed species across 11 growth stages. The datasets were preprocessed using both traditional computer vision techniques and the advanced SAM-2 model, ensuring high-quality annotations with segmentation masks and precise bounding boxes. Our research evaluates several state-of-the-art object detection architectures, including DINO Transformer (with ResNet-101 and Swin backbones), Detection Transformer (DETR), EfficientNet B4, YOLO v8, and RetinaNet. Additionally, we propose a novel WeedSwin Transformer architecture specifically designed to address the unique challenges of weed detection, such as complex morphological variations and overlapping vegetation patterns. Through rigorous experimentation, WeedSwin demonstrated superior performance, achieving 0.993 ± 0.004 mAP and 0.985 mAR while maintaining practical processing speeds of 218.27 FPS, outperforming existing architectures across various metrics. The comprehensive evaluation across different growth stages reveals the robustness of our approach, particularly in detecting challenging "driver weeds" that significantly impact agricultural productivity. By providing accurate, automated weed identification capabilities, this research establishes a foundation for more efficient and environmentally sustainable weed management practices. The demonstrated success of the WeedSwin architecture, combined with our extensive temporal datasets, represents a significant advancement in agricultural computer vision, supporting the evolution of precision farming techniques while promoting reduced herbicide usage and improved crop management efficiency.

Getting Started

WeedSwin Installation

Step 1: Clone the WeedSwin repository:

git clone https://github.com/taminulislam/weeds.git
cd weeds

Step 2: Environment Setup:

Create and activate a new conda environment:

conda create -n weedswin python=3.8
conda activate weedswin

Install PyTorch and CUDA:

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Install MMDetection and dependencies:

pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
mim install "mmdet>=3.0.0"

Install additional packages:

pip install -r requirements.txt

Step 3: Verify Installation:

python -c "import torch, mmcv, mmdet; print(torch.__version__, mmcv.__version__, mmdet.__version__)"

Training WeedSwin

Train the model using the provided configuration:

python train_weedswin.py

Or use MMDetection's training script:

python tools/train.py configs/weedswin.py

Testing WeedSwin

Test the trained model:

python test_weedswin.py --checkpoint work_dirs/weedswin/epoch_12.pth

Or use MMDetection's testing script:

python tools/test.py configs/weedswin.py work_dirs/weedswin/epoch_12.pth

Model Architecture

WeedSwin combines:

• Swin Transformer backbone (depths=[2, 2, 18, 4], embed_dims=192)
• DINO detection head with 4-scale features
• 174 weed classes covering 16 species across 11 growth stages

Key features:

• Multi-scale feature extraction with hierarchical architecture
• Deformable attention for efficient processing
• Optimized for 218.27 FPS inference speed
• Achieves 0.993 ± 0.004 mAP

Dataset Format

WeedSwin uses COCO format for annotations. Organize your dataset as follows:

data/
├── annotations/
│   ├── train.json
│   ├── val.json
│   └── test.json
├── train/
│   └── *.jpg
├── val/
│   └── *.jpg
└── test/
    └── *.jpg

Update the data_root path in configs/weedswin.py to point to your dataset location.

Dataset

The Weed Growth Stage Dataset used in this research is publicly available on Zenodo:

Dataset: https://doi.org/10.5281/zenodo.15808623

Additional Documentation

• QUICKSTART.md - Quick reference guide for training and testing
• STRUCTURE.md - Repository structure and file organization

Citation

If you use WeedSwin in your research, please cite our paper:

@article{islam2025weedswin,
  title={WeedSwin hierarchical vision transformer with SAM-2 for multi-stage weed detection and classification},
  author={Islam, Taminul and Sarker, Toqi Tahamid and Ahmed, Khaled R and Rankrape, Cristiana Bernardi and Gage, Karla},
  journal={Scientific Reports},
  volume={15},
  number={1},
  pages={23274},
  year={2025},
  publisher={Nature Publishing Group UK London}
}