UAV Swarm Intelligence Platform

Autonomous multi-drone simulation system integrating real-time YOLOv8 perception, ADS-B transponder fusion, agentic swarm coordination, and LLM-driven autopilot backends — built on Microsoft AirSim.

---

Overview

This project is a research-grade UAV software stack developed as part of graduate AI research. It provides a full Python control layer over AirSim's multirotor RPC interface, enabling:

• Multi-modal sensor fusion — RGB, depth, LiDAR, IMU, GPS, barometer
• Real-time object detection — YOLOv8 with configurable model variants
• Autonomous mission planning — deterministic waypoint generation with safety enforcement
• Agentic swarm control — role-based coordination across multiple UAVs
• LLM-integrated autopilot — pluggable backends (Ollama, Hugging Face Transformers, NVIDIA API)
• ADS-B synthetic traffic — SBS-1 feed compatible with QGroundControl
• Training data pipeline — automated JSONL export from patrol replay artifacts

---

Architecture

uav-swarm-airsim/
├── AirSimSensors/
│   ├── detect.py                    # Fused RGB / depth / LiDAR perception + YOLO overlay
│   ├── autopilot.py                 # Single-drone autonomous waypoint pilot (LLM-backed)
│   ├── vehicle_runtime.py           # Vehicle selection + PX4 / SimpleFlight fallback logic
│   ├── all_sensors.py               # Combined sensor integration test
│   ├── rgb_camera.py
│   ├── depth_camera.py
│   ├── lidar.py
│   ├── imu.py
│   └── imu_mag_gps_barometer.py
├── SwarmRun/
│   ├── agentic_swarm.py             # Role-based multi-drone formation controller
│   ├── monitoring_patrol.py         # Patrol mission with alerts, geofence, and replay logging
│   ├── multidrone.py                # Spawn and initialize a SimpleFlight swarm
│   ├── MultidroneManual.py          # Keyboard control for spawned swarm
│   └── delete_drones.py             # Clean up spawned Drone* vehicles from scene
├── tools/
│   └── export_training_dataset.py   # Convert patrol replay runs to JSONL training data
├── Architecture_Doc/
│   └── architecture.md              # System design and data-flow documentation
├── manualcontrol.py                 # Keyboard flight control for a single drone
├── adsb_sbs_server.py               # Synthetic SBS-1 ADS-B server for QGroundControl
├── app.py                           # Multi-drone YOLO detection demo (yellow-X target)
├── gui.py                           # PX4 / MAVSDK Tkinter UI
├── test1.py / test2.py              # MAVSDK integration smoke tests
└── README.md

---

Workflow Reference

Workflow	Entry Point	Description
Sensor fusion viewer	AirSimSensors/detect.py	Live RGB, depth, LiDAR, DBSCAN clustering, YOLO overlay, trajectory replay
Manual single-drone	manualcontrol.py	Keyboard flight via AirSim RPC with PX4 diagnostics and fallback spawning
Autonomous pilot	AirSimSensors/autopilot.py	LLM-driven waypoint missions with deterministic planning and safety enforcement
Monitoring patrol	SwarmRun/monitoring_patrol.py	Sweep/waypoint patrol with geofencing, YOLO watch labels, event logging, and MP4 recording
Swarm spawn	SwarmRun/multidrone.py	Initialize a default SimpleFlight swarm in the AirSim scene
Agentic swarm	SwarmRun/agentic_swarm.py	Role-based formation control for discovered Drone* vehicles
Swarm manual	SwarmRun/MultidroneManual.py	Keyboard control over a spawned swarm
ADS-B server	adsb_sbs_server.py	Synthetic SBS-1 traffic feed for QGroundControl
MAVSDK GUI	gui.py	Tkinter interface for PX4/MAVSDK testing

---

Requirements

Simulator

A running AirSim scene exposing the RPC server is required before any script can connect.

• SimpleFlight — Python scripts control the drone directly via RPC.
• PX4Multirotor — Requires PX4 SITL running with matching ports before launch.
• QGroundControl (optional) — For external telemetry and ADS-B visualization.
• Ollama (optional) — Local LLM backend for autopilot and patrol decision-making.

Python Dependencies

Category	Packages
Core	airsim, numpy, opencv-python, msgpack-rpc-python
Perception	ultralytics, open3d, scikit-learn
LLM Backends	transformers, accelerate
PX4 / MAVSDK	mavsdk, async-tkinter-loop

---

Setup

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Upgrade pip
python -m pip install --upgrade pip

# Install core dependency
python -m pip install airsim

# Install all optional dependencies
python -m pip install ultralytics open3d scikit-learn transformers accelerate mavsdk async-tkinter-loop

Note: Model weights (*.pt) are not included in this repository due to file size constraints. Download YOLOv8 weights from Ultralytics and place them in the project root.

---

Usage

1. Launch the AirSim Simulator

Start your AirSim scene. If using PX4Multirotor, start PX4 SITL with matching ports first. If using SimpleFlight, the Python layer connects directly.

2. Navigate to the project root

cd uav-swarm-airsim

3. Select a workflow

Sensor Fusion Viewer

python AirSimSensors/detect.py --vehicle Drone

Flag	Description
--no-yolo	Disable YOLO object detection overlay
--no-lidar	Disable LiDAR point cloud view
--save-on-exit cloud.ply	Export final point cloud on exit
--trajectory-file <path>	Overlay a live trajectory JSON from autopilot or patrol

---

Single-Drone Manual Control

python manualcontrol.py --vehicle Drone --fallback-simpleflight

Keyboard bindings: W/A/S/D (horizontal), R/F (altitude), Z/C (yaw), Q (land + quit).

---

Autonomous Waypoint Pilot (LLM-backed)

python AirSimSensors/autopilot.py

When prompted, enter a mission such as:

0,0,-8;30,0,-8;30,20,-8
fly to 500,0,-10

Supported backends: Ollama, Hugging Face Transformers, NVIDIA API, or the built-in deterministic safety pilot (automatic fallback).

---

Monitoring Patrol

python SwarmRun/monitoring_patrol.py \
  --pattern sweep \
  --sweep-origin 0,0 \
  --sweep-size 40,20 \
  --sweep-lane-spacing 8 \
  --sweep-altitude -8 \
  --watch-labels person,car \
  --rth-on-complete

Live operator controls during patrol:

Key	Action
p	Pause / Resume
r	Return to Home
c	Manual Capture
l	Land
q	Land and Exit

---

Swarm Workflows

# Step 1 — Spawn the swarm
python SwarmRun/multidrone.py

# Step 2a — Run the agentic role-based controller
python SwarmRun/agentic_swarm.py --waypoints "0,0,-4;25,0,-4;25,15,-4"

# Step 2b — Or control the swarm manually
python SwarmRun/MultidroneManual.py

# Cleanup spawned drones
python SwarmRun/delete_drones.py

---

ADS-B Server for QGroundControl

python adsb_sbs_server.py --host 127.0.0.1 --port 30003

Point QGroundControl's ADS-B / SBS input to the same host and port.

---

Generated Artifacts

Patrol missions write structured run data under runs/monitoring_patrol/<timestamp>/:

Artifact	Description
trajectory.json	Latest trajectory frame for real-time overlays
trajectory_frames.jsonl	Per-cycle replay frames with state, decision, enforced command, and planner context
events.jsonl	Mission-level events: alerts, waypoint completions, operator actions
captures/	RGB snapshots with associated metadata
patrol.mp4	Optional video recording (requires --record-video)

Export Training Data

python tools/export_training_dataset.py --runs-root runs/monitoring_patrol

Outputs training_examples.jsonl — structured for fine-tuning or behavioral cloning workflows.

---

Troubleshooting

Error	Fix
ModuleNotFoundError: airsim	Activate the virtual environment and run pip install airsim
Vehicle not responding / PX4 timeout	Start PX4 SITL with matching ports, or use a SimpleFlight vehicle with --fallback-simpleflight
No Open3D point-cloud window	Run pip install open3d; OpenCV-only views will still work without it
No YOLO overlay	Run pip install ultralytics, or pass --no-yolo to skip detection
No DBSCAN LiDAR clustering	Run pip install scikit-learn; falls back to grid clustering automatically

---

Research Context

This platform was developed as part of graduate AI research, focusing on autonomous multi-agent systems, real-time aerial perception, and LLM-integrated mission planning for UAV applications.

---

License

MIT License — see LICENSE for details.