A multi-modal remote sensing stack for dark vessel detection. Sentinel-1 SAR + Sentinel-2 optical + AIS, fused through a geospatial foundation model backbone, with physics-informed anomaly reasoning from the TGARD / Pi-DPM thesis line. xView3-SAR benchmark, Microsoft Planetary Computer ingest, 7 canonical Earth-observation tasks under one repo.
Maritime domain awareness fails when ships go "dark," that is, when an AIS transponder is silenced, spoofed, or jammed. Detecting these vessels requires fusing what the sea is broadcasting (AIS) with what the sky is observing (Sentinel-1 SAR, Sentinel-2 optical, Landsat-9 thermal). DarkVesselNet is a single repository that ingests these heterogeneous sensor streams, embeds them through a shared geospatial foundation model backbone (Prithvi-2 / Clay / SatMAE++), and runs the seven canonical remote sensing tasks needed end-to-end: detection, segmentation, classification, change detection, super-resolution, time-series forecasting, and anomaly reasoning. The anomaly head is the published TGARD / Pi-DPM line from the author's dissertation; everything upstream is the modern Earth-observation stack a hiring manager expects a senior CV / remote sensing scientist to wield.
The result: a paste-into-a-pipeline reference implementation of "what you do when a vessel disappears off AIS for 47 minutes in the Gulf of Oman."
Most remote sensing demos are narrow. They show classification on EuroSAT, or segmentation on SpaceNet, or change detection on LEVIR-CD, but never a single project that touches every sensor type, every canonical task, and a real downstream decision (here, "is this a dark vessel"). DarkVesselNet is the single-project answer to remote sensing interview screens. It is also the bridge between the author's dissertation on distortion-aware spatial data science and the production geospatial ML stack used at Microsoft AI for Good (Features of the World), Planet, Maxar, Allen AI (xView3), Global Fishing Watch, and Matter Intelligence.
- Seven canonical EO tasks under one config tree. Detection (DETR-on-SAR), instance segmentation (SAM 2 + SatlasNet), classification (Prithvi-2 head), change detection (siamese Clay encoder), super-resolution (PhysFlow-Earth pipeline integration), time-series forecasting (Prithvi-WxC / TimeSformer), and anomaly reasoning (TGARD + Pi-DPM).
- Four sensor modalities. Sentinel-1 SAR (VV / VH, GRD + SLC), Sentinel-2 optical (L1C + L2A multispectral), Landsat-9 (thermal), PlanetScope (sub-3 m optical via tasking). All ingested through
stackstacagainst Microsoft Planetary Computer's STAC catalog. - Three open geospatial foundation models, swappable. Prithvi-2 (NASA / IBM, 600M params, six bands), Clay v1 (Made with Clay, ViT-L), SatMAE++ (multi-spectral MAE, Cong et al. NeurIPS 2023). All loaded through Hugging Face; swap via Hydra config.
- Distortion-aware projection. Inherits the closed-form RPC / pushbroom Jacobians from
sat-splat-distortso SAR-optical fusion respects the actual sensor geometry rather than naive UTM warping. This is the dissertation contribution made operational. - AIS fusion with anomaly heads. Reuses the trajprompt traj-CLIP encoder; the TGARD module (Sharma et al., ACM TIST 2024) flags trajectory gaps; the Pi-DPM diffusion module (Sharma et al., SIGSPATIAL GeoAnomalies 2025) reconstructs the missing segment and scores spoofing likelihood.
- xView3-SAR benchmark. Targets the Allen AI / DIU dark vessel detection leaderboard (Paolo et al., NeurIPS Datasets and Benchmarks 2022), including the close-to-shore subset, vessel-vs-non-vessel head, and the length regression task.
- Reproducible, testable, deployable. Hydra configs, pytest math + smoke tests, Gradio Space, MSI Slurm submit scripts, Diffusers / Transformers-compatible checkpoints, ONNX export.
| # | Task | Backbone | Dataset | Tie to thesis |
|---|---|---|---|---|
| 1 | Object detection | DETR head on Prithvi-2 features | xView3-SAR | none (pure RS) |
| 2 | Instance segmentation | SAM 2 + SatlasNet adapter | SpaceNet 6, xView3 | none (pure RS) |
| 3 | Classification | Prithvi-2 + linear probe | fMoW-Sentinel, BigEarthNet, EuroSAT | none (pure RS) |
| 4 | Change detection | siamese Clay v1 encoder | LEVIR-CD, OSCD, xBD | port activity priors feed AGM |
| 5 | Super-resolution | PhysFlow-Earth pipeline | WorldStrat, SEN2VENuS | physics constraints from Pi-DPM |
| 6 | Time-series forecasting | TimeSformer + Prithvi-WxC | SEN12MS-CR-TS, custom AIS-aligned | feeds TGARD vessel-track continuation |
| 7 | Anomaly reasoning | TGARD + Pi-DPM | MarineCadastre, xView3 dark labels | direct thesis: STAGD, DRM, AGM, Pi-DPM |
[Microsoft Planetary Computer STAC]
|
v
[stackstac] --Sentinel-1 VV/VH--+ +--[ AIS feed: MarineCadastre / DMA / xView3 AIS ]
| |
v v
[GeoFoundation backbone]
Prithvi-2 / Clay / SatMAE++
|
+------------------+------------------+----------+----------+
v v v v v
[Detection] [Segmentation] [Classify] [SR] [Change det]
DETR SAM 2/Satlas linear PhysFlow siamese
| | | | |
+--------+---------+---------+--------+----------+----------+
v v
[Vessel candidate set] [Wake / port-state features]
|
v
[Traj-CLIP align with AIS] <----- [trajprompt encoder]
|
v
[TGARD gap detector] <----- thesis: rendezvous + STAGD + DRM
|
v
[Pi-DPM trajectory reconstruction] <----- thesis: physics-informed diffusion
|
v
[Dark vessel probability map + reasoning trace]
git clone https://github.com/arunshar/darkvessel-stack
cd darkvessel-stack
uv venv --python 3.11 .venv && source .venv/bin/activate
uv pip install -e ".[dev,space]"
bash scripts/download_xview3_sample.sh # 8 GB sample, full set is 130 GB
python -m darkvessel.training.train +experiment=xview3_prithvipytest # math + smoke
python /tmp/launch_smoke.py "$(pwd)" space/app.py # boots Gradio on a real portVerified status (CPU smoke, target):
- math tests: SAR speckle filter consistency, Sentinel-2 atmospheric correction (Sen2Cor parity), Prithvi-2 patch embed shape, traj-CLIP L2 norm, TGARD haversine distance, Pi-DPM kinematic residual.
- Space smoke tests: STAC tile fetch (mocked), foundation backbone forward, multi-head dispatch, AIS join, UI build, HF README frontmatter,
space/requirements.txtparseable. - Gradio Space launches on a local port and serves HTTP 200 with valid Gradio HTML.
HF Space. Click an AOI on the Mapbox dark-theme globe (Gulf of Oman, Strait of Hormuz, South China Sea, Galapagos EEZ, Sea of Japan); the public demo runs a CPU-safe scaffold of the seven-head pipeline and renders a reasoning trace per detection.
| Modality | Sentinel mission | Spatial | Temporal | Spectral | Strength | Weakness |
|---|---|---|---|---|---|---|
| SAR | Sentinel-1 (C-band) | 5 m x 20 m (IW GRD) | 6 d (one sat) / 12 d (constellation) | VV / VH polarizations | All-weather, day-night, sees through cloud | Speckle, layover, foreshortening |
| Optical multispectral | Sentinel-2 | 10 m (RGB+NIR) / 20 m (red edge, SWIR) / 60 m (aerosol, cirrus) | 5 d at equator | 13 bands, 443-2190 nm | Vegetation indices, true color, easy interp | Clouds, shadows, sun angle |
| Thermal | Landsat-9 TIRS | 100 m | 16 d | 10.8 + 12.0 um | Heat signatures (engines, fires) | Low spatial res |
| High-res optical | PlanetScope / SkySat | 3 m / 0.5 m | Daily | RGB-NIR | Visible vessel structure | Tasking cost, cloud-limited |
DarkVesselNet uses all four. SAR is the workhorse for vessel detection in the open ocean because it is the only modality that sees at night and through cloud. Optical confirms. Thermal disambiguates wakes and engine heat. PlanetScope, where licensed, classifies vessel type.
- Calibration: GRD products are radiometrically calibrated to sigma0 via the official calibration LUT; we apply Lee speckle filter (5x5) before backbone embedding.
- Polarization: VV is the primary channel for vessel returns on water (water is a near-specular reflector at VV, vessels are corner reflectors); VH adds discrimination of bright clutter (rigs, buoys).
- Geocoding: terrain correction via SNAP / pyroSAR against Copernicus DEM-30 before fusion with optical.
- Layover handling: vessels near coastline within layover masks are routed to a separate close-to-shore head, matching the xView3 evaluation protocol.
- Atmospheric correction: Sen2Cor L2A products preferred; if only L1C is available, we run a learned correction head distilled from Sen2Cor outputs (Cloud-Net + SatlasNet preprocessing).
- Cloud masking: s2cloudless v1.7, threshold 0.4, with morphological dilation by 3 px.
- Sun-glint suppression: linear regression on NIR / SWIR per scene, residualized for vessel scoring (specular glint mimics vessel signal otherwise).
- Co-registration with SAR: we use the RPC + pushbroom Jacobians from sat-splat-distort to warp Sentinel-1 GRD pixels onto the Sentinel-2 grid without resampling-induced shift, then fuse at the backbone token level.
| Backbone | Source | Params | Pretraining data | When we use it |
|---|---|---|---|---|
| Prithvi-2 | NASA + IBM, Apache 2.0 | 600M | 4.2 TB HLS L30/S30 | default detection + classification head |
| Clay v1 | Made with Clay, MIT | 300M | 70 TB multi-sensor | change detection (siamese) |
| SatMAE++ | Cong et al., NeurIPS 2023 | 305M | fMoW-Sentinel | scarce-label fine-tunes |
| DOFA | Xiong et al., CVPR 2024 | 350M | multi-sensor unified | cross-sensor transfer |
| SatlasNet | Allen AI | 90M Swin-B | Satlas | instance segmentation head |
| RemoteCLIP | Liu et al., CVPR 2024 | 304M | RS-image text pairs | open-vocabulary retrieval |
All are loaded via darkvessel.backbones.GeoBackbone.from_pretrained(...); swap with one Hydra flag.
darkvessel-stack/
├── src/darkvessel/
│ ├── backbones/{prithvi.py, clay.py, satmae.py, dofa.py, satlas.py, remoteclip.py, geo_backbone.py}
│ ├── heads/{detection.py, segmentation.py, classification.py, change.py, sr.py, forecast.py, anomaly.py}
│ ├── sar/{calibration.py, speckle.py, terrain_correction.py, polarimetry.py}
│ ├── optical/{atm_correction.py, cloud_mask.py, sun_glint.py, band_ratios.py}
│ ├── fusion/{coregistration.py, token_fusion.py, late_fusion.py}
│ ├── ais/{trajclip.py, tgard.py, pidpm.py} # vendored from trajprompt + Pi-DPM repo
│ ├── data/{xview3.py, sen12ms.py, spacenet.py, fmow.py, ais_marinecadastre.py, planetary_computer.py}
│ ├── eval/{xview3_bench.py, oscd_bench.py, fmow_bench.py, worldstrat_bench.py}
│ ├── training/train.py # Hydra-configured
│ └── viz/{folium_map.py, attention_rollout.py, reasoning_trace.py}
├── space/app.py # Gradio HF Space
├── configs/ # Hydra
│ ├── experiment/xview3_prithvi.yaml
│ ├── experiment/oscd_clay_change.yaml
│ ├── experiment/worldstrat_physflow_sr.yaml
│ └── ...
├── tests/ # math + smoke
├── paper/{main.tex, supplement.tex} # CVPR EarthVision 2027
├── docs/{architecture.md, sensor_primer.md, foundation_models.md, anomaly_pipeline.md}
└── scripts/{download_xview3_sample.sh, submit_msi.slurm, hf_push.sh}
python -m darkvessel.eval.xview3_bench \
--backbone prithvi-2 \
--checkpoint hf://Arun0808/darkvessel-stack-xview3 \
--metric aggregate_score detection_fscore length_rmse close_to_shore_fscoreTarget leaderboard (baselines from Paolo et al., NeurIPS D&B 2022, and the public xView3 leaderboard at challenge close):
| Method | Aggregate score | Detection F1 | Length RMSE (m) | Close-to-shore F1 |
|---|---|---|---|---|
| xView3 baseline (FRCNN) | 27.7 | 0.39 | 32.7 | 0.27 |
| 1st place (BloodAxe et al.) | 50.6 | 0.62 | 19.5 | 0.42 |
| SatlasNet-XL (Allen AI) | 47.8 | 0.59 | 21.1 | 0.40 |
| DarkVesselNet (Prithvi-2 + Pi-DPM) | 52.4 | 0.64 | 18.2 | 0.45 |
Numbers above are targets; the repo ships with the baseline + Prithvi-2 frozen-features setup that reproduces the literature baselines on the publicly released xView3 validation split, and a training recipe for the full leaderboard run on MSI / Polaris (estimated 180 A100h).
| Thesis contribution | Where it lives in this repo | Original venue |
|---|---|---|
| TGARD (trajectory rendezvous + anomaly gap detection) | src/darkvessel/ais/tgard.py |
ACM TIST 2024 |
| Pi-DPM (physics-informed diffusion for trajectory anomalies) | src/darkvessel/ais/pidpm.py and src/darkvessel/heads/anomaly.py |
ACM SIGSPATIAL GeoAnomalies 2025 |
| STAGD + DRM (denial-based distortion) | src/darkvessel/ais/dark_distortion.py |
ACM TIST 2024 |
| Distortion-aware projection (RPC / pushbroom Jacobians) | src/darkvessel/fusion/coregistration.py |
reused from sat-splat-distort, CVPR EarthVision 2027 |
| Kriging-informed conditional diffusion for downscaling | src/darkvessel/heads/sr.py (via physflow-earth) |
SIGSPATIAL 2024 |
@inproceedings{sharma2027darkvesselnet,
title = {DarkVesselNet: A Multi-Modal Remote Sensing Stack for Dark Vessel Detection},
author = {Sharma, Arun},
booktitle = {CVPR EarthVision Workshop},
year = {2027}
}Apache 2.0. Prithvi-2 retains its Apache 2.0 license; Clay v1 is MIT; SatMAE++ checkpoints are CC-BY-NC and not redistributed (download script fetches from the authors). xView3 data is under the Allen AI Impact License.