Camera Trap Group Size Estimation with VLMs

Estimating the number of distinct individual animals in a camera trap event using vision-language models (Gemini Flash).

Problem

Camera traps take bursts of images (roughly one per second) when triggered by motion. A group of animals walking past produces a sequence of images, and the same individuals appear across multiple frames. Ecologists need to know the group size -- the total number of distinct animals in the event -- but current AI tools process one image at a time and can only count animals per frame, not across a sequence.

Counting the maximum number of animals in any single frame is not adequate. For example, if five images each show at most two animals, the true group size could be anywhere from two (standing still) to ten (moving quickly through frame). To human eyes the answer is usually clear based on position, heading, and coarse features like size or antler presence.

Approach

We use a two-stage pipeline:

1. Detection + classification: MegaDetector detects animals and SpeciesNet classifies species. Bounding boxes are rendered on images.
2. Group size estimation: A sequence of bounding-box images is sent to Gemini Flash 3.0 with a prompt asking it to track individuals across frames and estimate the total group count.

Images are resized to 768px on the long side and sent in chronological order with timestamp labels. Images with no confident animal detections (conf < 0.15) are filtered out; the prompt notes how many images were omitted.

For full experimental details, see PLAN.md.

Dataset

The dataset contains ~40,000 camera trap images of ground-dwelling birds from Guatemala, across two field seasons (2018, 2019). Species include Great Curassow, Ocellated Turkey, Great Tinamou, and others. Human-annotated group counts are available for ~3,400 events (30-minute grouping windows).

Example Sequences

Correct: 2 Ocellated Turkeys (predicted 2)

Frame 1	Frame 2	Frame 3	Frame 4	Frame 5

Gemini's reasoning: "The first two images seem to show the same individual moving to the left. The last three images show two individuals, and based on size and position relative to each other and to the camera's field of view, these are likely the same two individuals throughout."

Correct: 3 Ocellated Turkeys (predicted 3)

Frame 1	Frame 2	Frame 3	Frame 4	Frame 5

Gemini's reasoning: "The animals on the right in the first three images appear to be the same individual. The animals in the last two images are two distinct individuals. It is also possible that the other animals in the first three images are the same individual, but not the one in the last two images."

Results

Gemini Flash 3.0 (100 sampled events)

Metric	Value
Exact match	42.0%
Within +/-1	73.0%
Within +/-2	88.0%
MAE	1.15
Mean error (bias)	+0.85

Breakdown by true count

True Count	N	Exact Match	MAE	Predictions
1	40	62%	0.68	mostly 1, some 2-3, occasional 5
2	25	52%	0.60	mostly 2-3, one outlier at 5
3	15	27%	1.27	scattered 2-7
4	8	0%	2.12	5-9, consistent over-count
5	2	0%	2.50	2 and 7
6	4	0%	1.75	3-7
7+	6	0%	4.33	highly variable

Key observations:

• Strong over-counting bias (+0.85 mean error), with 48 over-counts vs 10 under-counts
• Good performance for counts 1-2 (62% and 52% exact match), but accuracy degrades sharply for higher counts
• Zero exact matches for true counts >= 4
• The model struggles most with large groups (count=9 events had MAE of 5.75)

Gemini Flash 2.0 (76 of 100 sampled events, partial)

Metric	Value
Exact match	59.5%
Within +/-1	82.4%
Within +/-2	97.3%
MAE	0.62
Mean error (bias)	+0.59

Note: the Flash 2.0 run was stopped at 76/100 events before the higher-count events were processed, so its metrics are not directly comparable to Flash 3.0's full 100-event run. At the same stage (low counts only), Flash 3.0 performed comparably.

TODO

• Evaluate Gemini 3.1 Pro to see whether a more capable model improves accuracy, especially on higher-count events
• Add few-shot examples to the prompt, selected from difficult cases in prior runs, to help calibrate the model (e.g., reduce over-counting on single-animal events)
• Experiment with alternative strategies:
  • Strategy 1: raw images without bounding boxes
  • Strategy 3: structured position/crop data instead of full images (much cheaper in tokens)
• Scale evaluation to the full ~3,400 event dataset
• Analyze failure modes in more detail (e.g., does over-counting correlate with number of images, species, or detection confidence?)

Scripts

• 01_build_event_table.py -- Parse ground-truth CSVs, match images to events, filter by MegaDetector detections
• 02_run_experiment.py -- Sample events, send to Gemini, cache results (currently configured for Flash 3.0)
• 03_analyze_results.py -- Compute metrics and generate HTML report with visual examples

Requirements

pip install -r requirements.txt