Official code repository for the paper “From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models”.
ScreenAnnotator is an interactive annotation platform for building and improving vision-language model (VLM) datasets. It supports task-based image management, bounding-box annotation, model-assisted annotation, quality-control review, and on-policy model training loops.
Existing annotation platforms often suffer from three systematic bottlenecks: limited expressiveness for jointly binding spatial locations, free-form text, and attributes; annotation-training decoupling in offline pipelines; and poor data reusability across downstream VLM tasks. ScreenAnnotator is designed to address these limitations with a unified annotation schema and an interactive model-in-the-loop workflow.
ScreenAnnotator extends each bounding box into a unified annotation atom: a categorical label, a free-form description, and structured key-value attributes defined by a task-specific declarative schema. This representation supports richer supervision than bounding boxes alone while remaining reusable across downstream reasoning tasks.
ScreenAnnotator supports an on-policy annotation loop in which the current policy model pre-annotates new image batches, human annotators review and correct predictions, a Bayesian verifier flags suspicious annotations for a second review pass, and updated models are retrained on the accumulated quality-controlled dataset before the next round.
- backend/README.md: Django backend, database/object-storage configuration, API routes, annotation data model, and model orchestration workflow.
- frontend/README.md: Vue/Vite frontend, annotation UI, model workflow prerequisites, and frontend development notes.
- backend/models/README.md: FastAPI model runtime services, model-backend communication protocols, and runtime development notes.
Prepare PostgreSQL and MinIO first, then start the required services in separate terminals. Detailed setup instructions are available in the component-level README files above.
# Terminal 1: backend API
cd backend
python manage.py migrate
python manage.py runserver 0.0.0.0:8000# Terminal 2: frontend UI
cd frontend
npm install
npm run dev# Terminal 3: optional model runtime, for example YOLO
cd backend/models/yolo
python server.pyOpen the frontend at http://localhost:3010. Backend API docs are available at http://localhost:8000/docs/.
.
├── backend/ # Django API, persistence, annotation, and model orchestration
├── frontend/ # Vue/Vite annotation interface
├── assets/ # Figures used in the paper and repository documentation
└── backend/models/ # Standalone FastAPI model runtime services
The following screenshots show the current system interface:
A demonstration video of the current system is available here:
The paper is currently under double-blind review. The final BibTeX entry will be added after the review process.
This repository is released under the BSD 3-Clause License. For double-blind review, the copyright holder is temporarily listed as “Anonymous Authors” in LICENSE.