Skip to content

Latest commit

 

History

History
54 lines (47 loc) · 2.2 KB

Evaluation.md

File metadata and controls

54 lines (47 loc) · 2.2 KB
Raw

Evaluation

We evaluate GeoChat on a variety of tasks, including scene classification, region captioning, visual grounding, grounding description and VQA. Converted files in the input format for GeoChat are available at GeoChat-Bench

Below we provide a general guideline for evaluating datasets.

  1. LRBEN/HRBEN. Images and ground truth for evaluation need to be downloaded from the following sources: LRBEN, HRBEN Give the path to the extracted image folder in the evaluation script. We add the following text after each question during our evaluation.
<question>
Answer the question using a single word or phrase.

python geochat/eval/batch_geochat_vqa.py \
    --model-path /path/to/model \
    --question-file path/to/jsonl/file \
    --answer-file path/to/output/jsonl/file \
    --image_folder path/to/image/folder/
  1. Scene Classification. Download the images from the following sources, UCmerced, AID. We add the following text after each question during our evaluation.
<question>
Classify the image from the following classes. Answer in one word or a short phrase.

python geochat/eval/batch_geochat_scene.py \
    --model-path /path/to/model \
    --question-file path/to/jsonl/file \
    --answer-file path/to/output/jsonl/file \
    --image_folder path/to/image/folder/
  1. Region-Captioning/Visual grounding.

The evaluation images are present in the image.zip folder in GeoChat_Instruct.

python geochat/eval/batch_geochat_grounding.py \
    --model-path /path/to/model \
    --question-file path/to/jsonl/file \
    --answer-file path/to/output/jsonl/file \
    --image_folder path/to/image/folder/
python geochat/eval/batch_geochat_referring.py \
    --model-path /path/to/model \
    --question-file path/to/jsonl/file \
    --answer-file path/to/output/jsonl/file \
    --image_folder path/to/image/folder/