Add evaluation module, from_mot conversion script (#304)

* Add evaluation module, from_mot conversion script

* add label config + description, from_mot unit test, from_mot documentation

* fix TypedDict in py3.7

* new metadata config format, refactor usages of config, add eval documentation and cli

* fix from mot

* add testcase files

* eval testcases

* add config to from_coco

* ignore mypy flask import error

* update from_mot

* polish PR details

* rm DictAny

* unify cli arguments; revert from_mot.py

* dataset pydantic model

* fix small issues

* fix duplicate

* Add image size in config

* ImageSize to Frame, refactor metadata usages

* fix label.io.save

Co-authored-by: XiaLiPKU <ethanlee@pku.edu.cn>
Co-authored-by: Fisher Yu <i@yf.io>

doc/src/config.rst

@@ -1,21 +1,55 @@
 Configuration Format
 ----------------------
 
-The category configuration is simply a list of categories names. An example in
-yaml format:
-
-.. code-block:: yaml
-
-    - name: pedestrian
-    - name: rider
-    - name: other person
-    - name: car
-    - name: bus
-    - name: truck
-    - name: train
-    - name: trailer
-    - name: other vehicle
-    - name: motorcycle
-    - name: bicycle
-    - name: traffic sign
-    - name: traffic light
+The configuration should contain a list of category names. Additionally, the
+available attributes for the dataset as well as the image resolution (if all
+images have the same resolution, otherwise use the attribute "size" of each
+frame) may be optionally specified. For categories where objects should be
+detected, but are not tracked, set the attribute 'tracking' to false.
+
+BDD100K example configuration in toml format:
+
+.. code-block:: toml
+
+    resolution = [720, 1280]
+
+    [[attributes]]
+    name = "crowd"
+    toolType = "switch"
+    tagText = "c"
+
+    [[categories]]
+    name = "human"
+        [[categories.subcategories]]
+        name = "pedestrian"
+
+        [[categories.subcategories]]
+        name = "rider"
+
+    [[categories]]
+    name = "vehicle"
+       [[categories.subcategories]]
+        name = "car"
+
+       [[categories.subcategories]]
+        name = "truck"
+
+       [[categories.subcategories]]
+        name = "bus"
+
+        [[categories.subcategories]]
+        name = "train"
+
+    [[categories]]
+    name = "bike"
+        [[categories.subcategories]]
+        name = "motorcycle"
+
+        [[categories.subcategories]]
+        name = "bicycle"
+
+    [[categories]]
+    name = "traffic light"
+
+    [[categories]]
+    name = "traffic sign"

doc/src/eval.rst

@@ -0,0 +1,56 @@
+Evaluation
+===================
+
+We currently support evaluation of two tasks: Object detection and multi-object
+tracking.
+To evaluate your algorithms on each task, input your predictions and the
+corresponding ground truth annotations in Scalabel format.
+
+Detection
+-----------------
+The detection evaluation uses the AP metric and follows the protocol defined
+in the COCO dataset. You can start the evaluation by running:
+
+``python3 -m scalabel.eval.detect <args>``
+
+Available arguments:
+
+.. code-block:: bash
+
+    --gt, GT_PATH -g GT_PATH
+                            path to ground truth annotations.
+    --result, RESULT_PATH -r RESULT_PATH
+                            path to results to be evaluated.
+    --config CFG_PATH, -c CFG_PATH
+                            Config path. Contains metadata like available categories.
+    --out-dir OUT_DIR, -o OUT_DIR
+                            Output path for detection evaluation results.
+    --nproc NUM_PROCS, -p NUM_PROCS
+                            Number of processes for detection evaluation.
+
+
+Multi-object Tracking
+----------------------
+The MOT evaluation uses the CLEAR MOT metrics. You can start the evaluation
+by running:
+
+``python3 -m scalabel.eval.mot <args>``
+
+Available arguments:
+
+.. code-block:: bash
+
+    --gt, GT_PATH -g GT_PATH
+                            path to ground truth annotations.
+    --result, RESULT_PATH -r RESULT_PATH
+                            path to results to be evaluated.
+    --config CFG_PATH, -c CFG_PATH
+                            Config path. Contains metadata like available categories.
+    --out-dir OUT_DIR, -o OUT_DIR
+                            Output path for evaluation results.
+    --iou-thr IOU_TRESH
+                            IoU threshold for mot evaluation.
+    --ignore-iof-thr IGNORE_IOF_THRESH
+                            Ignore iof threshold for mot evaluation.
+    --nproc NUM_PROCS, -p NUM_PROCS
+                            Number of processes for mot evaluation.

doc/src/format.rst

@@ -29,7 +29,9 @@ other label types.
         - rotation
     - timestamp: int64 (epoch time ms)
     - frameIndex: int (optional, frame index in this video)
-    - size: [width, height]
+    - size:
+        - width: int
+        - height: int
     - labels [ ]:
         - id: string
         - index: int

doc/src/label.rst

@@ -1,18 +1,18 @@
 Label Conversion
 ===================
 
-Currently, we provide conversion scripts between scalabel and coco formats.
-They are in the ``label`` moduel of the  `Scalabel python package
+Currently, we provide conversion scripts between scalabel and coco formats as well as MOTChallenge to Scalabel format.
+They are in the ``label`` module of the  `Scalabel python package
 <https://github.com/scalabel/scalabel/tree/master/scalabel>`_. We recommend
 using the tools through python module invoke convention. For example,
 
 .. code-block:: bash
 
     python3 -m scalabel.label.to_coco ...
 
-To allow for more information store in the coco-format json files, we add a new
+To store more information in the coco-format json files, we add new
 property names "videos" to the coco format. It is a list like "videos" and
-"annotations", and each item is a have two properties: "id" and "name".
+"annotations", and each item has two properties: "id" and "name".
 
 from_coco
 -----------------
@@ -24,9 +24,9 @@ Available arguments:
 
 .. code-block:: bash
 
-    --label INPUT, -i INPUT
+    --input INPUT, -i INPUT
                             path to the coco-format json file
-    --out-dir OUT_DIR, -o OUT_DIR
+    --output OUTPUT, -o OUTPUT
                             output json path for ``det`` and ``ins_seg`` or
                             output jsons folder for ``box_track`` and ``seg_track``
 
@@ -36,7 +36,7 @@ to_coco
 ``to_coco`` converts scalabel json files into coco format.
 Now it support the conversions of four tasks: object detection as ``det``,
 instance segmentation as ``ins_seg``, multi-object tracking as ``box_track`` and
-multi-object trackings, or named segmentation tracking, as ``seg_track``.
+multi-object tracking, or named segmentation tracking, as ``seg_track``.
 
 Note that, for segmentation tasks, the mask conversion may not be reversible.
 ``Polygon`` format can be converted back with accuracy loss. Meanwhile ``RLE``
@@ -47,23 +47,30 @@ Available arguments:
 
 .. code-block:: bash
 
-    --label INPUT, -l INPUT
-                            path to the video/images to be processed
-    --output OUT_DIR, -o OUT_DIR
-                            output folder to save the frames
-    --height HEIGHT
-                            height of images
-    --width WIDTH
-                            width of images
+    --input INPUT, -i INPUT
+                            input json path for ``det`` and ``ins_seg`` or
+                            input jsons folder for ``box_track`` and ``seg_track``
+    --output OUTPUT, -o OUTPUT
+                            path to the output file to save the coco file
     --mode MODE, -m MODE
                             one of [det, ins_seg, box_track, seg_track]
-    --remove-ignore, -ri
-                            remove the ignored annotations from the label file
-    --ignore-as-class, -ic
-                            put the ignored annotations to the `ignored` category
-    --mask-mod, -mm,
+    --mask-mode, -mm,
                             conversion mode: rle or polygon
     --nproc
                             number of processes for mot evaluation
     --config
-                            config file for COCO categories
+                            config file for COCO categories
+
+from_mot
+-----------------
+
+``from_mot`` converts MOTChallenge annotations into Scalabel format.
+
+Available arguments:
+
+.. code-block:: bash
+
+    --input INPUT, -i INPUT
+                            path to MOTChallenge data (images + annotations).
+    --output OUTPUT, -o OUTPUT
+                            Output path for Scalabel format annotations.

mypy.ini

@@ -65,4 +65,12 @@ ignore_missing_imports = True
 
 [mypy-humps]
 
+ignore_missing_imports = True
+
+[mypy-motmetrics]
+
+ignore_missing_imports = True
+
+[mypy-pandas]
+
 ignore_missing_imports = True

scalabel/bot/server.py

@@ -8,7 +8,13 @@
 
 import numpy as np
 import requests
-from flask import Flask, Response, jsonify, make_response, request
+from flask import (  # type: ignore
+    Flask,
+    Response,
+    jsonify,
+    make_response,
+    request,
+)
 from PIL import Image
 
 from .seg_base import SegBase

scalabel/common/io.py

@@ -2,6 +2,7 @@
 
 import json
 import os
+from typing import List
 
 import toml
 import yaml
@@ -29,3 +30,10 @@ def load_config(filepath: str) -> DictStrAny:
         raise NotImplementedError(f"Config extention {ext} not supported")
     assert isinstance(config_dict, dict)
     return config_dict
+
+
+def load_file_as_list(filepath: str) -> List[str]:
+    """Get contents of a text file as list."""
+    with open(filepath, "r") as f:
+        contents = f.readlines()
+    return contents

scalabel/common/io_test.py

@@ -20,3 +20,9 @@ def test_load_toml_config() -> None:
     """Test load toml config."""
     config = io.load_config(get_test_file("config.toml"))
     assert config["hello"] == "world"
+
+
+def test_get_file_as_list() -> None:
+    """Test test_get_file_as_list."""
+    config = io.load_file_as_list(get_test_file("config.toml"))
+    assert config == ['hello = "world"']

scalabel/eval/__init__.py

@@ -0,0 +1 @@
+"""Scalabel evaluation package."""