Scoring software for the TRECVID Activities in Extended Video (ActEV) evaluation
Clone or download
Latest commit 6fc44aa Sep 24, 2018

README

ActEV Scoring Software
Version: 0.3.4
Date: September 24, 2018

This software package contains a scoring script for the TRECVID
Activities in Extended Video (ActEV) task.  The script
"ActEV_Scorer.py", is a Python 2.7 script that will validate and score
a system output file adhering to the JSON format defined in the ActEV
evaluation plan.  A collection of unit and integration test cases have
also been included, see the 'Setup' section for more detail.

ActEV_Scorer.py, when run with the '-h' option, will show the script's
usage text.  The "example_run.sh" contains an example of typical usage
(using the provided test data)

Setup:

  This package assumes a Unix-like environment.  Included Python files
  are written for Python 2.7.

  1) Install Python 2.7, if not already installed
  2) Install the following Python packages (version number used for
     development):
     - munkres (1.0.12)
     - scipy (1.0.0)
     - matplotlib (2.0.2)
     - jsonschema (2.5.1)
  3) Run the tests (optional, but strongly recommended)
     % make check

Option description:

  SCORING_PROTOCOL - Positional argument, from a fixed set of values
  (e.g. ActEV18_AD).  This required argument controls what system
  output formats are valid, and what metrics are computed.  A
  description of each supported protocol can be found in the
  "Protocols" section of this document

  -s SYSTEM_OUTPUT_FILE - Required; path to the system output JSON
  file to be scored

  -r REFERENCE_FILE - Required unless running validation only; path to
  the reference JSON file to score against

  -a ACTIVITY_INDEX - Required; path to activity index JSON file.
  This file lists what activities the system output will be evaluated
  on

  -f FILE_INDEX - Required; path to file index JSON file.  This file
  lists the files, and temporal ranges within those files, that the
  system output will be evaluated on ( **NOTE** currently the temporal
  ranges specified in FILE_INDEX are not used when considering what
  portion of the system output and reference to evaluate, and are only
  used to compute the duration of files.  This will be implemented in
  a future release)

  -o OUTPUT_DIR - Required unless running validation only; directory
  for computed scores and figures

  -d - Optional; by default, the script will produce a Detection Error
  Tradeoff (DET) curve figure for each activity and a combined figure
  of all activity curves.  If the '-d' option is set, no DET curves
  will be produced by the script

  -p SCORING_PARAMETERS_FILE - Optional; path to a scoring parameters
  JSON file.  If provided, overwrites the default parameters used for
  scoring

  -v - Optional; if enabled, the script will be more verbose
  (i.e. provide some scoring progress information)

  -V - Optional; if enabled, the SYSTEM_OUTPUT_FILE will be validated
  but not scored.  REFERENCE_FILE and OUTPUT_DIR parameters are not
  required if this option is enabled

  -F - Optional; if enabled, ignores extraneous "filesProcessed" and
  ignores system and reference instance localizations for extraneous
  files.  Note that extraneous files in this sense are those not
  included in the FILE_INDEX

Protocols:

  ActEV18_AD - Scoring protocol for the ActEV 2018 Activity Detection
  task, the following measures are computed:
    - PMiss at RFA for RFA values of 1, 0.2, 0.15, 0.1, 0.03, and 0.01
    - NMIDE (**NOTE** currently using a no-score collar size of 0
      frames, this will likely change in a future release)
    - NMIDE at RFA for RFA values of 1, 0.2, 0.15, 0.1, 0.03, and 0.01

  ActEV18_AOD - Scoring protocol for the ActEV 2018 Activity and
  Object Detection task.  This protocol computes both the PMiss at RFA
  and NMIDE measures reported for the ActEV18_AD protocol, but over an
  activity instance alignment that also considers object detections.
  The following additional measure are computed:
    - minMODE - The minimum NMODE score for an object detection
      alignment; reported for each aligned activity instance pair
    - PMiss at RFA for RFA values of 0.5, 0.2, 0.1, 0.033 for object
      detections (these measures are prefixed with "object-" to
      differentiate them from PMiss at RFA measures on activity
      detections)

  ActEV18_AODT - Scoring protocol for the ActEV 2018 Activity and
  Object Detection and Tracking task. This protocol computes both th
  PMiss at RFA, NMIDE, and minMODE measures reported for the ActEV18_AD
  and ActEV18_AOD protocols, but over an activity instance alignment
  that also considers object  detections. The following additional
  measures are computed:
    - MOTE - the Multiple Object Tracking Error for an object
      detection and tracking alignment.
      
Output:

  The scoring script writes to several files in the specified
  OUTPUT_DIR directory (all *.csv files are pipe separated):

    - scores_by_activity.csv - Scores by activity; computed with
      respect to the selected scoring protocol
    - scores_aggregated.csv - Aggregated scores over all activities
    - scoring_parameters.json - Lists the scoring/protocol parameters
      used
    - alignment.csv - Lists each of the matched and unmatched system
      and reference instances, along with the matching kernel
      components and similarity scores
    - pair_metrics.csv - Metrics computed on the matched
      system/reference instance pairs
    - figures/DET_<activity>.png - Unless disabled with the '-d'
      option, the DET curve figure for <activity>

  Option/Protocol dependent:

    - object_alignment.csv - For the "ActEV18_AOD" and "ActEV18_AODT"
      protocols, enabled with '-j'; Lists the frame-by-frame object
      alignments for each pair of aligned activity instances

Changelog:

  Jan 31, 2018 - Version 0.0.1
    - Initial release

  Feb 22, 2018 - Version 0.0.2
    - Fixed an issue where system or reference files without instances
      for an activity listed in the activity index would cause the
      script to fail
    - Fixed an issue with older versions of the matplotlib package,
      which would cause the script fail when plotting activities with
      no false alarms

  March 16, 2018 - Version 0.1.0
    - Renamed the "ActEV18" protocol to "ActEV18_AD", as it's
      specifically for the Activity Detection task
    - Added scoring protocol for "ActEV18_AOD".  This protocol
      includes an option (-j) to dump out the frame-by-frame object
      alignments for each pair of aligned activity instances
    - Updated schema and code to expect "presenceConf" instead of
      "decisionScore" for activity detections.  The headers of some
      output files have been updated to reflect this change
    - Changed "config.csv" output to be a serialized JSON, named
      "scoring_parameters.json"
    - Added a command line option to accept a scoring parameters JSON
      file to overwrite the protocol's default parameters for the
      scoring run
    - The script now checks the file index against the
      "filesProcessed" reported in the system output file for
      congruence
    - Added N-MIDE measure to aggregate scores, which is computed over
      the entire alignment (regardless of activity) in addition to the
      already reported N-MIDE macro-averaged over activities
      (currently reported as "mean-n-mide")
    - The N-MIDE computation now ignores pairs where the reference
      instance has been reduced to a duration of zero due to the size
      of the no-score collar.  The number of ignored pairs are
      reported as "n-mide_num_rejected"

  March 27, 2018 - Version 0.1.1
    - Added object detection PMiss@RFA measures for the "ActEV18_AOD"
      protocol
    - Fixed object congruence calculation for "ActEV18_AOD".  Should
      be calculated as 1 - minMODE, rather than simply minMODE
      (**NOTE** this change affects alignment, and as a result your
      scores may have changed from the previous version)
    - For "ActEV18_AOD", updated the default object congruence delta
      to be 0 instead of 1 to reflect the updated object congruence
      calculation
    - Fixed an issue where DET curve points with PMiss of 1 or 0
      weren't being plotted.  Note that these points will not be
      directly visible within the DET curve figures due to the y-axis
      scaling, but will connect to other points within the view

  April 9, 2018 - Version 0.1.2
    - Fixed an issue where a reference activity instance spanning the
      entire duration of the source video would cause the N-MIDE
      computation to fail.  These instances are now ignored for
      N-MIDE, and are included in the "n-mide_num_rejected" count
    - Added an optimization to the kernel builder function whereby
      unnecessary filter computations are skipped

  April 23, 2018 - Version 0.2.0
    - Now using global range of "presenceConf" scores for detection
      congruence score component of alignment kernel for both
      "ActEV18_AD" and "ActEV18_AOD"
    - The "ActEV18_AOD" protocol can now accept an "objectTypeMap" for
      each activity in the provided activity index.  Reference and
      system "objectType" strings are passed through the map (if
      provided) prior to alignment
    - Added additional columns to the "object_alignment.csv" output
      file, which specifies both the original "objectType" strings and
      re-mapped strings for both reference and system instances
    - The "ActEV18_AOD" protocol will now ignore any objects provided
      by the reference or system output with an "objectType" not
      included in the list of "objectTypes" for a given activity in
      the activity index file.  If the "objectTypes" property is
      provided as an empty list, or is simply omitted, no such
      filtering takes place

  April 25, 2018 - Version 0.2.1
    - Added an option (-V, --validation-only) where the system output
      file will be validated but not scored.  With this option
      enabled, the reference file (-r), and output directory (-o)
      parameters are not required
    - Fixed an issue where the global range of "presenceConf" scores
      for object detections was being computed more often than
      necessary

  April 27, 2018 - Version 0.2.2
    - Fixed a divide by zero issue when computing MODE for the
      "ActEV_AOD" object detection congruence kernel component.
      Specifically when there are no reference objects
    - Added "temporal_fa" and "temporal_miss" to the pair metrics
      output

  May 16, 2018 - Version 0.3.0
    - Renamed existing scoring parameters to be less ambiguous.  Added
      scoring parameters to control error weights for some metrics as
      well as the target rates of false alarm
    - Refactored portions of the alignment and metric computation code
      to improve performance

  May 24, 2018 - Version 0.3.1
    - Now reporting a PMiss@RFA of 1.0 instead of None when there are
      no system reported instances
    - For the "ActEV_AOD" protocol, now reporting a mean object
      PMiss@RFA of 1.0 instead of None when there are no aligned
      activity instances

  June 13, 2018 - Version 0.3.2
    - Added an option (-F, --ignore-extraneous-files) to ignore
      "filesProcessed" and reference and system localizations for
      files not included in the provided FILE_INDEX
    - Miscellaneous improvements

  June 25, 2018 - Version 0.3.3
    - Added NMIDE at RFA measures for both "ActEV_AD" and "ActEV_AOD"
      protocols
    - Moved DET curve plot legend to the right of the plot

  September 24, 2018 - Version 0.3.4
    - Added AODT Task
    - Added MOTE to output files
    - Added integration test 8_0
    
Contact:

  Please send any issues, questions, or comments to
  actev-nist@nist.gov

Authors:

  David Joy
  Andrew Delgado