Inferring Hidden Statuses and Actions in Video by Causal Reasoning
Python Matlab Jupyter Notebook R PHP C++ Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
ActionSequenceSmoother.cpp
LICENSE
ProtocolForTestingCode.txt
README.md
README_EXAMPLES.txt
SummerDataSummary.txt
amy_cvpr2012.db
analyze_results.R
analyze_results.go
base_unittests.py
causal_grammar.py
causal_grammar_hallway.py
causal_grammar_office.py
causal_grammar_office_bigger.py
causal_grammar_summerdata.py
cvpr2012_data.php
database_unittests.py
dealWithDBResults.py
demo.py
extractingActionResults.m
getCutPoints.m
hallway.py
hallway.xml
hitrate.py
inference-overlappingevents.ipynb
insertion_unittests.py
officeview.py
officeview1.xml
officeview2.xml
officeview3.xml
parsingSummerActionAndFluentOutput.py
plotAllResults.sh
plotPR.R
plotPR.sh
plotResults.R
plotResults.py
pr.py
runActionSequenceSmoother.py
scanFolders.py
slidingWindow.m
summerDataAnnotationByClip.txt
summerdata.py
summerdata_unittests.py
videoCutpoints.py
writeTemporalParses.m
xml_stuff.py

README.md

Causal And-Or Graph Inference Project

The associated paper and data can be found at http://amyfire.com/projects/inferringcausality/.

Required Data

The required data can be obtained from the project page.

Source Data for Learning + Inference

  • CVPR2012_fluent_result (fluent source data by Bruce)
  • CVPR2012_* (action source data (.txt) by Ping, parsed results (.mat) by Mingtian, parsed results (.py) by ...?)

Human Comparison Data

  • CVPR2012_humanTestAnnotation.txt -- this file defines how the clips were broken up for human annotation
  • amy_cvpr2012.db -- a sqlite file containing raw human annotations as they progressed through the clips

Intermediate Files

  • cvpr_db_results.csv (humans, source data, causalgrammar all merged) -- this is generated by dealWithDBResults.py

Workflow

dealWithDBResults.py

requires CVPR2012_humanTestAnnotation.txt, CVPR2012_fluent_result (fluent detections) and CVPR2012_*/* (one directory of action detections).

Typical usage:

python dealWithDBResults.py (upload|download|upanddown)

You can specify a non-default action directory with the -a parameter. (the default will not run out of the box as it was essentially a symlink to two different directories, depending). Those directories are in the "minimal" dataset--see the minimal dataset readme for details.

upload:

Runs the following methods:

  • causal_grammar.import_summerdata -- importing from CVPR2012_reverse_slidingwindow_action_detection_logspace/* by default, which was changed in the "minimal" dataset to two different choices--see the minimal dataset readme for details - (python files)
  • munge_parses_to_xml(fluent_parses, temporal_parses) --> orig_xml results
  • causal_grammar.process_events_and_fluents --> fluent_and_action_xml results
  • orig_xml results are inserted to db as 'origdata' and 'origsmrt'
  • fluent_and_action_xml results are inserted to db as causalgrammar and causalsmrt via uploadComputerResponseToDB; if source.endswith('smrt'), buildDictForFluentBetweenFramesIntoResults is called, which does some very basic fixing of local inconsistencies (versus buildDictForDumbFluentBetweenFramesIntoResults which does no fixing of local inconsistencies)
download:
  • creates unified results/cvpr_db_results/*.csv for each user (human or algorithm)
  • analyze_results.R converts the cvpr_db_results/*.csv to a single cvpr_db_results.csv
  • R --vanilla < analyze_results.R
  • plotAllResults.sh loops through each table/element in summerdata and generates timeline heatmaps for every agent (human, computer, source, ...)
upanddown:

Simply calls upload and then download.

python dealWithDBResults.py --help

usage: dealWithDBResults.py [-h]
                            [-d | -o EXAMPLES_ONLY | -x EXAMPLES_EXCLUDE | -g EXAMPLES_GREP]
                            [-s] [-i] [-a ACTIONFOLDER] [-n] [--debug]
                            [--database {mysql,sqlite}]
                            {upload,download,upanddown,list}

positional arguments:
  {upload,download,upanddown,list}

optional arguments:
  -h, --help            show this help message and exit
  -d, --dry-run         Do not actually upload data to the db or save
                        downloaded data; only valid for "upload" or
                        "download", does not make sense for "upanddown" or
                        "list"
  -o EXAMPLES_ONLY, --only EXAMPLES_ONLY
                        specific examples to run, versus all found examples
  -x EXAMPLES_EXCLUDE, --exclude EXAMPLES_EXCLUDE
                        specific examples to exclude, out of all found
                        examples
  -g EXAMPLES_GREP, --grep EXAMPLES_GREP
                        class of examples to include
  -s, --simplify        simplify the summerdata grammar to only include
                        fluents that start with the example name[s]
  -i, --ignoreoverlaps  skip the "without overlaps" code
  -a ACTIONFOLDER, --actionfolder ACTIONFOLDER
                        specify the action folder to run against
  -n, --inconsistentok  don't require consistency in parse building
  --debug               Spit out a lot more context information during
                        processing
  --database {mysql,sqlite}

Analysis

  • hitrate.py, formerly analyzeData-nearesthuman-hitrate.py -- "hitrate for CVPR 2015 submission" (Nov 2015)
  • pr.py calculates average precision and recall and F1
  • Obsolete: plotPR.R generates a set of precision/recall graphs from the output of analyzeData-nearesthuman-pr.py analyzeData-nearesthuman-pr.py has been removed, last existed in git hash 3032f8
  • Obsolete: analyze_results.go by Mingtian reads cvpr_db_results.csv; analyzeData.py1 refers to the results of the go code in findDistanceGivenLineOfGoOutput, calling it obsolete. # analyzeData.py has been removed, last existed in git hash 3032f8
  • Obsolete: plotPR.sh calls analyzeData-nearesthuman-pr.py and then plotPR.R to generate a set of precision/recall graphs # analyzeData-nearesthuman-pr.py has been removed, last existed in git hash 3032f8

evaluateCausalGrammer

  • getFluentChangesForFluent (operates on causal_grammar.process_events_and_fluents xml)

  • getFluentChangesForFluentBetweenFrames (operates on causal_grammar.process_events_and_fluents xml)

  • 108 videos -- cut up from "maybe 10" -- in SIX scenes (9406, 9404, 8145, lounge) -- 2 grammars