NOVA is a tool for annotating and analyzing behaviours in social interactions. It supports Annotators using Machine Learning already during the coding process. Further it features both, discrete labels and continuous scores and a visuzalization of streams recorded with the SSI Framework.
C# Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
App auto-load waveform for videos with sound Apr 20, 2018
Controls added fishers z-score to stats, fix for nans in PCC Jul 30, 2018
Database calculates p values for PCCs, added code to compare 2 PCCs in terms o… Jul 31, 2018
Interfaces Parallel feature extraction, OSCPlugin, CML updates Jul 10, 2017
Packages added costura dependencies May 4, 2018
Properties updated to .net 4.6.1 Feb 13, 2018
References support for FLAC, fixes related to feature extraction and training Jan 30, 2018
Resources adjusted bayesnet window to new continuous format Dec 11, 2017
Tools updated fody/costura for compatibility with VS2017 Feb 13, 2018
Types fix for utf8 encoded annotations in the database Apr 20, 2018
bin calculates p values for PCCs, added code to compare 2 PCCs in terms o… Jul 31, 2018
docs added description on how to run MongoDB as a service to the documenta… Apr 26, 2018
packages updated fody/costura for compatibility with VS2017 Feb 13, 2018
plugins updated to .net 4.6.1 Feb 13, 2018
updater updated to .net 4.6.1 Feb 13, 2018
.gitattributes line endings Mar 29, 2017
.gitignore Session completion now works for continuous annotations Dec 7, 2017
EVRPresenter64.dll added dlls for hardware video acceleration Jun 26, 2017
FodyWeavers.xml readded fody dependencies Mar 28, 2017
LICENSE Initial commit Jul 26, 2016
README.md updated readme Feb 8, 2018
app.config updated to .net 4.6.1 Feb 13, 2018
nova.csproj database sessions now have a duration attribute. fix for dbn learning Apr 13, 2018
nova.sln Parallel feature extraction, OSCPlugin, CML updates Jul 10, 2017
packages.config updated fody/costura for compatibility with VS2017 Feb 13, 2018

README.md

(Non)verbal Annotator

NOVA is a tool for annotating and analyzing behaviours in social interactions. It allows to visualize data recorded with the SSI Framework, as well as from external sources. A main feature of NOVA is that it allows to employ a collaborative annotation database where annotation work can be split between multiple sides, but also between a human annotator and a machine by supporting human annotators with machine learning techniques already during the annotation process - A process we call Collaborative Machine Learning.

alt tag

NOVA allows framewise labeling for a precise coding experience, and value-continuous annotations for labeling e.g emotions or social attitudes. The interface is customizable and allows loading and labeling data of multiple persons.

The Annotation format can easily be imported in other tools, like ELAN or Excel. NOVA further supports the Import of Discrete Annotation files from ELAN and ANVIL for a seamless workflow.

The Cooperative Machine Learning capabilities allow to train and evaluate machine learning models, such as Support Vector machines or Artificial neural networks directly from the interface with both, a session completion step, where a model is trained on the first minutes of an annotated sessions to predict the remaining session, and a session transfer step where a model is trained on multiple sessions to predict completly unknown data. With the help of human input the models can be refined. The collaborative ML workflow is illustrated below.

alt tag

The latest binaries can always be found here.

FAQ:

Help, I opened a video but it doesn't show up

Please make sure you installed the according Video Codec on your System. E.g. The K-Lite Codec Pack might be a good solution for most missing codecs. Also make sure you installed a Visual Studio 2015 redistribution package if it doesn't come with your system.

I don't know what to do, where can I get help?

The offical documentation can be found here. If you run into trouble, please create an issue on git.

Will NOVA run on my Mac/Linux Machine?

As NOVA is an WPF Application it will run on Windows. Nevertheless you can of course use a Virtual Machine to use it.

I found a bug, can you fix it?

NOVA is Software under development and is provided “as is”. If you run into any problems or find bugs (or want to contribute to the project) feel free to open an issue here on github.

Is there an example pipeline to automatically create annotations?

check out https://github.com/hcmlab/kinect2-record for a kinect 2 example recording pipeline.

I would like to contribute to the project

Please feel free to fork or create an issue

Publications:

If you are using NOVA in your research please consider giving us a citation:

“Applying Cooperative Machine Learning to Speed Up the Annotation of Social Signals in Large Multi-modal Corpora”, Johannes Wagner, Tobias Baur, Yue Zhang, Michel F. Valstar, Björn Schuller, Elisabeth André, 2018, https://arxiv.org/abs/1802.02565



@article{
  title={Applying Cooperative Machine Learning to Speed Up the Annotation of Social Signals in Large Multi-modal Corpora},
  author={Wagner, Johannes and Baur, Tobias and Zhang, Yue, and Valstar, Michel F. and Schuller, Bj{\"o}rn and Andr{\'e}, Elisabeth},
  journal={arXiv:1802.02565},
  url={https://arxiv.org/abs/1802.02565},
  year={2018}  
}
  

“Context-Aware Automated Analysis and Annotation of Social Human-Agent Interactions”, Tobias Baur, Gregor Mehlmann, Ionut Damian, Florian Lingenfelser, Johannes Wagner, Birgit Lugrin, Elisabeth André, Patrick Gebhard, in ACM Transactions on Interactive Intelligent Systems (TiiS) 5.2, 2015



@article{
  title={Context-Aware Automated Analysis and Annotation of Social Human-Agent Interactions},
  author={Baur, Tobias and Mehlmann, Gregor and Damian, Ionut and Lingenfelser, Florian and Wagner, Johannes and Lugrin, Birgit and Andr{\'e}, Elisabeth and Gebhard, Patrick},
  journal={ACM Transactions on Interactive Intelligent Systems (TiiS)},
  volume={5},
  number={2},
  pages={11},
  year={2015},
  publisher={ACM}
}

“NovA: Automated Analysis of Nonverbal Signals in Social Interactions” Tobias Baur, Ionut Damian, Florian Lingenfelser, Johannes Wagner and Elisabeth André, in Human Behavior Understanding, LNCS 8212, 2013.


@incollection{
year={2013},
isbn={978-3-319-02713-5},
booktitle={Human Behavior Understanding},
volume={8212},
series={Lecture Notes in Computer Science},
editor={Salah, Albert Ali and Hung, Hayley and Aran, Oya and Gunes, Hatice},
doi={10.1007/978-3-319-02714-2_14},
title={NovA: Automated Analysis of Nonverbal Signals in Social Interactions},
url={http://dx.doi.org/10.1007/978-3-319-02714-2_14},
publisher={Springer International Publishing},
author={Baur, Tobias and Damian, Ionut and Lingenfelser, Florian and Wagner, Johannes and André, Elisabeth},
pages={160-171}}