Multimodal Learning with NMF
A set of tools and experimental scripts used to achieve multimodal learning with nonnegative matrix factorization (NMF).
This repository contains code to reproduce the experiments from the publications:
- O. Mangin, D. Filliat, L. ten Bosch, P.Y. Oudeyer, MCA-NMF: Multimodal concept acquisition with non-negative matrix factorization PLOS ONE (October 21, 2015) (bibtex)
- O. Mangin, P.Y. Oudeyer, Learning semantic components from sub symbolic multi modal perception Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics (ICDL EpiRob), Osaka (Japan) (2013) (bibtex)
Please consider citing these papers when re-using the code in scientific publications.
This code is intended to work on its own. It however requires the dependencies mentioned below to be installed and available on the operating system. Please note the following elements.
- Locations: The file local.py is meant to hold local configuration of paths to data, features, etc. A template is provided in the repository. The paths defined in that file are only used when no path argument is provided to related functions (for examples the dataset loaders, or feature generators).
- Data: The experimental scripts require data (see below). This
data must be accessible from the script and eventually pre-processed.
For the motion data, pre-processing can be achieved through
multimodal/db/scripts/build_choreo2_features. Alternatively, all features and metadata for the databases can be downloaded by running
multimodal/db/scripts/download_dbs. This removes the need to pre-process the choreo2 dataset.
- Experiments: Just run the experimental script. Be sure to generate the required features. For the moment, the ACORNS Caregiver dataset is not available online, however feature files can be provided on request.
So for a quick hand on, just:
- Install dependencies:
pip install numpy scipy matplotlib librosa
- Download and install the sources:
git clone http://github.com/omangin/multimodal.git cd multimodal python setup.py develop --user
- Download databases features and metadata (run from where you downloaded the repository):
- Reproduce [Mangin2013]'s experiment:
Source files hierarchy:
- db: helpers to load and pre-process data
- features: helpers to build features
- lib: general purpose helpers
- experiments: experimental scripts
- test: a few unit tests
Main files in
experiment.pyThis file contains the main logic of the experimental setup. It defines classes to run multimodal experiments in a consistent and simplified manner. These include data preparation, training and evaluation.
learner.pyContains a class that abstracts the process of learning from multiple modalities. Contains the learning mechanism based on the NMF algorithm (itself included in
evaluation.pyHelpers to evaluate learning results.
pairing.pyHelpers to generate associations of samples from different modalities with same labels.
plots.pyPlot functions to generate specific figures.
Experimental scripts (in
icdl2013.pyRuns experiments and generate results corresponding to Mangin2013. Uses Acorns Caregiver database for speech sounds as well as the Choreo2 dataset.
image_sound.pyRuns experiments similar to Mangin2013 with speech sounds from the Acorns Caregiver database and images from the image database.
launcher.pyScript to run many experiments with variations of parameters. Requires expjobs and some time or a cluster.
image_sound_eval_sliding_windows.pyScript to run experiments with speech sound and images in which the learner is evaluated on its recognition sound from small time windows.
plot_image_sound_eval_sliding_windows.pyInteractive and static plots from the previous experiment.
plot_info_matrix.pyScript to generate plot of the mutual information between sample labels and internal coefficients. The script re-uses a trained dictionary obtained from
three_modalities.pyrun an experiment from a configuration file. Mainly used by
- Acorns Caregiver is available
Please refer to the article by Bergmann et
permission. However, feature and metadata files can be downloaded
through the code (see the
multimodal/db/scripts/download_dbs). See also .
- Choreo1 can be found at flowers.inria.fr/choreo/doc.
- Choreo2 can be found at flowers.inria.fr/choreo2. See also for features.
- Object images (not publicly available yet). Pictures acquired by
Natalia Lyubova and David
as frames from interaction with an iCub robot, through an RGBD sensor
(red, green, and blue camera coupled with a depth sensor). Feature
and meta files can be downloaded (see the
multimodal/db/scripts/download_dbs). See also .
This code is distributed under the new BSD license. Please see LICENSE for more details.
transformation.pyfrom ROS tf package.
- Uses a few functions imported from
scikit-learn. These are grouped in file
- Uses functions adapted from prettyplotlib.
- python >2.7 or >3
- librosa (for sound feature computation)