Learning Precise Temporal Point Event Detection with Misaligned Labels

AAAI 2021

This work addresses the problem of robustly learning precise temporal point event detection despite only having access to poorly aligned labels for training. While standard (cross entropy-based) methods work well in noise-free setting, they often fail when labels are unreliable since they attempt to strictly fit the annotations. A common solution to this drawback is to transform the point prediction problem into a distribution prediction problem. We show however that this approach raises several issues that negatively affect the robust learning of temporal localization. Thus, in an attempt to overcome these shortcomings, we introduce a simple and versatile training paradigm combining soft localization learning with counting-based sparsity regularization. In fact, unlike its counterparts, our approach allows to directly infer clear-cut point predictions in an end-to-end fashion while relaxing the reliance of the training on the exact position of labels. We achieve state-of-the-art performance against standard benchmarks in a number of challenging experiments (e.g., detection of instantaneous events in videos and music transcription) by simply replacing the original loss function with our novel alternative---without any additional fine-tuning.

---

(Section 5.1) Golf Swing Sequencing

            [Benchmark Code and Dataset] by McNally et al.

To run the original code (CE) with noise level (n) on split (s), (default: n=0, s=1):

python original_train.py $n $s
python original_eval.py $n $s

To run either the classical one-sided-smoothing (classic=True) or our SoftLoc approach (classic=False) with noise level (n) on split (s), (default: n=0, s=1, SoftLoc loss):

python soft_train.py $n $s $classic
python soft_eval.py $n $s $classic

The results are then save in .txt file in /results.

For the causal experiments, manually change the bidirectional argument in the EventDetect defintion to False (soft_train.py (l.69), soft_eval.py (l.92), original_train.py (l.42), and original_eval.py (l.80)).

---

(Section 5.2) Time Series Detection

            [Benchmark Code] by Adams and Marlin

            [Dataset Request] The dataset has to be requested from the authors of puffMarker: A Multi-Sensor Approach for Pinpointing the Timing of First Lapse in Smoking Cessation.

To run the experiment on split (s) with noise level (n):

python PuffDetection.py $loss_id $s $n $distribution_id

• loss_id: SoftLoc (loss_id=0), cross-entropy (loss_id=1), one-sided-smoothing (loss_id=2)

• distribution_id: normal (distribution_id=0), binary (distribution_id=1), skew normal (distribution_id=2)

---

(Section 5.3) Piano Onset Experiment

            [Benchmark Code] by Hawthorne et al.

            [Dataset Request] for MAPS Database.

To run the SoftLoc pipeline (to modify the noise level, change value in bash):

bash piepline.sh

Modify the oneSided variable in pipeline.sh in order to run the one-sided smoothing benchmark.

The project is structured as follows:

• pipeline.sh (Full pipeline)
• SoftNetworkModel.py (Tensorflow model with SoftLoc loss)
• main.py (Main script that runs the training)
• createDataset.py and google_create_dataset.py (dataset creation)
• infer.py and final_score.py (inference)
• config.py (Configuration file)

In addition, subfolders contains all utility functions used throughout the project.

---

(Section 5.4) Drum Detection Experiment

            [Benchmark Code] by Schroeter et al.

            [Dataset Request] for IDMT-SMT-Drums.

In order to compute and add one random point to the heatmap (Figure 6), run the following line:

python multi_main.py

This will run a single run of the experiment with random softness and noise level.

• The results are computed and added to a results file in addSweep.py.
• The heatmap is generated with sweepVisualization.py