Flipping food during grilling tasks: a dataset of utensils kinematics and dynamics, food pose and subject gaze

Matlab and Python code used with the Food Flipping Dataset: a multivariate dataset recorded during 2866 food flipping movements, performed by 4 chefs and 5 home cooks. The subjects flipped different grilled food (hamburgers, chicken breast, zucchini and eggplant slices) using one of two possible utensils (spatula and tweezers).

The dataset contains: 3D trajectories of the utensils, 2D trajectories of the food, forces/torques applied with the utensils, subject's gaze data.

Matlab code for MoCap data (viconDataProcessing)

1. A__get_transformation_matrices_from_tweezers_markers_to_tips.m: computes the transformation matrices from the position of the MoCap markers to the tweezers' tips using the respective static trial.
2. B__get_transformation_matrices_from_spatula_markers_to_blade.m: computes the transformation matrices from the position of the MoCap markers to the corners of the spatula blade using the respective static trial.
3. C__post_process_dynamic_trials_for_the_dataset.m: used to (i) recover missing data in the markers trajectories of the dynamic trials and to (ii) calculate the trajectories of the tweezers' tips or spatula blade corners, during the dynamic trials (food flipping movements).
4. G__split_C3D_and_ASCII_files.m: used to split the trials by movement; some commands in this code can be reused with other datasets and exemplifies how to interact with Vicon Nexus to automatically manipulate data in this software.
5. visualize_the_trajectories_of_tweezers_and_spatula.m: this code can be used with any of the MAT files from the VICON_spatula and VICON_tweezers folders of the dataset, to visualize the tweezers/spatula moving in a 3D plot.

(code 3. uses the MoCap data recovery method from Tits M. et al. [1]; code 5. uses the plot functions from the matlab mocap toolbox [2])

Installation

• Download the repository (or just the folder viconDataProcessing);
• Download the repository of Tits M. et al (Robust and automatic MoCap data recovery) from: https://github.com/numediart/MocapRecovery;
• Download the repository of the MoCap Toolbox for Matlab from:https://www.jyu.fi/hytk/fi/laitokset/mutku/en/research/materials/mocaptoolbox;
• Place the folder of the toolbox inside the folder of the folder MocapRecovery-master;
• Place the folder MocapRecovery-master inside the folder viconDataProcessing;
• Explore the code in each .m file.
  
  

Python code for food tracking in video records (foodTracking)

Computer vision (CV) method to track the 2D position of the food in the video records of the experiments. The object detecting algorithm is based on the Mask R-CNN architecture [3].

1. retrain_network.py: use this to retrain the object detecting algorithm;
2. food_tracker.py: detects the food in the videos and tracks the food centroid; outputs a video having the bounding boxes and centroid of the detected food superimposed on the original video frames.
3. Object_Detection_Metrics/: code to compute 4 standard metrics (accuracy, precision, recall, F1 score), to assess the algorithm performance.

• read_csv_output_from_food_tracking.m: example to open, in Matlab, the CSV files in the dataset, generated by the food tracking algorithm, with the food trajectories.

Installation

• Download the repository (or just the folder foodTracking);
• Download and unpack the Mask-RCNN repository (https://github.com/matterport/Mask_RCNN)
• Copy the foodTracking repository into your_path/Mask_RCNN/samples/
• Download any required Python library currently missing in your Python environment
• Modify any path according to your environment in retrain_network.py and food_tracker.py
• Run the desired scripts on the videos of the dataset
  
  

Cite as

This research can be cited as:

Pereira D., De Pra Y., Tiberi E., Monaco V., Dario P., Ciuti G. (2022) Flipping food during grilling tasks: a dataset of utensils kinematics and dynamics, food pose and subject gaze. Scientific Data

Acknowledgments

This work was partially supported by Electrolux Professional SpA and by European Union’s Horizon 2020 research and innovation programme under grant agreement N. 730994, project TERRINet - The European Robotics Research Infrastructure Network.

References

[1] Tits, M., Tilmanne, J. & Dutoit, T. Robust and automatic motion-capture data recovery using soft skeleton constraints and model averaging.PloS one13, e0199744 (2018).

[2] Burger, B. & Toiviainen, P. MoCap Toolbox – A Matlab toolbox for computational analysis of movement data. In Bresin, R. (ed.)Proceedings of the 10th Sound and Music Computing Conference, 172–178 (KTH Royal Institute of Technology, Stockholm, Sweden, 2013).

[3] He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. Preprint at https://arxiv.org/abs/1703.06870 (2014).