[NeurIPS 2023 Dataset and Benchmark Track] Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context
Official API for our dataset submission to NeurIPS 2023.
This repository is work-in-progress until NeurIPS 2023 (Dec. 10th 2023)
Forecasting human motion of multiple persons is very challenging. It requires to model the interactions between humans and the interactions with objects and the environment. For example, a person might want to make a coffee, but if the coffee machine is already occupied the person will have to wait. These complex relations between scene geometry and persons arise constantly in our daily lives, and models that wish to accurately forecast human behavior will have to take them into consideration. To facilitate research in this direction, we propose Humans in Kitchens, a large-scale multi-person human motion dataset with annotated 3D human poses, scene geometry and activities per person and frame. Our dataset consists of over 7.3h recorded data of up to 16 persons at the same time in four kitchen scenes, with more than 4M annotated human poses, represented by a parametric 3D body model. In addition, dynamic scene geometry and objects like chair or cupboard are annotated per frame. As first benchmarks, we propose two protocols for short-term and long-term human motion forecasting.
Download the dataset (preliminary version 0.0.1). The dataset has the following structure
data/body_models/SMPLX_NEUTRAL.npz
poses/{dataset}_{pid}_{seqid}.npz
scenes/{dataset}_scene/{object_name}.json{object_name}.npy
In the rest of the documentation we assume that the folder data/ is located at /path/to/dataset/data/.
Install the tool via pip:
pip install git+https://github.com/jutanke/hik.git
For evaluation, ndms has to be installed too:
pip install git+https://github.com/jutanke/ndms.git
import matplotlib.pylab as plt
import numpy as np
from hik.data.kitchen import Kitchen
from hik.data import PersonSequences
from hik.vis import plot_pose
# load geometry
kitchen = Kitchen.load_for_dataset(
dataset=dataset,
data_location="/path/to/dataset/data/scenes"
)
# load poses
person_seqs = PersonSequences(
person_path="/path/to/dataset/data/poses"
)
smplx_path = "/path/to/dataset/data/body_models"
fig = plt.figure(figsize=(10, 10))
ax = fig.add_subplot(111, projection="3d")
dataset = "C" # ["A", "B", "C", "D"]
frame = 12345
kitchen.plot(ax, frame)
for person in person_seqs.get_frame(dataset, frame):
plot_pose(ax, person["pose3d"], linewidth=1)
ax.axis('off')| 3D joints (body) | 3D joints (head) |
|---|---|
 |
 |
@article{tanke2023humanskitch,
title={Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context},
author={Tanke, Julian and Kwon, Oh-Hun and Mueller, Felix and Doering, Andreas and Gall, Juergen},
journal={Advances in Neural Information Processing Systems},
year={2023}
}
Speed-up sequences for all 4 kitchens
