Dataset

Dataset and split list:

Name	Setting
Blockworld	Dynamic
	Static
	Textured
MNIST World	Dynamic partially observed
	Static partially observed
	Dynamic fully observed
	Dynamic fully observed, no self motion

For this anonymous version we do not include the download script for the dataset.

Adding a New Video Dataset

To add a new video dataset, structure the dataset directory as follows:

data/
├── {dataset_name}/
│   ├── training/
│   │   ├── video_xxx.mp4
│   │   ├── ...
│   ├── validation/
│   │   ├── video_xxx.mp4
│   │   ├── ...
│   ├── metadata/
│   │   ├── training.pt
│   │   ├── validation.pt
│   │   ├── ...

Next, implement the dataset class in datasets/video/{dataset_name}.py, inheriting BaseAdvancedVideoDataset, then override the necessary methods. Additionally, define the dataset configuration in configurations/dataset/{dataset_name}.yaml. While having all three splits (training, validation, test) is not required, each split folder can be structured arbitrarily as long as the metadata files correctly reference the video files.

Please see the example in blockworld.py for more guidance. If training DFoT or DFoT-SSM, you may need to train a new VAE as well; instructions for that are below. Fill in _ALL_SPLITS, and the metadata construction function in order to use it. You may also choose not to use the metadata structure.