Inference and Reproducing Results

Checkpoints

Using your own checkpoints

All logs are logged to the cloud on wandb automatically. Your training will produce a wandb link that leads to the logs. To resume the a run with your checkpoint, fill in the model training checkpoint path to load= or the model weights path to algorithm.load_model_state. To resume the same wandb run, fill in resume={the_wandb_run_id}.

Using our pre-trained checkpoints

The pretrained checkpoints are not available for download in this anonymous version.

• These commands assume ckpt_map=default (see configurations/ckpt_map/default.yaml).
• +name=... is the run name (used for logging / output directories); feel free to change.

Blockworld Inference Commands

The following commands correspond to the paper inference configs under configurations/shortcode/exp/blockworld/**/infer/, use them to run inference on Block world:

Split	Method	# Total Rollout Frames	Command
Dynamic Blockworld	FloWM	140	python -m main shortcode=exp/blockworld/flowm/infer/metrics_140/dynamic_70ctx +name=infer_blockworld_flowm_dynamic_70ctx_140 algorithm=flowm_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/flowm/infer/metrics_280/dynamic_70ctx +name=infer_blockworld_flowm_dynamic_70ctx_280 algorithm=flowm_video dataset=blockworld ckpt_map=default
	DFoT	140	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_140wf/dynamic +name=infer_blockworld_dfot_dynamic_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_280wf/dynamic +name=infer_blockworld_dfot_dynamic_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default
	DFoT-SSM	140	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_140wf/dynamic +name=blockworld_v2_dynamic_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_280wf/dynamic +name=infer_blockworld_ssm_dynamic_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default
---	---	---	---
Textured Blockworld	FloWM	140	python -m main shortcode=exp/blockworld/flowm/infer/metrics_140/tex_70ctx +name=infer_blockworld_flowm_tex_70ctx_140 algorithm=flowm_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/flowm/infer/metrics_280/tex_70ctx +name=infer_blockworld_flowm_tex_70ctx_280 algorithm=flowm_video dataset=blockworld ckpt_map=default
	DFoT	140	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_140wf/tex +name=infer_blockworld_dfot_tex_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_280wf/tex +name=infer_blockworld_dfot_tex_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default
	DFoT-SSM	140	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_140wf/tex +name=infer_blockworld_ssm_tex_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_280wf/tex +name=infer_blockworld_ssm_tex_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default
---	---	---	---
Static Blockworld	FloWM	140	python -m main shortcode=exp/blockworld/flowm/infer/metrics_140/static_70ctx +name=infer_blockworld_flowm_static_70ctx_140 algorithm=flowm_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/flowm/infer/metrics_280/static_70ctx +name=infer_blockworld_flowm_static_70ctx_280 algorithm=flowm_video dataset=blockworld ckpt_map=default
	DFoT	140	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_140wf/static +name=infer_blockworld_dfot_static_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/dfot/infer/metrics_70ctf_280wf/static +name=infer_blockworld_dfot_static_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default
	DFoT-SSM	140	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_140wf/static +name=infer_blockworld_ssm_static_70ctx_140wf algorithm=dfot_video dataset=blockworld ckpt_map=default
		280	python -m main shortcode=exp/blockworld/block-ssm/infer/metrics_70ctf_280wf/static +name=infer_blockworld_ssm_static_70ctx_280wf algorithm=dfot_video dataset=blockworld ckpt_map=default

MNIST World Inference Commands

The following commands correspond to the paper inference configs under configurations/shortcode/exp/mnist_world/*/infer/.... They evaluate on the validation splits (dynamic_validation for 70 frames and dynamic_validation_200 for 200 frames).

Dataset Split	Method	# Total Rollout Frames	Command
Dynamic PO MNIST WOrld	FloWM	70	python -m main shortcode=exp/mnist_world/flowm/infer/dynamic_po_70wf +name=mnist_world_inference_dynamic_paper_70 algorithm=flowm_video dataset=mnist_world ckpt_map=default
		200	python -m main shortcode=exp/mnist_world/flowm/infer/dynamic_po_200wf +name=mnist_world_inference_dynamic_paper_200 algorithm=flowm_video dataset=mnist_world ckpt_map=default
	DFoT	70	python -m main shortcode=exp/mnist_world/dfot/infer/dynamic_po_70wf +name=dfot_mnist_70 algorithm=dfot_video dataset=mnist_world ckpt_map=default
		200	python -m main shortcode=exp/mnist_world/dfot/infer/dynamic_po_200wf +name=dfot_mnist_200 algorithm=dfot_video dataset=mnist_world ckpt_map=default
	DFoT-SSM	70	python -m main shortcode=exp/mnist_world/block-ssm/infer/dynamic_po_70wf +name=ssm_mnist_70 algorithm=dfot_video dataset=mnist_world ckpt_map=default
		200	python -m main shortcode=exp/mnist_world/block-ssm/infer/dynamic_po_200wf +name=ssm_mnist_200 algorithm=dfot_video dataset=mnist_world ckpt_map=default