This code package contains links and instructions for downloading/using the demonstration and evaluation datasets presented in the paper "BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks".
This code repository contains an example implementation of behavioural cloning using the demonstration dataset.
scripts/filelists/*.txt contains text files that list links for downloading the individual trajectories for the demonstration dataset. You can use wget or your favorite download tools to download the files. We have also included scripts/download_demonstration_data.sh to download the data using wget. This command will download all ~700GB of the data:
bash ./scripts/download_demonstration_data.sh path_to_directory_for_dataTo download only part of the data, add the optional argument for maximum download size in MB. This command will download 1000MB per task (i.e., a total of 4000MB plus 300MB VPT model):
bash ./scripts/download_demonstration_data.sh path_to_directory_for_data 1000The evaluation dataset is available in the following Zenodo record: https://zenodo.org/record/8021960.
For using the dataset to do human evaluation, see evaluation_server/README.md for further instructions.
Tested system: Ubuntu with 16 CPU cores, 64GB of RAM and Nvidia Tesla T4 GPU.
Additional requirements:
sudo apt install xvfb ffmpegInstallation:
conda env create -f environment.yml
conda activate basalt
# Install the `basalt` library so that it is available anywhere.
# This is also required for running the code.
pip install -e .For a small-scale test run (5GB of data downloaded, minimal amount of training), run this command:
bash ./scripts/test_pipeline.sh .This will do the following:
- Download 5GB of data: a small subset of data for each BASALT task + the smallest OpenAI VPT foundation model for playing Minecraft.
- Embed the dataset using the VPT model (turn video frames into smaller embeddings).
- Train a behavioural cloning agent on top of the embeddings using the
imitationlibrary - Roll out the trained model in Minecraft to produce the videos ready for human evaluation.
This will create a new directory pipeline_test_data, which will contain all the downloaded data, trained models and resulting rollouts.
If everything is successful, you should have a directory pipeline_test_data/rollouts/ directory with the following structure:
pipeline_test_data/rollouts/
├── MineRLBasaltBuildVillageHouse-v0
│ ├── seed_12658.jsonl
│ ├── seed_12658.mp4
| ...
│ ├── seed_99066.jsonl
│ └── seed_99066.mp4
├── MineRLBasaltCreateVillageAnimalPen-v0
│ ├── seed_14975.jsonl
│ ├── seed_14975.mp4
| ...
│ ├── seed_85236.jsonl
│ └── seed_85236.mp4
├── MineRLBasaltFindCave-v0
│ ├── seed_14169.jsonl
│ ├── seed_14169.mp4
| ...
│ ├── seed_95099.jsonl
│ └── seed_95099.mp4
└── MineRLBasaltMakeWaterfall-v0
├── seed_12095.jsonl
├── seed_12095.mp4
...
├── seed_95674.jsonl
└── seed_95674.mp4
These video files are now ready to be used for human evaluation. See evaluation_server/README.md for instructions on how to set up the server and gather human rankings of this agent versus the ones shared in the Evaluation Dataset.
For full training, run the following scripts. Make sure the experiment output directory basalt_output_directory has at least 1TB of free space, as this will place all downloaded and produced data there.
basalt_output_directory="/path/to/place/for/all/the/data"
bash ./scripts/download_demonstration_data.sh $basalt_output_directory
bash ./scripts/embed_trajectories.sh $basalt_output_directory
bash ./scripts/train_bc.sh $basalt_output_directory
bash ./scripts/rollout_bc.sh $basalt_output_directoryThe whole process takes roughly 2-3 days on a 16-core, 64GB RAM, Nvidia Tesla T4 system.
MIT. See LICENSE.
Contents under basalt/vpt_lib are originally from this repository, shared under MIT license.