Simian

A synthetic data generator for video caption pairs.

Simian creates synthetic data that is usable for generative video and video captioning tasks. The data consists of videos and captions. The videos are generated using Blender, a 3D modeling software.

🖥️ Setup

NOTE: Simian requires Python 3.11.

1. Install dependences:

pip install -r requirements.txt

2. Download the datasets:

./scripts/get_data.sh

3. [OPTIONAL] If you're on a headless Linux server, install Xorg and start it:

sudo apt-get install xserver-xorg -y && \
sudo python3 scripts/start_x_server.py start

📸 Usage

Generating Combinations

python3 -m simian.combiner --count 1000 --seed 42

Generating Videos or Images

Configure the flags as needed:

• --width and --height are the resolution of the video.
• --combination_index is the index of the combination to render.
• --output_dir is the directory to save the rendered video.
• --hdri_path is the directory containing the background images.
• --start_frame and --end_frame are the start and end frames of the video.
• --animation_length is the total number of frames, this as a result controls the how fast the animation will appear.
• --images adding this will output images instead of video at random frames. Creates multiple images per combination of varying sizes

Or generate all or part of the combination set using the batch.py script:

Generating Videos or Images

To generate a video(s):

python3 -m simian.batch --start_index 0 --end_index 1000 --width 1024 --height 576 --start_frame 1 --end_frame 65 --animation_length 120

To generate an image(s):

python3 -m simian.batch --start_index 0 --end_index 1000 --width 1024 --height 576 --start_frame 1 --end_frame 65 --animation_length 120 --images

You can also generate individually:

# MacOS
python -m simian.render

# Linux
python -m simian.render

## Kitchen sink
python -m simian.render -- --width 1920 --height 1080 --combination_index 0 --output_dir ./renders --hdri_path ./backgrounds --start_frame 1 --end_frame 65

Clean up Captions

Make captions more prompt friendly.

NOTE: Create a .env file and add your OpenAI API key

python3 scripts/rewrite_captions.py

Distributed rendering

Rendering can be distributed across multiple machines using the "simian.py" and "worker.py" scripts.

You will need to set up Redis and obtain Huggingface API key to use this feature.

Set Up Redis

You can make a free Redis account here.

For local testing and multiple local workers, you can use the following script to set up a local instance of Redis:

scripts/setup_redis.sh

Huggingface API Key

You can get a Huggingface API key here.

Now, start your workers

export REDIS_HOST=<myhost>.com
export REDIS_PORT=1337
export REDIS_USER=default
export REDIS_PASSWORD=<somepassword>
export HF_TOKEN=<token>
export HF_REPO_ID=<repo_id>
celery -A simian.worker worker --loglevel=info

You can also build and run the worker with Docker

# build the container
docker build -t simian-worker .

# run the container with .env
docker run --env-file .env simian-worker

# run the container with environment variables
docker run -e REDIS_HOST={myhost} -e REDIS_PORT={port} -e REDIS_USER=default -e REDIS_PASSWORD={some password} -e HF_TOKEN={token} -e HF_REPO_ID={repo_id} simian-worker

Finally, issue work to your task queue

python3 -m simian.distributed --width 1024 --height 576

If you want to use a custom or hosted Redis instance (recommended), you can add the redis details like this:

export REDIS_HOST=<myhost>.com
export REDIS_PORT=1337
export REDIS_USER=default
export REDIS_PASSWORD=<somepassword>

To run all tests

python3 -m simian.tests.__run__

To run tests look into the test folder and run whichever test file you want

python3 -m simian.tests.object_test

📁 Datasets

We are currently using the following datasets: Objaverse

Backgrounds are loaded from: Poly Haven

🦝 Contributing

We welcome contributions! We're especially interested in help adding and refining datasets, improving generation quality, adding new features and dynamics and allowing the project to meet more use cases.

How to contribute

1. Check out the issues here.
2. Join our Discord here.
3. Get in touch with us so we can coordinate on development.
4. Or, you know, just YOLO a pull request. We're pretty chill.

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

If you use it, please cite us:

@misc{Simian,
  author = {Raccoon Research},
  title = {Simian: A Synthetic Data Generator for Video Caption Pairs},
  year = {2024},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/RaccoonResearch/simian}}
}

Contributors ✨

This project follows the all-contributors specification. Contributions of any kind welcome!

M̵̞̗̝̼̅̏̎͝Ȯ̴̝̻̊̃̋̀Õ̷̼͋N̸̩̿͜ ̶̜̠̹̼̩͒ 🚇 ⚠️ 💻 📖

Eric S 💻 ⚠️ 📖

Sponsors

Raccoon Research is sponsored by the following organizations:

Interested in working with us? Join our Discord or post an issue to get in touch.