Fruit fly body model tutorial

Local setup (Linux/macOS/Windows):

1. Clone this repository.
2. Install the Pixi package manager.
3. Run pixi install in the body_tutorial directory.
4. Point your notebook editor of choice to the Python interpreter installed in the .pixi directory and open tutorial.ipynb.

If you don't have a notebook editor installed, you can run

pixi run jupyter notebook tutorial.ipynb

to open the tutorial in the Jupyter editor.

Colab setup (Ubuntu 22.04 runtime, the default as of July 2025):

Open tutorial.ipynb in Colab, and then run the following in the terminal, or in a notebook cell with a preceeding "!":

pip install h5py matplotlib mujoco==3.3.3 numpy onnx onnxruntime pillow rich scipy && \
git clone https://github.com/TuragaLab/flysim_tutorials.git /tmp/tutorial_repo && \
mv /tmp/tutorial_repo/body_tutorial/projectlib . && \
rm -rf /tmp/tutorial_repo && \
echo '{"file_format_version": "1.0.0", "ICD": {"library_path": "libEGL_nvidia.so.0"}}' \
     > /usr/share/glvnd/egl_vendor.d/10_nvidia.json

Paperspace Gradient setup (Ubuntu 22.04 runtime, the default as of July 2025):

Clone this repository, and then run the following in the terminal:

pip uninstall -y tensorboard tensorflow wandb && \
pip install h5py matplotlib mujoco==3.3.3 numpy onnx onnxruntime pillow rich scipy

Note: The Gradient platform doesn't currently support hardware-accelerated rendering with Mujoco, so running the body tutorial code on Gradient will probably be somewhat slower than running it locally or on Colab.

Project ideas

• State space mapping: How complicated of a behavior is six-legged walking on a flat plane? One might imagine that the positions of some joints and/or the measurements collected from some sensors are highly correlated. How many dimensions are needed to faithfully characterize the fly's instantaneous position at some point in a straight walking trajectory on a flat plane? What about if the fly is moving in a winding trajectory, or walking over obstacles?

• Controller distillation: Is it possible to make the walking controller simpler and easier to understand while approximately preserving its function? If you train simpler networks to mimic the controller's behavior, how does the control quality fall off as the network size decreases? In addition to simplifying the controller internally, it might also be possible to simplify or ignore some of its inputs.

• Controller stress testing: Mujoco allows us to simulate flies in arbitrary environments. In which environments does the walking controller work well, and in which environments does it fail? (e.g., how uneven can the terrain be before the fly falls over?) You can consider other deviations from the training conditions as well. For example, many animals gradually change size over the course of their life, and maintain their ability to walk as they grow. How robust is the walking controller to changes in the size of some or all of the fly's body parts?

• Controller training: The imitation learning procedure used to optimize the walking controller takes about a week to run, so we don't suggest trying to replicate it exactly. But it could still be interesting to see whether it's possible to approximate it (or improve upon it) using a simpler training objective. e.g., optimizing for some combination of walking speed, energy consumption, head stability, joint stress, and/or trajectory smoothness.

Acknowledgments

This tutorial was copied from the body model tutorial by Mason McGill.