PhyFu: A Fuzzer for Modern Physics Simulation Engines

A Logic Bug Detector for Physics Simulation Engines

Summary • Installation • Quick Start
Philosophy • Misc • Credits • Paper

Summary

PhyFu (Phy - physics, Fu - fuzzer) is a fuzzing framework designed specifically for Physics Simulation Engines (PSEs) to uncover logic errors. PhyFu can holistically detect errors in both the forward and backward simulation phase of PSEs.

What is PhyFu?

PhyFu is a fuzzing framework for uncovering logic errors, i.e., errors that silently cause incorrect results without directly causing crashes, in the Physics Simulation Engines (or abbreviated as PSEs). To date, PhyFu has detected over 5K error-triggering inputs and over 20 crashes in 8 combinations of PSEs and physical scenarios.

To learn more about the philosophy of the project check the philosophy section.

❗ IMPORTANT: PhyFu is young software and just experienced a major refactoring process. Sometimes the code may not work as expected.

🔧 Installation

PhyFu requires a Linux machine with CUDA drivers and cudatoolkit to operate.

You can install it through your favorite environment manager:

• conda

  git clone https://github.com/PhyFuzz/phyfu.git
  cd phyfu
  conda create -n phyfu python=3.8
  conda activate phyfu
  make develop

• git clone https://github.com/PhyFuzz/phyfu.git
  cd phyfu
  python3 -m pip install --user --upgrade pip
  python3 -m pip install --user virtualenv
  python3 -m venv env
  source env/bin/activate
  make develop

  virtualenv

The last step of the installation process may take several to dozens of minutes, depending on your network speed.

So far, PhyFu is installed in development mode, so that you can easily modify the code without need to rerun the installation process. To install PhyFu in production mode, run make install instead of make develop. This will install PhyFu in your system's Python environment.

⚙ Quick Start

To quickly try our tool, run the following to fuzz the Taichi PSE with 200 seeds and seed scheduling enabled:

phyfu.fuzz taichi two_balls --test_times 200 --seed_getter art

Wait for about 3 minutes, and you will see the following output in the end:

#loss_too_large: 17
#deviated_init_state: 3

It means that PhyFu has found 17 errors that are caused by the loss value being too large (backward errors), and 3 errors that are caused by the deviation of the initial state (forward errors).

It works, cool! What are the next steps?

There are three things that you can do next:

1. Run large scale experiments to evaluate PhyFu's effectiveness.
2. Analyze the errors found by PhyFu in the large scale experiments.
3. Extend PhyFu to test more PSEs and physical scenarios.

Setting these up is discussed in the readthedocs, so be sure to check there if you are interested!

❓ Philosophy

One of the key challenges in detecting logic errors is designing an oracle that can decide whether the output results are wrong or not. Based on principled physics laws, PhyFu proposes two testing oracles that can holistically detect errors in both the forward and backward simulation phases of PSEs. Specifically,

1. PhyFu mutates initial states and asserts if the PSE under test behaves consistently with respect the two testing oracles.
2. Further with feedback-driven test case scheduling, the search process for errors is significantly accelerated.
3. PhyFu also leverages a set of design choices to ensure the validity of testing seeds.

For more details, please refer to our paper.

📚 FAQ

The readthedocs is the go-to place if you need answers to anything PhyFu-related. Usage, APIs, Extensions? It's all there, so we recommend you seriously go read it!

🍀 Misc

The hyper-parameters we use are all listed in phyfu/configs/. For the meaning and usage of each hyper-parameter, please refer to the readthedocs.

For the omitted proofs of some theorems in the paper, please refer to here.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template. This REAME file is adapted from nvim-neorg/neorg.

Paper

This paper is accepted at The 38th IEEE/ACM International Conference on Automated Software Engineering, a.k.a. ASE 2023. Please cite the published version as soon as this paper appears in the conference publications.

📜 PHYFU: Fuzzing Modern Physics Simulation Engines. [click :: citation]

@misc{xiao2023phyfu,
      title={PHYFU: Fuzzing Modern Physics Simulation Engines}, 
      author={Dongwei Xiao and Zhibo Liu and Shuai Wang},
      year={2023},
      eprint={2307.10818},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}