This repo contains the code of the paper:
Data-driven Synthesis of Smoke Flows with Cnn-based Feature Descriptors, Mengyu Chu and Nils Thuerey.
- Install CMake, python, tensorflow (numpy), matplot, sklearn, flann(more details in the very last section)
- Clone this repo
git clone https://github.com/RachelCmy/mantaPatch.git
- The code is implemented on mantaflow. Here is an installation guide: http://mantaflow.com/install.html
- When building mantaflow with CMake, it is necessary to check
NUMPYon. It is recommended to check the
GUI(depends on Qt) and
OPENMPon as well.
- eigen(http://eigen.tuxfamily.org/index.php?title=Main_Page) is used in our code.
We have one under
dependenciesfolder, and it is used by default on Windows. On Linux, please edit the eigen path in
CMakeLists.txtfile, at line 481
- Related scene files are in folder
- Output files are usually in folder
- Note the working folder used below is always
mantabelow stands for the executable file.
- On Windows, it is
- On Linux, it is
- On Windows, it is
Try Patch Advection
Examples of deformation-limiting patch advection scenes in 3d and 2d:
manta ../scenes/patch_2d.py # or manta ../scenes/patch_3d.py
2d is visualized with smile faces and 3d is visualized with meshes( try shift+m ).
Train a patch-descriptor-learning CNN
A Simulation Pair Generation:
# for 2D manta ../tensorflow/example3_smoke_patch/genSimData.py # or for 3D manta ../tensorflow/example3_smoke_patch/genSimData.py dim 3
It will generate one simulation pair. All data is saved in
../tensorflow/data/sim_????/folder. 3D simulations will take longer time. It is necessary to generate lots of pairs (with different seeds) to get good training results.
Examples for parameters:
# example1, input a single number, as random seed, # for e.g. the 42 will be used as random seed here. manta ../tensorflow/example3_smoke_patch/genSimData.py 42 # example2, input a parameter-list. # Only right names are accepted # ( take a look at line 252 in genSimData.py) manta ../tensorflow/example3_smoke_patch/genSimData.py npSeed 42 resetN 20 saveppm False
Patch Pairs Generation (Training Data for step 3):
manta ../tensorflow/example3_smoke_patch/genPatchData.py 1000
This command will generate the patch pairs from the simulation in
data/sim_1000/All data is saved in
../tensorflow/data/patch_1000/folder. 1000 can be replaced with any other valid
simNo. A special case is
simNo == -1
manta ../tensorflow/example3_smoke_patch/genPatchData.py -1 # or manta ../tensorflow/example3_smoke_patch/genPatchData.py simNo -1 savePatchPpm True
This command will generate patch pairs from all existed simulations in folders named as
data/sim_????/, except for the ones whose patch folders already exist.
Examples for parameter-list:
# Only right names are accepted #( take a look at line 128 in genSimData.py) manta ../tensorflow/example3_smoke_patch/genPatchData.py \ simNo 1002 PATCH_SIZE 48 savePatchPpm False
Note that the factor between coase and fine are already defined in the simulation pair.
PATCH_SIZEis used as the size for high resolution patches, and for coarse ones the size is
PATCH_SIZE/factor. And 36 (by default) is recommanded.
savePatchPpmis true, ppm image files will be outputed with patch uni files. They are very large ( no image compression used ), but can help to visualize patches, with the software djv_view(http://djv.sourceforge.net/).
Train a Saimese CNN: Parameter-list is at line 253 in trainPatchData.py. Parameter
data_dirsstands for the training dataset pathes. It should be in a pair of quotes, multiple pathes should be seperated with commas, for e.g.,
# train a density descriptor first manta ../tensorflow/example3_smoke_patch/trainPatchData.py \ data_dirs "../tensorflow/data/patch_1000/, ../tensorflow/data/patch_1001/" # train a curl-of-velocity descriptor as well manta ../tensorflow/example3_smoke_patch/trainPatchData.py \ data_dirs "../tensorflow/data/patch_1000/, ../tensorflow/data/patch_1001/" dataFlag 2 # make sure that data_dirs path strings end with "/" or "\\" !
Training log, trained models and other files are saved in
Evaluate the CNN: Parameter-list is at line 207 in RecallEval.py.
For a given evaluation dataset( should be different from the training one), we evaluate the given trained model with the recall over rank curve.
# evaluate a density descriptor (dataFlag = 1) manta ../tensorflow/example3_smoke_patch/RecallEval.py \ data_dirs "../tensorflow/data/patch_1002/" \ saved_model ../tensorflow/data/_trainlog_yyyymmdd_hhmmss/model_yyyymmdd_hhmmss.ckpt \ dataFlag 1 # evaluate a curl descriptor (dataFlag = 2) manta ../tensorflow/example3_smoke_patch/RecallEval.py \ data_dirs "../tensorflow/data/patch_1002/" \ saved_model ../tensorflow/data/_trainlog_yyyymmdd_hhmmss/model_yyyymmdd_hhmmss.ckpt \ dataFlag 2 # param color could be used to set the color of the recall curve # make sure that dataFlag and the saved_model are consistent # (both for density or both for curl, and both for 2D or 3D).
Evaluation log, recall curve, descriptors and other files are saved in
data/_evaltlog_yyyymmdd_hhmmss/folder. For parameter
saved_model, our trained models are offered in folder
mantaflow_path/tensorflow/example3_smoke_patch/models/Note here, unlike the training code, there is no automatic dataset separation, so a relatively smaller evaluation dataset is supposed to be used.
Prepare a repository
High-resolution Simulation Generation Only: It is possible to reuse the
training/evaluationdataset, but it is better to create some even larger-resolution simulations, for e.g.,
manta ../tensorflow/example3_smoke_patch/genSimData.py doCoarse False res 128 simNo 2000
You can also use specially designed simulations ( a special simNo should be used ), for e.g.,
# for 2d, the only "special" part is that the random seeds are fixed ( at line 62 ) manta ../tensorflow/example3_smoke_patch/genSimDataSpecial.py # or for 3d, the only "special" part is that the random seeds are fixed ( at line 62 ) manta ../tensorflow/example3_smoke_patch/genSimDataSpecial.py dim 3 # high-res in 3D is very slow
The output files should have the same folder structure. Then generate larger patches as well (PATCH_SIZE = 64, 72 or even larger):
manta ../tensorflow/example3_smoke_patch/genPatchData.py \ doCoarse False simNo 2000 PATCH_SIZE 64
Use a trained CNN model to extract all descriptors:
# density des # note that PATCH_SIZE should be the same as the trained CNN # instead of the repository patch size. manta ../tensorflow/example3_smoke_patch/libDesPack.py dataFlag 1 PATCH_SIZE 36 saved_model ../tensorflow/example3_smoke_patch/models/model_2D/den_cnn.ckpt data_dirs "../tensorflow/data/patch_2000/" # curl of velocity des manta ../tensorflow/example3_smoke_patch/libDesPack.py dataFlag 2 PATCH_SIZE 36 saved_model ../tensorflow/example3_smoke_patch/models/model_2D/curl_cnn.ckpt data_dirs "../tensorflow/data/patch_2000/" # or use some other model: # saved_model ../tensorflow/data/_trainlog_yyyymmdd_hhmmss/model_yyyymmdd_hhmmss.ckpt # set DIM as 3 to work with 3D repository
Output descriptor npz file will be saved in
data/_deslog_yyyymmdd_hhmmss/folder, unless parameter
desdataPathis assigned as other output directory. If the parameter
True(as default), the descriptor file will be copied to the repository folder as well.
Get new high-resolution simulations
A 2D example of smoke synthesis with repository patches using CNN descriptors.
manta ../tensorflow/example3_smoke_patch/patch_2d.py \ SynCurlFlag True data_dirs "../tensorflow/data/patch_2000/" \ den_model ../tensorflow/data/_trainlog_yyyymmdd_hhmmss/model_yyyymmdd_hhmmss.ckpt \ curl_model ../tensorflow/data/_trainlog_yyyymmdd_hhmmss/model_yyyymmdd_hhmmss.ckpt # den_model & curl_model, our trained models are used by default (if not given)
An extra library, flann, is used to build the kd-tree of the repository. It can be installed with pip, more info: https://github.com/primetang/pyflann
For python3, it is necessary to fix some problems, according to: primetang/pyflann#1 It is important to use the same model as used in the repository (See Prepare a repository) Our repository patches (using our models) are not uploaded with the code. But the 2D version can be downloaded from: https://ge.in.tum.de/download/data/mantaPatchData.7z
wget https://ge.in.tum.de/download/data/mantaPatchData.7z 7z x mantaPatchData.7z -o../tensorflow/example3_smoke_patch/ # or on windows, 64 bit (install https://eternallybored.org/misc/wget/) wget64 https://ge.in.tum.de/download/data/mantaPatchData.7z "path/to/7z.exe" x mantaPatchData.7z -o../tensorflow/example3_smoke_patch/ # then you can run the patch_2d.py with all default parameters manta ../tensorflow/example3_smoke_patch/patch_2d.py
Output simulations are saved in folder
../tensorflow/data/_simlog_yyyymmdd_hhmmss/, and ppm files are saved there for visualization.
A 3D example:
manta ../tensorflow/example3_smoke_patch/patch_3d.py \ data_dirs "../tensorflow/data/patch_7001/" \ den_model ../tensorflow/example3_smoke_patch/models/model_3D/den_cnn.ckpt \ curl_model ../tensorflow/example3_smoke_patch/models/model_3D/curl_cnn.ckpt
In this code, we still load all the patches and do the volume synthesis frame by frame. Additionally, we save all the high-res output uni files in to output folder. That's why this is still slow. As written in the paper, one can also only save all patch info to the rendering scene file, and don't high-res patches until rendering when every frame could work in parallel. In that way, it is more effective. The rendering part (as well as the generation of rendering scene files) is not included with this code yet.
This work was funded by the ERC Starting Grant realFlow (ERC StG-2015-637014).