"generating annotated images of animals in complex environments with Unreal Engine"
by Fabian Plum, René Bulla, Hendrik Beck, Natalie Imirzian, and David Labonte (2023)
- Blender (v3.0.1 or later)
- send2unreal (v2.1 or later)
- Unreal engine (installed via Epic Games Launcher, see above, v5.0.2 or later)
- Patience. (lots, where available)
- download send2unreal and don't unzip the file. You can install the addon in its zipped form straight from Blender. In Blender, under edit/preferences/Add-ons click on Install... and navigate to the send2ue zip file on your computer.
- You should now see a new top at the top of your Blender Window, aptly named Pipeline. Click on it, and then select Export/Settings_Dialog to configure how things are to be sent over to Unreal.
- Inside the Paths tab, enter the desired location of your mesh / textures / armatures inside the Unreal project. In our case, this will be Game/Subjects/NAME_OF_YOUR_SPECIMEN. You will have to manually create the NAME_OF_YOUR_SPECIMEN folder inside the Subjects folder of the Unreal project, but this will make it possible to make changes to our model in Blender and simply send the updated version over to Unreal without having to set up everything again. Worthwhile!
- Double check the export settings, specifically the .FBX settings. We want Apply Unit to be checked, the Z axis to be up, and NO leaf bones to be added.
- If everything went according to plan (well...), you should also be able to see an additional folder in your Scene Collection titled Export. Anything placed in here, Unreal will attempt to import, when Send to Unreal is clicked (But don't do that now. We still have a few things to check first.)
- If you haven't done so already, launch replicAnt.uproject by double clicking on the file. We need to check whether Unreal has the right plugins enabled to receive data from our Blender instance. Click on Settings (top left) and then on Plugins.
- Check, whether the Editor Scripting Utilities and Python Ediotor Script plugins are activated. If they are not, enable them now.
Nicely done - installation complete! Time to prepare a model in Blender!
Generally, the lower the vertex count, the better, providing things still look reasonably realistic. Unreal engine deals very well with ridiculous numbers of polygons, but simulating hundreds of high resolution insects can still be a challenging task for your machine. Aim for >20,000 triangles inside your model, and you should be retain a good relationship with your computer. For further details, refer to our 01 Retopologising 3D models guide.
To make use of our provided animation blueprints, follow our standard rigging convention, as detailed in the 02 Rigging 3D models guide.
In theory, any model and rig combination is possible. Make sure, your model is posed with all tarsi in ground contact in a somewhat natural pose. This is important, as we are going to use IK solvers to have the appendages interact with elements in the generated environment.
This guide assumes that your model is already 01 retopologised and 02 rigged! So let's get started:
- Apply your armature in the modifier tab, and delete all vertex groups. We need to do this once, so all units and joint orientations will scale correctly.
- Apply all transforms (scale, rotation, position) to your mesh. In Object mode, select the mesh, hit "ctrl + a" and select All Transforms. Repeat this step for the armature. Without applying transforms, Unreal and Blender will throw all sorts of fun error messages at you. Nobody wants that.
- Now select the armature and go into pose mode. Click Pose > Apply > Apply Pose as Rest Pose, to make the current pose the default.
- Parent the mesh back to the armature with automatic weights. Double check the influence of each joint via weight painting. If joints affect unwated areas, this will lead to unrealistic generated poses.
- [Optional] Create Normal maps. This can be done either in Blender directly by baking a normal map as an additional image texture (see texture baking), in Photoshop, or, nowadays it seems, in literally any image editing program… However, it’s best to do it directly in Blender to preserve information between unwrapped regions, which avoids sharp artifacts between islands in the unwrapped texture.
- Finally, in the Scene Collection, move both your armature and your mesh into the Export collection. Now you can click on Pipeline > Export > Send to Unreal. Remember, your Unreal project needs to be open in order for this to work, and you must have created the specified folder referenced in the export settings (see step 3 of the installation guide above)
… and hope for the best.
- Check your transferred models in the Content browser to ensure they made it over to the Unreal Project.
- Create a material instance from Subjects/M_Subject01_SSInsects and drag it into your model folder.
- Open the material instance and replace the existing textures (albedo & optionally normal) with your own. (In case these materials are not editable, instead of making a material instance of Subjects/M_Subject01_SSInsects, make a copy of it and try again.)
- Assign the SK_InsectBase_Skeleton to your skeletal mesh by right clicking it and selecting Skeleton > Assign Skeleton.
Quick note:
In some cases, you will see an additional skeletal element listed here, named root, which may prevent correct assignment of the skeleton (If you proceed, the animation blueprints may no longer work correctly for this particular mesh).
Double-check (in Blender) whether your Armature follows the same naming convention as our base_rig. It had better. Refer to the Troubleshooting guide for additional guidance.
- Compute the low-poly collision mesh from the Physics Asset of your imported mesh. Open the Physics Asset and delete any hierarchy that is already present. Then, under Tools (right side) match the parameters shown in the image below and click on Genrate All Bodies.
- Create a Child Blueprint Class from SubjectBase/BP_sbj_InsectBase and also move it into your mesh folder.
- Rename the Child Blue Print Class to something more fitting, so you can find it when you want to select it inside the generator. Then open the file. Next, provide a "Class name" which is how your subject model will be referred to in the generated Batch Data File.
- Click on Open Full Blueprint Editor. In there, select the Insect_SkeletalMesh on the left side and then make the following changes on the right side:
- for the Anim Class, select ABP_InsectBase_C
- for the Skeletal Mesh, select your newly imported skeletal mesh
- for the Materials > Element 0, select the created Material Instance (not the original material, as otherwise the generator cannot apply variations to it at runtime)
- Select your new mewly created Child Blue Print Class as part of your Colony in the FARTS_Interface, under Colony > Types. You can also add additional subjects here - the sky (available computational power) is the limit. By pressing on Spawn Preview Colony you can see what the generated colony will look like. (You may need to move to the top of the generated world, where the Colony Preview Groundplane is located.
- You did it! (well, hopefully)
If everything has gone according to plan, you should now be able to hit Generate and watch the magic happen. In the next section, we will detail what each of the generator parameters does.
In case you encounter any problems, consult our troubleshooting guide, or consider raising an issue on the replicAnt GitHub page.
© Fabian Plum, Rene Bulla, David Labonte 2023 MIT License