Depth image for translucent objects

[bmunkhtulga]

Hi there, I am trying to create dataset with translucent object but had no luck. In class segmentation case, I was able to bring translucent objects in image thanks to this step.

Question is what do i have to do in order to capture depth image of translucent objects in NDDS?

Here are my results so far. (Cup)

[thangt]

In UE4, if you use a 'Translucent' type material on a 3d model, it will skip the depth of that model and only store the depth of the pixel behind it.
If you want to use translucent material and want UE4 to record the depth of the model, it will be a new feature I need to implement.
Right now the only work around I can think of is to change the material blend mode to be 'Opaque' and set up its refraction. That way you can still capture the refractioness of the object's material and still capture the object's depth.

[bmunkhtulga]

Thank you very much. I will try that workaround.

[bmunkhtulga]

I am still trying to take depth image of transparent object. And i heard there is one more workaround which duplicates the mesh with an opaque material currently. You can set that copy to not render in main pass and to render only in custom depth.

But it seems SceneCapturer only captures objects that are rendered in main pass. If SceneCapture can capture depth image from CustomDepth pass, i think this workaround will work much better than one mentioned above.

Does anyone know about anything about this?

[thangt]

I do know about this. It was a bit complicated so I wanted to implement it as a new feature to make controlling which transparent object should be captured in the depth map.
You can actually do similar thing with the current NDDS by using the IgnoreActors array in the feature extractors.

There is a bug in the code so you need to change the code first:

1. Go to Source\Plugins\NVSceneCapturer\Source\NVSceneCapturer\Private\NVSceneFeatureExtractor_ImageExport.cpp
2. Go to line 116, it should say: if (!CheckActor)
3. Remove ! so it become: if (CheckActor)
4. Build the NDDS again.
   NOTE: I will push a fix for this in the next patch, in the mean time, you can do this locally.

After you changed the code, go to the map:
1, Duplicate the transparent actor (RealActor) to get a new one: DupActor (example name)
2, In DupActor:
2.1, Replace the material of its mesh with a non translucent one (it can be anything, event default debug material)
2.2, Remove the AnnotationTag
3, Set up the transform of DupActor to be the same as the real one
4, Go to the Feature Extractor Settings part in the SceneCapturer (the one you placed in the map)
5, Select the FeatureExtractorTrueColor - the one that capture RGB image:
5.1, Add new element to its Ignore Actors array
5.2, Add the DupActor to the array
NOTE: This will make sure the RGB image doesn't include the DupActor

The depth feature extractor should be able to capture the depth from the DupActor and the DupActor won't appear on the RGB image.

Again, as you can see, the whole process is quite manual and need to be repeated to all the objects. I have plan to make it more automated but we need more time to implement and test it.

[bmunkhtulga]

Thank you for walk-through. Manual work won't be problem.

As I was following the steps, i wasn't able to add the DupActor to the array. I selected it is still displaying None. And i tried other actors such as SceneManager_BP or SceneCapturer_AllFeatureExtractors, they were okay. Is there any reason I can't add DR_AnnotatedActor_BP to the array?

[bmunkhtulga]

Thank you for walk-through. Manual work won't be problem.

As I was following the steps, i wasn't able to add the DupActor to the array. I selected it is still displaying None. And i tried other actors such as SceneManager_BP or SceneCapturer_AllFeatureExtractors, they were okay. Is there any reason I can't add DR_AnnotatedActor_BP to the array?

I was able to do it on SceneCapturer_Simple, not on SceneCapturer_AllFeatureExtractors.

[thangt]

The reason you can't add some of the actors is because they belong to different sub-level. https://docs.unrealengine.com/en-US/Engine/Levels/LevelsWindow/index.html
The SceneCapturer can only reference to the actor that belong to the same sublevel as itself. I think in the default map, the SceneCapturer belong to a sublevel while the annotated actor belong to the main level.