Semantic Segementation Camera with Niagara Systems

[ali-0-1]

Hi,

Platform: Carla ue5-dev built from source on Ubuntu 22.04. Nvidia RTX 5090 24gb.

Developed a custom gas system via Niagara Systems. It is CPU based gas emitter, Sprite renderer.
We also have a custom map, in our map almost all objects tagged well in semantic camera but gas particles are not visible in semantic segmentation camera.
Modified Tagger.cpp as:

#include "NiagaraComponent.h"

for (USkeletalMeshComponent *Component : SkeletalMeshComponents) {
 existing code
}

// Iterate Niagara components
TArray<UNiagaraComponent*> NiagaraComponents;
Actor.GetComponents<UNiagaraComponent>(NiagaraComponents);

for (UNiagaraComponent* Component : NiagaraComponents)
{
  if (Component == nullptr) continue;

   //  reuse existing Fences label
  crp::CityObjectLabel Label = crp::CityObjectLabel::Fences; 
  // or define Gas if you extend enum

  SetStencilValue(*Component, Actor.GetUniqueID(), Label, bTagForSemanticSegmentation);

  Component->SetRenderCustomDepth(true);  // VERY IMPORTANT
  Component->SetCustomDepthStencilWriteMask(ERendererStencilMask::ERSM_Default);
  Component->SetCustomDepthStencilValue((uint8)CastEnum(Label));

  Component->ComponentTags.Add(FName(*GetTagAsString(Label)));

#ifdef CARLA_TAGGER_EXTRA_LOG
  UE_LOG(LogCarla, Log, TEXT("  + NiagaraComponent: %s"), *Component->GetName());
  UE_LOG(LogCarla, Log, TEXT("    - Label: \"%s\""), *GetTagAsString(Label));
#endif
}

Passed Niagara in build.cs both private and public dependency module names.


in gas material:
Allow Custom Depth Writes enabled,
Material Domain is Surface,
Blend mode changed to Masked,
Shading mode changed to Unlit, also experimented with Default,
Experimented also by: [Surface, Additive, Unlit], [Surface, Translucent, Unlit], [Surface, Translucent, Default Unlit],
Post Process Material section in gas material is faded out, cannot modify these values, may the issue is hidden there.

For test purpose placed the gas material into a known folder like Static/Fence to get gas particle is tagged as Fence color, but no appearance in semantic camera.
Also, used Mesh renderer in Niagara System asset. As a mesh, reused any fence mesh from Static/Fence folder, even though the fence placed to the level is tagged (labeled) in semantic camera, emitted fences from Niagara System are not tagged. It is a bit funny but just experimenting, as seen in below image, the fence behind the pipe is labeled, while the emitted ones are not.
What is preventing gas particles to be labelled? Is there any work around for this situation? Any help will be appreciated.

[ali-0-1]

Actually the code for Tagger.cpp helps to get given label class (in our case it is Fence label).

Actual solution is hidden in Unreal/CarlaUnreal/Plugins/Carla/Source/Carla/Sensor/PostProcessConfig.cpp
By default particle is disabled and needs to be enabled by:

EngineShowFlags.SetParticles(true);

Then in gas material:

Blend Mode = Masked or Opaque
Shading Model = Unlit or some few more others

but not the translucent, additive, modulate, AlphaComposite or AphaHoldout
As emitter renderer Sprite Renderer is fine too. Particles have masked to get kind of particle shape but in semantic segmentation their whole shape is labelled.

There is a work around for this also. if we pick a sphere as mesh renderer then the whole sphere mesh is labeled as below.