Training with masks results in artefacts outside the mask regions

[nepfaff]

Hi, I want to train a NeRF of a masked object for mesh + texture map extraction. However, I'm getting unexpected results where the renders include artefacts outside the masked regions.

I'm using the same dataset (download here) for all the following results. I additionally copied the mask parsing logic from the nerfstudio_dataparser to the instant_ngp_dataparser.

I built nerfstudio from source, commit 4d333f4.

instant-ngp-bounded is giving me the most similar results to the official instantNGP implementation. However, I'm still getting artefacts at the top of the scene bounding cube.

Are there any settings that I could change to obtain my desired results?

Desired results (Using the official instantNGP implementation with masked images and black_transparent=True)

Nerfstudio Nerfacto results (default settings)

ns-train nerfacto instant-ngp-data --data data/mustard

Nerfstudio instant-ngp-bounded results (default settings)

ns-train instant-ngp-bounded instant-ngp-data --data data/mustard

[tancik]

Mess around with --scene-scale argument.

[nepfaff]

This does not seem to solve my problem as it would only remove the artefacts completely if I could get the ROI bounding box to be exactly my object of interest.

I looked more into the difference to the official instantNGP implementation and realized that its black_transparent works very differently. It aims to achieve zero density at coordinates corresponding to black image pixels. This is more powerful than only using masked pixels for rendering. instantNGP also provides the weaker mask option as nerfstudio does, as this is the desired approach if images contain, for example, dynamic objects that should be ignored.

In this case, can I convert this into a feature request to also support transparency enforcement?

[jkulhanek]

I think this would be a nice feature and many users (myself included) would find it useful. I needed this to mask out sky recently.

[chl2]

Looking forward to the processing of the alpha channel

[qhdqhd]

Looking forward to the processing of the alpha channel

[f-dy]

There are really two kinds of mask:

1. masks to indicate areas that shouldn't be sampled , such as moving objects, or the black area outside of the image disc in a fisheye camera
2. masks that indicate areas that should have zero density

nerfstudio is implementing the first kind, which is useful in many situations (fisheye being one). instant-ngp implements the second.

I think both options are useful, and should be kept separate

[nepfaff]

The alpha transparent training implemented in #2025 produces the desired results when used with instant-ngp-bounded. See #2025 (comment). This could be closed if that PR or a similar implementation is merged.