How do batch size come into play in the S3IM formulation

[haiphamcse]

Hi there, loved your work and currently trying to add it to some neural field research. I have a question about the batch size of a ray in your formulation:
1/ Suppose a batched ray size (batch, num_rays, 3). How would we construct a patch P? I'm currently iterating through each batch (num_rays, 3) batch times and averaging the results. However I noticed here that the loss varies with different batch size (more batch -> higher losses). Is this consistent with your formulation?
2/ Have you tested with other patch size other than 64x64. Does bigger patch size have any affects on perfomance?

[Madaoer]

Hello haiphamcse

1. We use the final rgb piexl(batch, 3) instead of intermediate rgb points(batch, num_rays, 3) to make a virtual patch. For example, (4096, 3) --> (64, 64, 3).
2. We use 64 x 64 because in original nerf methods(DVGO, TensoRF) use 4096 as default setting. Due to the repeat time in our methods, I think s3im already have enough randomness, so a bigger patch size maybe not that helpful.

[haiphamcse]

What is the right way to use S3IM if the number of rgb pixels is larger than (4096,3). Do we just change the kernel size proportionally?

[Madaoer]

No need for increasing kernel size. For example, we can keep the virtual patch as a shape like (64, batch_size / 64 / 3, 3)

[haiphamcse]

Thank you for your reply! I understand now