Benchmarking nerf

[kwea123]

Hi, I tried to reproduce paper result, but cannot get to the number reported. The following is how I did the experiment, could you tell me where I did wrong?

kwea123/ngp_pl#3 (comment)

[Tom94]

Hi there, the NeRF / NSVF / mip-NeRF codebases do not respect color spaces / pre-multiplied alpha correctly and hence their PSNR calculation behaves in non-intuitive ways. instant-ngp tries to be more principled about it, resulting in worse PSNR numbers by default, but you can revert to original NeRF's behavior with the --nerf_compatibility argument.

The following command should let you reproduce the numbers from the paper:

./scripts/run.py --mode nerf --nerf_compatibility --scene lego/transforms_train.json --test_transforms lego/transforms_test.json 

PS: note that you can just pass lego/transforms_train.json instead of the folder to avoid having to delete validation and test transforms.

[kwea123]

Thanks for the tip for passing only .json. However, I was using --nerf_compatible already. The whole command is:

python scripts/run.py --mode nerf --scene /home/ubuntu/data/nerf_data/nerf_synthetic/lego/transforms_train.json --nerf_compatibility --test_transforms /home/ubuntu/data/nerf_data/nerf_synthetic/lego/transforms_test.json

also, to do a really "fair" comparison, I uncomment this line

instant-ngp/scripts/run.py

Line 180 in e4e0053

# testbed.background_color = [1.0, 1.0, 1.0, 1.0]

to make the background white instead of black. The PSNR is like what I showed in the issue, 35.72 (5m) and 35.79 (10m). Switching the background back to black makes the PSNR become 36.06 (higher than white background!?), so I guess black background + longer training will get to the number? However, since most previous methods measure PSNR with white background, I think it's worth noting this difference.

Next, concerning the FPS, in the paper it says 60 FPS, but when I measure the time by enclosing this line of code

instant-ngp/scripts/run.py

Line 282 in e4e0053

image = testbed.render(ref_image.shape[1], ref_image.shape[0], spp, True)

and changing the spp to 1, rendering_min_transmittance to 1e-2

instant-ngp/scripts/run.py

Lines 239 to 241 in e4e0053

	spp = 8
	testbed.nerf.rendering_min_transmittance = 1e-4

I get only 0.033s per frame, i.e ~30FPS. What are the other secrets to get to 60FPS?

[Tom94]

The low PSNR is surprising to me -- it looks like there has been a regression since the paper upload in January. Thanks for reporting it!

Checking out commit 2b6a55f6bfbeda8a01e0da27904cec59e5fd6010 (from Jan 15) and running the command above, I get a PSNR of 36.2 with white background (and 36.4 with black). I'm unsure about what changed (I can't tell a qualitative difference based on visual inspection / flipping back and forth between predictions of old vs. new), but will post an update in case I find something.

As for performance: I suspect you're limited by the CPU<->GPU transfer of the image. Calling testbed.frame() instead of render() after having disabled training and having created a dummy window should give you a more accurate reading.

On my RTX 3090 GPU, I get 95 FPS for the lego dataset using the following code:

	if args.test_transforms:
		print("Evaluating test transforms from ", args.test_transforms)
		with open(args.test_transforms) as f:
			test_transforms = json.load(f)
		data_dir=os.path.dirname(args.test_transforms)

		testbed.snap_to_pixel_centers = True
		testbed.nerf.rendering_min_transmittance = 1e-2
		testbed.background_color = [1.0, 1.0, 1.0, 1.0]
		testbed.init_window(800, 800)
		testbed.fov_axis = 0
		testbed.fov = test_transforms["camera_angle_x"] * 180 / np.pi
		testbed.shall_train = False

		begin = time.monotonic()
		with tqdm(list(enumerate(test_transforms["frames"])), unit="images", desc=f"Rendering test frame") as t:
			for i, frame in t:
				testbed.set_nerf_camera_matrix(np.matrix(frame["transform_matrix"])[:-1,:])
				testbed.reset_accumulation(True, True)
				testbed.frame()

		fps = len(test_transforms["frames"]) / (time.monotonic() - begin)
		print(f"FPS={fps}")