Bug in model evaluation—datasets with more than 1000 images have very low PSNRs. Easy fix.

[tristanengst]

The render_test_evaluation() function in train_nvfi() writes generated images to disk with files names r_some_number.png where the some_number indicates the position of the corresponding target in all_targets. However, the read_images_in_dir() function sorts these images by their filenames as strings and not numbers, and hence there is a mismatch between the ordering of generated examples and targets.

This shouldn't be impactful if there are fewer than 1000 images since the images have up to three zeros prepended to their index in the corresponding filename.

The fix is simple: in the read_images_in_dir() function in utils/metrics.py, one must replace

fnames.sort()

with

remove_non_numeric = lambda x: ''.join(filter(str.isdigit, x))
fnames = sorted(fnames, key=lambda x: int(remove_non_numeric(x)))