RCF with empirical mode

[cullen-molitor]

Hello, excited to see the new release include updates to the RCF model that include the option to sample patches empirically from a dataset! I have the newest version installed and I am trying to implement now.

One thing that has come up is the need to standardize image sizes in this implementation. I think it comes from this line which uses the first image in the dataset to define the height and width. Is this an intentional choice? I was hoping for an approach that did not include any image resizing. I am chopping up satellite imagery from a standard grid that is created in degrees, so depending on where I am in the world, the number of pixels in the height and width dimensions change a bit. Not 100% sure this is my problem, but resizing seemed to get things working.

As an aside, no image resizing is required for the standard gaussian patches.

Thanks!

[calebrob6]

Hey Cullen, there isn't any resizing that happens with empirical sampling in RCF so I'm not entirely sure what you are referring to. There currently is the assumption that all images in the NonGeoDataset that you pass in are the same size (just for sampling patches to use as kernels) -- i.e. the line you linked to -- but this isn't necessary.

[cullen-molitor]

Sorry I worded the question poorly, but yes I was just confirming it was expecting standard sizes for all images passed in when sampling patches. I was also wondering if the choice was intentional and if you see any differences in performance using standardized image sizes. I see now that you can use resized images for the patches, and apply the model to any size images if you wanted to.

Would you expect any benefit or cost from standardizing image sizes when running images through the model? (i.e. not just sampling patches).

Appreciate the help!

[calebrob6]

Got it, to set up empirical RCF there is no reason all the images need to be the same size (other than the way we've currently implemented it). You just need to sample a bunch of K x K patches from your set of input images. I.e. we probably should change the implementation to something like:

    if mode == "empirical":
            assert dataset is not None
            assert num_channels == in_channels
            patches = np.zeros(
                (num_patches, num_channels, kernel_size, kernel_size), dtype=np.float32
            )
            idxs = torch.randint(
                0, len(dataset), (num_patches,), generator=generator
            ).numpy()  # we should probably weight this by the size of each image in the dataset to make the sampling uniform
 
            for i in range(num_patches):
                img = dataset[idxs[i]]["image"]
                
                ### New stuff
                # get the size of this particular `img`
                # choose an x and y coordinate based on the size of this particular `img`
                
                
                patches[i] = ... # sample from `img` based on the x and y coordinate we chose