Using other samplers for Aggregator apart from GridSampler

[alabamagan]

Is it possible to use sampler other than the GridSampler for inference?

Currently, the GridAggregator requires a GridSampler but sometimes it is advantageous to use a probabilistic sampler to optimize the results by averaging the overlapping results (sort of a majority vote).

[fepegar]

I have never heard of this. But if you have ideas, we can discuss supporting new samplers for inference.

[alabamagan]

I mean using the same type of sampler (e.g. WeightedSampler) in both training and inference. I generally experience bad pixel prediction at locations where the WeightedSampler seldom reaches with inference done using GridSampler (understandable), so I used to sample with identical sampler before I used torchio and got better results in 2D cases. Of course a much higher patch_per_volume or patch_per_slice would be needed to ensure the interested tissues are properly covered.

[alabamagan]

After a little bit of digging I think its quite possible that the WeightedSampler can be used in a fashion like GridSampler during the inference pipeline. It seems like WeightedSampler just need a method to generate patches from a subject and a __getitem__ method that returns the patches as tio.Subject with the crop location availble through tio.LOCATION

Do correct me but It seems that 'index_ini' and tio.LOCATION are equivalent?

[fepegar]

The samplers infrastructure is sometimes a bit inconsistent due to historical reasons. I just committed a fix to remove index_ini and add tio.LOCATION to all patches. That will hopefully help you experiment.

It seems like WeightedSampler just need a method to generate patches from a subject and a getitem method that returns the patches as tio.Subject with the crop location availble through tio.LOCATION

This is pretty much available. Except that you need to decide how many patches you want to extract, because sampler(subject) is a generator that will yield an infinite number of patches.

[alabamagan]

The samplers infrastructure is sometimes a bit inconsistent due to historical reasons. I just committed a fix to remove index_ini and add tio.LOCATION to all patches. That will hopefully help you experiment.

Thanks for the clarification! I understand the struggle.

This is pretty much available. Except that you need to decide how many patches you want to extract, because sampler(subject) is a generator that will yield an infinite number of patches.

I have managed to pull off a working (but ugly) way to do this, though there are still some memory issues, but I will settle for now. Thanks again for the help!

[romainVala]

Note sure if it is related, to what you describe here, but I give my 2 cents here ...

I experience bad prediction on the patch border at inference, so I used a gridsampler with overlap, and when I tested different overlap values, I observed improved results as the overlap increase, but the cost was very high in computation time (because a lot of patch to process),
then I test an inference of the whole volume (without patch) and the results was much better, (and shorter to compute)
But ok, this may not be possible with any model, you need to have a Fully convolutional model, so that it can take a different input size (compare to the training patch size)
So if it is possible, give a try to run the inference on the all input volume

[fepegar]

Yes, I've always obtained better results using the whole volume. Automatic mixed precision (AMP) helps. And yes, larger overlap helps as well!

[alabamagan]

Note sure if it is related, to what you describe here, but I give my 2 cents here ...

I experience bad prediction on the patch border at inference, so I used a gridsampler with overlap, and when I tested different overlap values, I observed improved results as the overlap increase, but the cost was very high in computation time (because a lot of patch to process),
then I test an inference of the whole volume (without patch) and the results was much better, (and shorter to compute)
But ok, this may not be possible with any model, you need to have a Fully convolutional model, so that it can take a different input size (compare to the training patch size)
So if it is possible, give a try to run the inference on the all input volume

@romainVala Thanks for the input! But I think we are describing two different issues, I am guessing you were referring to the incoherence at the patch edges because of local contrast sharpening perhaps?

My case is different. I deal with tractography of certain brain region, say the frontal lobe, so I trained the network with a frontal lobe mask using WeightedSampler, problem is, this also means the network won't see the rest of the brain tractography during training. Now this is alright as long as the network sees the same thing in inference by using the identical sampler with identical sampling statistics, which simply means GridSampler would be unsuitable for inference. This might be easier to solve in segmentation (still could be a problem though), but my goal was to construct a final prediction for the each case from multiple predictions deduced from many frontal lobe patches, so contributions from irrelevant patches will affect outcome and there seems to be no clean to filter them away in the current setting.

[fepegar]

I think this is the typical use case of a weighted sampler. Why not let your network see other parts of the brain as well? If you trained a face detector, you wouldn't show the net only faces, but images containing faces and other things, or only other things. You could use a stronger weight for your frontal lobe. Some people use distance maps of the segmentation, for example.

[alabamagan]

I incline to say you are correct, but clinical doctors are having increased trust issues with the black-box nature of DNNs, hence this. Some of them would not be convinced the unless the information exposed to the network has biology relevance. Although I have to say, in some situation, they have a point, as in cysts are typically harmless, unless it grown in some particular region.