noisy data set

[ranch-hands]

hello dear author @edgarschnfld

I downloaded the pre-trained model and added noise to the labels that are input for the network but the test failed and the model didn't work for the noisy dataset at all.
what should I do for testing the model on noise?

[SushkoVadim]

Hi,

Could you please describe in more detail what kind of noise you want to add?
Please note that the input label maps are represented as one-hot encodings of shape [NxWxH], where N is the number of classes used in the dataset. So adding gaussian noise, for example, is not a meaningful operation, since the resulting tensor is no longer a one-hot encoding tensor.
On the other hand, adding "noise" by slight random reshaping of semantic boundaries should work.

[ranch-hands]

Hi I wanted to add gaussian, spekle, salt and pepper and jpeg compression noise to labels. I added salt and pepper and gaussian and the network failed to run. It must be because of what you said but I don't know what it is and how I am suppose to test the network. Does the test work if I train it with noisy data set? Thanks alot Best

…

On Thu, Jul 29, 2021, 01:17 SushkoVadim ***@***.***> wrote: Hi, Could you please describe in more detail what kind of noise you want to add? Please note that the input label maps are represented as one-hot encodings of shape [NxWxH], where N is the number of classes used in the dataset. So adding gaussian noise, for example, is not a meaningful operation, since the resulting tensor is no longer a one-hot encoding tensor. On the other hand, adding "noise" by slight random reshaping of semantic boundaries should work. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASMJBK3HVVJ7E4TSOL7PUETT2BUHTANCNFSM5BCVNYLA> .

[SushkoVadim]

Hi again,

What you are describing is usually applied to images, not label maps.
Label maps that are used in semantic images synthesis are not RGB images, instead, they are discreate maps saying "which class ID is correct for which pixel". Have a look at top-left image here: https://github.com/boschresearch/OASIS/blob/master/overview.png

So if you wish to develop a model on noisy datasets, you should add noise to images.
Feel free to describe in more detail your use case (e.g. why you need to add noise). Then I would be probably able to give a more detailed advice.

Best,
Vadim

[ranch-hands]

Thank you for your reply. I wanted to add noise to input images so that I could evaluate the output of the gan network. Since the input of the network is those label maps, I added noise to them and it was wrong. but I dont know how I am suppose to add noise to the input. If I add noise to the images it is like I added noise to the labels of network somehow. Am I correct? By the way, how can I use the network vice versa?I mean is it possible to give the network pictures and it turns it to those lables? Best

…

On Fri, Jul 30, 2021, 17:57 SushkoVadim ***@***.***> wrote: Hi again, What you are describing is usually applied to images, not label maps. Label maps that are used in semantic images synthesis are not RGB images, instead, they are discreate maps saying "which class ID is correct for which pixel". Have a look at top-left image here: https://github.com/boschresearch/OASIS/blob/master/overview.png So if you wish to develop a model on noisy datasets, you should add noise to images. Feel free to describe in more detail your use case (e.g. why you need to add noise). Then I would be probably able to give a more detailed advice. Best, Vadim — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASMJBKYTYYFHFRVQYU3C55TT2KSDDANCNFSM5BCVNYLA> .

[SushkoVadim]

Hi, it is actually still not clear to me what is your purpose of distorting label maps with noise.
If you wish to use the noise of the latent space (for example, to replicate our results on global-local resampling, or interpolations), then this noise is already implemented in the generator's forward pass:

OASIS/models/generator.py

Line 36 in ce5d97d

z = torch.randn(seg.size(0), self.opt.z_dim, dtype=torch.float32, device=dev)

For the second question: yes, you can use our segmentation-based discriminator to predict semantic label maps of images.
For this, you just need to take D's predictions, remove the "fake" class, and take argmax among the remaining classes.

[ranch-hands]

HI,
thanks for your descriptions. by the way I don't understand them completely. because I'm new to these subjects.
how much does it take to be as mature as you in these subjects( like PyTorch, building gans, etc)?
It is also hard to change this network to work vice-versa.
by the way, I just wanted to evaluate how much your network is resistant to noise, I decided to add noise like the gaussian one to the input to see the results in the output. to do this where should I add noise?
and another question is that when I want to test the network it wants both annotations and images, but I'm confused because it needs the images just for training. and in the test phase, it should just use the annotations to run, shouldn't it?
Best Dear Vadim

[SushkoVadim]

Hi,

It is very difficult to say how long it takes, since such things depend a lot on one's background knowledge :)
(like python, math, basics of machine learning and neural networks, etc ).

The input to our network are a semantic label map with shape [BxCxWxH] and gaussinan noise [Bx64xHxW].
If you want to see how noise in labels affects performance, you should apply perturbations to the label maps.
These perturbations should preserve the structure of the label map shape [BxCxWxH], and have one-hot encoding structure along axis 1. So gaussian noise is not a good perturbation in this case, because the resulting tensor is not a label map anymore.
Examples of possible perturbations: randomly change class IDs for some semantic regions, change some pixels to "don't care" class, etc.

The network actually does not need images for testing, it asks only labels to be feeded:

OASIS/test.py

Line 24 in ce5d97d

_, label = models.preprocess_input(opt, data_i)

However, our dataloader implementation assumes you always have images in the dataset too, in order to be able to compute the FID metric to validation set during training. You can simply deactivate validation images loading in the dataloader class to avoid this.

[ranch-hands]

I tested the network with a custom dataset but it doesn't work. (the CK points are from ade20k)
it says: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
(the same error when i added noise to labels! so I can guess I should do sth about the labels, but what? )
can you guide me what to do?
thanks alot for your patience

[ranch-hands]

Hi,

It is very difficult to say how long it takes, since such things depend a lot on one's background knowledge :)
(like python, math, basics of machine learning and neural networks, etc ).

The input to our network are a semantic label map with shape [BxCxWxH] and gaussinan noise [Bx64xHxW].
If you want to see how noise in labels affects performance, you should apply perturbations to the label maps.
These perturbations should preserve the structure of the label map shape [BxCxWxH], and have one-hot encoding structure along axis 1. So gaussian noise is not a good perturbation in this case, because the resulting tensor is not a label map anymore.
Examples of possible perturbations: randomly change class IDs for some semantic regions, change some pixels to "don't care" class, etc.

The network actually does not need images for testing, it asks only labels to be feeded:

OASIS/test.py

Line 24 in ce5d97d

_, label = models.preprocess_input(opt, data_i)

However, our dataloader implementation assumes you always have images in the dataset too, in order to be able to compute the FID metric to the validation set during training. You can simply deactivate validation images loading in the dataloader class to avoid this.

what are the B and C in [BxCxWxH]? and when you say have one-hot encoding structure along axis 1, you mean along with C?
why is Gaussian noise has 64 channels in [Bx64xHxW]?
how can I check that my perturbations preserve the structure of the label map shape [BxCxWxH]?

thanks a lot
Best